SAS Cabling

SAS Disk ShelvesUniversal SAS and ACP Cabling Guide

NetApp, Inc.495 East Java DriveSunnyvale, CA 94089 USATelephone: +1 (408) 822-6000Fax: +1 (408) 822-4501Support telephone: +1 (888) 4-NETAPPDocumentation comments: [email protected] Web: www.netapp.com

Part number: 215-05500_A0September 2010

Copyright and trademark information

Copyright information

Copyright © 1994–2010 NetApp, Inc. All rights reserved. Printed in the U.S.A.

No part of this document covered by copyright may be reproduced in any form or by any means—graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an electronic retrieval system—without prior written permission of the copyright owner.

NetApp reserves the right to change any products described herein at any time, and without notice. NetApp assumes no responsibility or liability arising from the use of products described herein, except as expressly agreed to in writing by NetApp. The use or purchase of this product does not convey a license under any patent rights, trademark rights, or any other intellectual property rights of NetApp. The product described in this manual may be protected by one or more U.S.A. patents, foreign patents, or pending applications.

RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.277-7103 (October 1988) and FAR 52-227-19 (June 1987).

Trademark information

NetApp; the NetApp logo; the Network Appliance logo; Bycast; Cryptainer; Cryptoshred; DataFabric; Data ONTAP; Decru; Decru DataFort; FAServer; FilerView; FlexCache; FlexClone; FlexShare; FlexVol; FPolicy; gFiler; Go further, faster; Manage ONTAP; MultiStore; NearStore; NetCache; NOW (NetApp on the Web); ONTAPI; RAID-DP; SANscreen; SecureShare; Simulate ONTAP; SnapCopy; SnapDrive; SnapLock; SnapManager; SnapMirror; SnapMover; SnapRestore; SnapValidator; SnapVault; Spinnaker Networks; Spinnaker Networks logo; SpinAccess; SpinCluster; SpinFlex; SpinFS; SpinHA; SpinMove; SpinServer; SpinStor; StorageGRID; StoreVault; SyncMirror; Topio; vFiler; VFM; and WAFL are registered trademarks of NetApp, Inc. in the U.S.A. and/or other countries. Network Appliance, Snapshot, and The evolution of storage are trademarks of NetApp, Inc. in the U.S.A. and/or other countries and registered trademarks in some other countries. The StoreVault logo, ApplianceWatch, ApplianceWatch PRO, ASUP, AutoSupport, ComplianceClock, DataFort, Data Motion, FlexScale, FlexSuite, Lifetime Key Management, LockVault, NOW, MetroCluster, OpenKey, ReplicatorX, SecureAdmin, Shadow Tape, SnapDirector, SnapFilter, SnapMigrator, SnapSuite, Tech OnTap, Virtual File Manager, VPolicy, and Web Filer are trademarks of NetApp, Inc. in the U.S.A. and other countries. Get Successful and Select are service marks of NetApp, Inc. in the U.S.A.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. A complete and current list of other IBM trademarks is available on the Web at www.ibm.com/legal/copytrade.shtml.

Apple is a registered trademark and QuickTime is a trademark of Apple, Inc. in the U.S.A. and/or other countries. Microsoft is a registered trademark and Windows Media is a trademark of Microsoft Corporation in the U.S.A. and/or other countries. RealAudio, RealNetworks, RealPlayer, RealSystem, RealText, and RealVideo are registered trademarks and RealMedia, RealProxy, and SureStream are trademarks of RealNetworks, Inc. in the U.S.A. and/or other countries.

All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.

NetApp, Inc. is a licensee of the CompactFlash and CF Logo trademarks. NetApp, Inc. NetCache is certified RealSystem compatible.

ii Copyright and trademark information

Table of Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

Chapter 1 Cabling Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 2 SAS Cabling for New Storage Systems . . . . . . . . . . . . . . . . . . . . 3

SAS cabling rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Cabling SAS connections. . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

SAS cabling examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Chapter 3 ACP Cabling for New Storage Systems . . . . . . . . . . . . . . . . . . . 25

ACP cabling rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Cabling ACP connections . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

ACP cabling examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Appendix A Terms and Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Appendix B Document Update Record . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Table of Contents iii

iv Table of Contents

Preface

About this guide This guide provides information about cabling SAS disk shelves for new storage system installations.

This guide does not provide cabling information for hot-adding or replacing disk shelves. For those procedures, see the Installation and Service Guide at now.netapp.com for your disk shelf model.

The disk shelf illustrations show a generic SAS disk shelf, which has the two IOMs side-by-side. No power supplies are shown.

Audience This guide is for system administrators who are familiar with operating systems that run on the storage system’s clients, such as UNIX®, Windows 95™, Windows NT®, Windows® 2000, and Windows XP®. The procedures in this guide describe replacement, upgrade, and maintenance tasks for personnel with the following skills and experience:

◆ Working familiarity with small computer system hardware and operation

◆ Basic understanding of common networking concepts and practices

◆ Working familiarity with accepted tools and procedures for installing and operating sensitive electronic equipment

Formatting conventions

The following table lists the kinds of formatting this guide uses to identify special information.

Formatting convention Type of information

Italic type ◆ Words or characters that require special attention.

◆ File names and path names.

◆ Man page names.

◆ Book titles in cross-references.

◆ Placeholders for information you must supply. For example, if the guide says to enter the command arp -d hostname, you enter the characters “arp -d” followed by the actual name of the host.

Preface v

Special messages This guide contains special messages that are described as follows:

NoteA note contains important information that helps you install or operate the system efficiently.

AttentionAn attention notice contains instructions that you must follow to avoid a system crash, or loss of data, or damage to the equipment.

DANGERA danger notice warns you of conditions or procedures that can result in death or severe personal injury.

CAUTIONA caution notice warns you of conditions or procedures that can cause personal injury that is neither lethal nor extremely hazardous.

Monospaced font

◆ Command and daemon names.

◆ Information displayed on the system console or other computer monitors.

◆ The contents of files.

Bold

monospace fontWords or characters you type. What you type is always shown in lowercase letters, unless you must type it in uppercase letters for it to work properly.

Formatting convention Type of information

vi Preface

Chapter 1: Cabling Requirements

1
Cabling Requirements
Before you begin To cable a new storage system with SAS disk drives, your storage system must meet the following requirements:

◆ Your storage system must be running the minimum software release required to support your model of SAS disk shelves.

See the Installation and Service Guide for your particular disk shelf model or the System Configuration Guide at now.netapp.com.

◆ If your storage system was not shipped in a system cabinet, the controllers and disk shelves must already be installed in a rack.

◆ If your storage system will be using one or more PCIe SAS HBAs, they must already be installed and available for cabling the disk shelves.

◆ If you are cabling for the ACP capability and your storage system does not have a native onboard ACP port, you must be prepared to designate a network interface for each controller at system setup.

If you are using a four-port Ethernet card for this purpose, this card must already be installed and available in each controller.

1

2 Cabling Requirements

Chapter 2: SAS Cabling for New Storage Systems

2
SAS Cabling for New Storage Systems
About this chapter This chapter provides information about cabling the disk shelf SAS connections for a new storage system installation.

This chapter does not provide cabling information for hot-adding or replacing disk shelves. For those procedures, see the Installation and Service Guide at now.netapp.com for your disk shelf model.

NoteReferences to the top and bottom disk shelves in a stack are to the logical top and bottom of a stack. Your physical stack arrangement might be different.

Topics in this chapter

This chapter discusses the following topics:

◆ “SAS cabling rules” on page 4

◆ “Cabling SAS connections” on page 10

◆ “SAS cabling examples” on page 15

3

SAS cabling rules

Topics in this section

The following topics are covered in this section:

◆ “General SAS cabling rules” on page 4

◆ “Onboard SAS port rule” on page 5

◆ “Shelf-to-shelf connection rule” on page 6

◆ “Controller-to-shelf connection rules” on page 6

General SAS cabling rules

The following SAS cabling rules apply to all supported storage systems using SAS HBAs and on board SAS ports:

Supported configurations: Multipath HA is the supported configuration, with the following exceptions:

❖ For single-controller configurations, 2040 storage systems use single-path and all other supported platforms use dual-path.

❖ 2040 systems in an HA pair configuration use single-path HA.

SAS cables used: QSFP to QSFP SAS cables are used to connect disk shelves together and to connect disk shelves to the SAS ports on the controller.

QSFP to mini-SAS cables are used for 2050 configurations only, and connect disk shelves to the dual-port mini-SAS HBAs on the controller.

4 SAS cabling rules

SAS port designations: All SAS ports on the disk shelf are designated by a square symbol or a circle symbol. The following illustration shows square and circle labeling on the IOM ports.

NoteSAS connectivity does not use the concept of In ports and Out ports.

Onboard SAS port rule

To ensure that the SAS cabling rules apply to all SAS HBAs on a controller, whether in a physical PCI slot (slot 1-N) or on board the base controller, an on board HBA is defined as occupying PCI slot 0. Each port is named with sequentially increasing letters (0a, 0b, 0c, and so on).

For example, the following illustration shows a controller with one onboard SAS HBA in slot 0 and three SAS HBAs in physical PCI slots 1, 2, and 3.

Exception: The only exception is the 2040 storage system. Treat the onboard SAS port labeled 0d as if it were port 0a. Treating the port as 0a enables you to apply all the SAS cabling rules. (Data ONTAP continues to recognize this port as 0d.)

ACP SAS

IOM A IOM B

ACP SAS

B

A

C

D

Slot 1

B

A

C

D

Slot 3

B

A

C

D

Slot 2

B

A

Slot 0

2040 controller

0a

0d

Chapter 2: SAS Cabling for New Storage Systems 5

Shelf-to-shelf connection rule

All the disk shelves in a stack are daisy-chained when you have more than one disk shelf in a stack.

◆ IOM A circle port is connected to the next IOM A square port.

◆ IOM B circle port is connected to the next IOM B square port.

Controller-to-shelf connection rules

Each controller connects to each stack of disk shelves in the system through the controller SAS ports, which can be A, B, C, and D, (and can be on a SAS HBA in a physical PCI slot [slot 1-N] or on board the base controller).

For quad-port SAS HBAs, the controller-to-shelf connection rules ensure resiliency for the storage system based on the ASIC chip design. Ports A and B are on one ASIC chip, and ports C and D are on a second ASIC chip. By using ports A and C to connect to the top shelf in each stack, and using ports B and D to connect to the bottom shelf in each stack, the controllers maintains connectivity to the disk shelves if an ASIC chip fails.

The following illustration shows a quad-port SAS HBA with the two ASIC chips and their designated ports.

Bottom shelf

Top shelf

ACP SAS ACP SAS

IOM A IOM B

Stack of three shelves

Quad-port SAS HBA

ASIC chip

ASIC chip

C portD port

B portA port

PCI-e chip

6 SAS cabling rules

SAS port A and C connections: SAS ports A and C on a controller are connected to the stacks of disk shelves in this way:

◆ SAS ports A and C are connected to the top disk shelf square port in each stack of disk shelves.

❖ The A and C ports on controller 1/A connect to IOM A.

❖ The A and C ports on controller 2/B connect to IOM B.

◆ SAS ports A and C are connected to the disk shelf stacks in sequential order of the HBA slots (slot 0, slot 1, slot 2, and so on).

For example, a controller with slots 1, 2, and 3 occupied with quad-port SAS HBAs is cabled in order of HBAs and ports 1a, 2a, 3a, 1c, 2c, 3c, as shown below.

Controller 1/A

Slot 1 Slot 1

Controller 2/B

Stack 1 Stack 2

A

B B

C C

D D

A

ACP SAS ACP SAS ACP SAS ACP SAS

IOM A IOM B IOM A IOM B

Top shelf

Bottom shelf

Slot 1

B

D

Slot 2 Slot 3

Controller 1/A

A

C

B

D

A

C

B

D

A

C


SAS port B and D connections: SAS ports B and D on the controller are connected to the stacks of disk shelves in this way:

◆ Ports B and D are connected to the bottom disk shelf circle port in each stack of disk shelves.

❖ The B and D ports on controller 1/A connect to IOM B.

❖ The B and D ports on controller 2/B connect to IOM A.

◆ SAS ports B and D are connected to the disk shelf stacks by offsetting the order of the HBA slots by one so that the first HBA port is cabled last. The cabling offset ensures system resiliency.

For example, a controller with three quad-port SAS HBAs in slots 1, 2, and 3 is cabled in order of HBAs and ports 2b, 3b, 1d, 2d, 3d, 1b, as shown below.

Controller 1/A

Slot 1 Slot 1

Controller 2/B

Stack 1 Stack 2

A

B B

C C

D D

A



Top shelf

Bottom shelf

Slot 1

B

D D

Slot 2 Slot 3

Controller 1/A

A

C

B

D D

A

C

B

A

C

8 SAS cabling rules

A controller with one HBA is cabled in order of port 1d and then 1b, as shown below.

Stack cabling sequence: Controllers are cabled to the stacks in sequence of the stacks, such as stack 1, stack 2, stack 3, and so on.

20xx storage system connections: For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

Controller 1/A

Slot 1

B

D

A

C


Cabling SAS connections



◆ “Completing the SAS cabling worksheet” on page 10

◆ “SAS cabling worksheet” on page 12

◆ “Example worksheet” on page 12

◆ “Cabling SAS ports” on page 13

Completing the SAS cabling worksheet

By making a list of the SAS ports for each controller that need to be cabled to the disk shelf IOMs, you can simplify the cabling task of your storage system.

Step Action

1 Make a list of the top connections from the controller by listing all SAS HBA A ports and then all SAS HBA C ports on each controller.

You can use the “SAS cabling worksheet” on page 12 to write your information.

NoteIf you have an HA pair configuration, repeat this list for the second controller.

Examples:

A controller with three quad-port SAS HBAs is listed as 1a, 2a, 3a, 1c, 2c, 3c.

A controller with one quad-port SAS HBA is listed as 1a, 1c.

A controller with one dual-port SAS HBA is listed as 1a.

A 2040 controller is listed as 0a. 2040 system configurations only have a top connection to the disk shelves.

10 Cabling SAS connections

2 Make a list of the bottom connections from each controller.

You can use the “SAS cabling worksheet” on page 12 to write your information.

a. List the SAS HBA B ports and then all SAS HBA D ports.

Examples:

A controller with three quad-port SAS HBAs is listed as 1b, 2b, 3b, 1d, 2d, 3d.

A controller with one quad-port SAS HBA is listed as 1b, 1d.

A controller with one dual-port SAS HBA is listed as 1b.

b. Copy the first port in your list to the end of your list, and then cross out the first port in your list.

Offsetting the list by one (so that the bottom connection cabling begins on the second HBA) ensures system resiliency after you cable your storage system.

Examples:

A controller with three quad-port SAS HBAs is listed as 2b, 3b, 1d, 2d, 3d, 1b.

A controller with one quad-port SAS HBA is listed as 1d, 1b.

A controller with one dual-port SAS HBA is listed as 1b.

NoteIf you have an HA pair configuration, repeat this list for the second controller.

3 Go to “Cabling SAS ports” on page 13 to cable your storage system.

Step Action

1b, 2b, 3b, 1d, 2d, 3d, 1bXFor example:


SAS cabling worksheet

By filling in this worksheet as instructed in the “Completing the SAS cabling worksheet” on page 10, you identify the SAS ports for each controller that need to be cabled to the disk shelf IOMs.

Example worksheet This is an example of a completed worksheet for an HA pair configuration with three quad-port SAS HBAs and three stacks of disk shelves.

NoteThe grayed out SAS ports are unused because of the limited number of stacks in this configuration. If there were six stacks, all SAS ports would be used.

Connection type Controller SAS port cabling sequence Cable to disk shelf IOM

Top connections

C1/A To top shelf IOM A

C2/B To top shelf IOM B

Bottom connections

C1/A To bottom shelf IOM B

C2/B To bottom shelf IOM A

Connection type Controller SAS port cabling sequence Cable to disk shelf IOM

Top connections

C1/A 1a, 2a, 3a, 1c, 2c, 3c To top shelf IOM A

C2/B 1a, 2a, 3a, 1c, 2c, 3c To top shelf IOM B

Bottom connections

C1/A 2b, 3b, 1d, 2d, 3d, 1b To bottom shelf IOM B

C2/B 2b, 3b, 1d, 2d, 3d, 1b To bottom shelf IOM A


Cabling SAS ports To cable the SAS ports, complete the following steps.

Step Action

1 If you have more than one disk shelf in a stack, cable the shelf-to-shelf SAS connections; otherwise, go to Step 2.

Using the QSFP to QSFP SAS cables that came with your system:

a. Connect IOM A circle port to the next IOM A square port.

b. Connect IOM B circle port to the next IOM B square port.

NoteThe QSFP connectors are keyed and click into place when oriented correctly into a SAS port. Do not force a connector into a port.

2 Use your completed worksheet to cable the top connections for each controller.

NoteIf you are cabling a 2050 storage system, use the QSFP to mini-SAS cables for all controller to shelf connections; otherwise, use the QSFP to QSFP SAS cables.

Bottom shelf

Top shelf

ACP SAS ACP SAS

IOM A IOM B

Stack of three shelves


3 Use your completed worksheet to cable the bottom connections for each controller.

NoteFor quad-port SAS HBAs, because ports A and C never connect to the same stack of disk shelves and ports B and D never connect to the same stack of disk shelves, you might have unused HBA ports depending on the number of stacks in your configuration. For example, a single-controller with one quad-port SAS HBA and one stack of shelves only uses HBA ports A and D.

4 Verify the SAS connections by entering the applicable command:

◆ For 7-Mode, enter the following command at the system console:

sasadmin expander_map

For HA pair configurations, you can run this command on either node.

◆ For Cluster-Mode, enter the following command at the clustershell prompt:node run -node <node name> -command “sasadmin expander_map”

Result: The output shows information about SAS expanders attached to SAS channels on the controllers.

5 The next step depends on the output.

If... Then...

The output lists all of the IOMs The IOMs have connectivity. Go to Step 6.

Any of the IOMs are not shown - either the output will not show the IOM that is cabled incorrectly, or will not show all of the IOMs downstream from it.

Repeat Steps 1 through 5 to correct cabling errors.

6 The next step depends on whether you are using the ACP capability.

If... Then...

You are using the ACP capability Go to “Cabling ACP connections” on page 28.

You are not using the ACP capability Boot the storage system and begin setup.

Step Action


SAS cabling examples



NoteThe cabling examples show cabling to the logical top and logical bottom of a stack of disk shelves. Your physical stack arrangement might be different.

◆ “Examples of HA pair quad-port SAS HBA cabling” on page 15

◆ “Examples of single-controller quad-port SAS HBA cabling” on page 18

◆ “Examples of 2040 SAS cabling” on page 20

◆ “Examples of 2050 SAS cabling” on page 22

Examples of HA pair quad-port SAS HBA cabling

The following cabling examples show the SAS cabling sequence for HA pair configurations using quad-port SAS HBAs.

One quad-port SAS HBA and one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 1a; Bottom: 1d.

Stack 1

Controller 2/B

Ethernet portB

A

C

D

Slot 1

Controller 1/A

Ethernet portB

A

C

D

Slot 1

1. 2.

3.

4.

ACP SAS

IOM A IOM B

ACP SAS


Two quad-port SAS HBAs and two stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a; Bottom: 2b, 1d.

4.

1. 2.

5.

3.

6.

7.

8.

Controller 2/B

Ethernet portB

A

C

D

Slot 1

B

A

C

D

Slot 2

Controller 1/A

Ethernet portB

A

C

D

Slot 1

B

A

C

D

Slot 2


Stack 1 Stack 2


Top shelf

Bottom shelf

16 SAS cabling examples

Three quad-port SAS HBAs and three stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a; Bottom: 2b, 3b, 1d.

Four quad-port SAS HBAs and four stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a; Bottom: 2b, 3b, 4b, 1d.

Four quad-port SAS HBAs and eight stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 1c, 2c, 3c, 4c; Bottom: 2b, 3b, 4b, 1d, 2d, 3d, 4d, 1b.

Five quad-port SAS HBAs and five stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 5a; Bottom: 2b, 3b, 4b, 5b, 1d.

Five quad-port SAS HBAs and 10 stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 5a, 1c, 2c, 3c, 4c, 5c; Bottom: 2b, 3b, 4b, 5b, 1d, 2d, 3d, 4d, 5d, 1b.

Controller 2/BController 1/A

B

A

C

D

Slot 1

B

A

C

D

Slot 2

B

A

C

D

Slot 3

B

A

C

D

Slot 1

B

A

C

D

Slot 2

B

A

C

D

Slot 3

3.

1.4.

6.

2.

8.9.

7.

11.

12.


Stack 1 Stack 2


Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 3

IOM A IOM B

5. 10.


Examples of single-controller quad-port SAS HBA cabling

The following cabling examples show the SAS cabling sequence for single-controller configurations using quad-port SAS HBAs.

One quad-port SAS HBA and one stack of disk shelves: The SAS port cabling sequence is 1a, 1d.

B

A

C

D

Slot 1

Controller 1/A

Stack 1

1.

2.

Top shelf

Bottom shelf

IOM A IOM B

ACP SAS ACP SAS


One quad-port SAS HBA and two stacks of disk shelves: The SAS port cabling sequence is Top: 1a, 1c; Bottom: 1d, 1b.

Two quad-port SAS HBAs and two stacks of disk shelves: The SAS port cabling sequence is Top: 1a, 2a; Bottom: 2b, 1d.

Controller 1/A

B

A

C

D

2.

4.

3.

Slot 1

Stack 1 Stack 2



Top shelf

Bottom shelf

1.

1.

2.

4.

3.

Controller 1/A

Ethernet portB

A

C

D

Slot 1

B

A

C

D

Slot 2


Stack 1 Stack 2


Top shelf

Bottom shelf


Examples of 2040 SAS cabling

The following cabling examples show the SAS cabling sequence for 2040 HA pair and single-controller configurations.

NoteFor single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

NoteThe 2040 storage system’s onboard SAS port labeled 0d is treated as port 0a for cabling purposes. When the onboard port is treated as port 0a, all the SAS cabling rules can be applied.


2040 HA pair configuration with one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 0a.

2040 single-controller with one stack of disk shelves: The SAS port cabling sequence is Top: 0a.

Controller 1/A

0a

Controller 2/B

0a

1.

2040 configuration

0d 0d

Ethernet port

e0PEthernet port

e0P

Stack 1 2.

Top shelf

Bottom shelf

ACP SAS ACP SAS

IOM A IOM B

Controller 2/B

Ethernet port

0a

1.

2040 configuration

0d

e0P

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B


Examples of 2050 SAS cabling

The following cabling examples show the SAS cabling sequence for 2050 HA pair and single-controller configurations.

NoteFor single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

2050 HA pair configuration with one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 1a; Bottom: 1b.

Stack 1

Controller 1/A

Slot 1

BA

Ethernet port

Controller 2/B

Slot 1

B

A

Ethernet port

4.

3.

1. 2.

2050 configuration

Bottom shelf

Top shelf

ACP SAS ACP SAS

IOM A IOM B


2050 HA pair configuration with one disk shelf: The SAS port cabling sequence for both controllers is Top: 1a; Bottom: 1b.

Controller 1/A

Ethernet portB

A

Slot 1

Controller 2/B

Ethernet portB

A

Slot 1

4.

1.

3.

2.

2050 configuration

ACP SAS ACP SAS

IOM A IOM B

Stack 1


2050 single-controller configuration with one stack of disk shelves:

The SAS port cabling sequence is Top: 1a; Bottom:1b.

2050 single-controller configuration with one disk shelf: The SAS port cabling sequence is Top: 1a; Bottom: 1b.

Controller 2/B

Slot 1AB

1.

2.

2050 configuration

Stack 1

Top shelf

Bottom shelf

ACP SAS ACP SAS

IOM A IOM B

Controller 2/B

Ethernet portB

A

Slot 1

1.2.

2050 configuration

ACP SAS ACP SAS

IOM A IOM B

Stack 1


Chapter 3: ACP Cabling for New Storage Systems

3
ACP Cabling for New Storage Systems
About this chapter This chapter provides information about cabling the disk shelf ACP connections for a new storage system installation.

This chapter does not provide cabling information for hot-adding or replacing disk shelves. For those procedures, see the Installation and Service Guide at now.netapp.com for your disk shelf model.

NoteReferences to the top and bottom disk shelves in a stack are to the logical top and bottom of a stack. Your physical stack arrangement might be different.

Topics in this chapter

This chapter discusses the following topics:

◆ “ACP cabling rules” on page 26

◆ “Cabling ACP connections” on page 28

◆ “ACP cabling examples” on page 33

25

ACP cabling rules



◆ “General ACP cabling rules” on page 26

◆ “ACP connectivity rules” on page 26

General ACP cabling rules

These ACP cabling rules apply to all supported storage systems using SAS storage:

◆ You must use CAT6 Ethernet cables with RJ-45 connectors for ACP connections.

◆ If your storage system does not have a dedicated on board network interface for each controller, you must dedicate one for each controller at system setup. You can use a quad-port Ethernet card.

◆ All ACP connections to the disk shelf are cabled through the ACP ports, which are designated by a square symbol or a circle symbol.

ACP connectivity rules

All supported storage systems using the ACP capability are cabled in this way:

Shelf-to-shelf connections: All the disk shelves in a stack are daisy-chained when you have more than one shelf in a stack.

◆ IOM A circle port is connected to the next IOM A square port.

◆ IOM B circle port is connected to the next IOM B square port.

Intrastack connections: The bottom shelf and top shelf in each stack are connected to provide resiliency within each stack of disk shelves.

The bottom shelf IOM A circle port is connected to the top shelf IOM B square port. This applies to stacks containing one or more disk shelves.

Stack-to-stack connections: When you have two or more stacks of disk shelves, the last shelf in a stack is connected to the first shelf in the next stack until all stacks are connected.

The bottom shelf IOM B circle port of a stack is connected to the top shelf IOM A square port of the next stack.

Controller-to-stack connections: Each storage system controller is connected to each stack of disk shelves through a dedicated Ethernet port:

26 ACP cabling rules

◆ Controller 1/A always connects to the top shelf IOM A square port in a stack.

◆ Controller 2/B always connects to the bottom shelf IOM B circle port in a stack.

20xx storage system connections: For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

Chapter 3: ACP Cabling for New Storage Systems 27

Cabling ACP connections

Cabling ACP Any storage system supporting the ACP capability can be cabled using this cabling procedure.

Step Action

1 If you have more than one disk shelf in a stack, cable your shelf-to-shelf ACP connections; otherwise, go to Step 2.

Using the Ethernet cables that came with your system:

a. Connect each shelf IOM A circle port to the next shelf IOM A square port until all shelves in each stack are connected.

b. Connect each shelf IOM B circle port to the next shelf IOM B square port until all shelves in each stack are connected.

Daisy-chain connections

Top shelf

Bottom shelf

ACP SAS ACP SAS

IOM A IOM B

28 Cabling ACP connections

2 Cable the intrastack connection.

If... Then...

You have more than one disk shelf in a stack of disk shelves

Connect the bottom shelf IOM A circle port to the first shelf IOM B square port and repeat for all stacks.

You have only one disk shelf in a stack of disk shelves

Connect the IOMs within the disk shelf by connecting IOM A circle port to IOM B square of the disk shelf.

Step Action

Intrastack connection

Top shelf

Bottom shelf

ACP SAS ACP SAS

IOM A IOM B

Intrastack connection

ACP SAS ACP SAS

IOM A IOM B


3 If you have more than one stack of disk shelves, cable the stack-to-stack connections; otherwise, go to Step 4.

a. Beginning with Stack 1, connect the bottom shelf IOM B circle port to the next stack top shelf IOM A square port.

b. Repeat Step a until all stacks are connected.

Step Action

Stack-to-stack connections

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B

ACP SAS ACP SAS

Stack 2

IOM A IOM B

ACP SAS ACP SAS

Stack 3

IOM A IOM B


4 Cable controller-to-stack connections.

If... Then...

If your system has a controller 1/A Cable controller 1/A to the Stack 1 top shelf IOM A square port.

If your system has a controller 2/B Cable controller 2/B to the last stack bottom shelf IOM B circle port.

For example, an HA pair configuration with two stacks of disk shelves has the controller-to-stack connections cabled as follows:

Note20xx storage systems use the e0P port on each controller.

5 Verify that the ACP cabling is correct by entering the applicable command:

◆ For 7-Mode, enter the following command from the system console:

storage show acp

If you have an HA pair configuration, run this command on both nodes.

◆ For Cluster-Mode, enter the following command from the clustershell:node run -node local -command “storage show acp”

Step Action

Controller 2/B

Ethernet port

Controller 1/A

Ethernet port


Stack 1 Stack 2


Top shelf

Bottom shelf


6 The next step depends on what the command output is for each node.

If the result in the output shows ACP connectivity status... Then...

Full Connectivity ACP cabling is correct.

The number of shelf IOMs detected out-of-band (through ACP) and in-band (through SAS) are the same.

No Connectivity No ACP ports on the shelf IOMs are connected to the storage controllers.

Repeat Step 4.

Partial Connectivity Repeat Steps 1 through 6 to see what ACP connections you missed.

Fewer shelf IOMs are detected out-of-band (through ACP) than in-band (through SAS).

Additional Connectivity Repeat the “Cabling SAS connections” on page 10 to see what SAS connections you missed.

More shelf IOMs are detected out-of-band (through ACP) than in-band (through SAS).

7 Boot the storage system and begin setup.

See the Installation and Setup Instructions that came with your storage system.

NoteIf your storage system does not have a dedicated ACP on board Ethernet port on each controller for ACP, you will designate them at setup.

Step Action


ACP cabling examples



NoteThe cabling examples show cabling to the logical top and logical bottom of a stack of disk shelves. Your physical stack arrangement might be different.

◆ “Examples of HA pair configuration ACP cabling” on page 33

◆ “Examples of ACP cabling for all systems except 20xx” on page 35

◆ “Examples of 20xx single-controller ACP cabling” on page 36

Examples of HA pair configuration ACP cabling

The following cabling examples show ACP cabling for all supported HA pair configurations.

HA pair configuration with one disk shelf:

Controller 2/B

Ethernet port

Controller 1/A

Ethernet port

Top shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B


HA pair configuration with two stacks of disk shelves:

HA pair configuration with three stacks of disk shelves:

Controller 1/A

Ethernet port

Controller 2/B

Ethernet port


Stack 1 Stack 2


Top shelf

Bottom shelf

Controller 1/A

Ethernet port

Stack 2Stack 1 Stack 3

Controller 2/B

Ethernet port

ACP SAS ACP SAS

IOM A IOM B

ACP SAS ACP SAS

IOM A IOM B

ACP SAS ACP SAS

IOM A IOM B

Top shelf

Bottom shelf

34 ACP cabling examples

Examples of ACP cabling for all systems except 20xx

The following cabling examples show ACP cabling for all supported storage systems other than 20xx systems.

Single-controller configuration with one stack of disk shelves (except the 20xx system):

Single-controller configuration with two stacks of disk shelves (except the 20xx system):

Controller 1/A

Ethernet port

Non 20xx configuration

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B

Controller 1/A

Ethernet port

Non 20xx configuration

ACP SAS ACP SAS

Stack 2

IOM A IOM B

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B


Examples of 20xx single-controller ACP cabling

The following cabling examples show ACP cabling for 20xx single-controller configurations.

NoteFor single-controller 20xx configurations, the controller is in Slot B (the bottom slot); therefore, the controller needs to be cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

20xx single-controller configuration with one stack of disk shelves:

Controller 2/B

Ethernet port

20xx configuration

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B


20xx single-controller configuration with two stacks of disk shelves:

Controller 2/B

Ethernet port

20xx configuration

ACP SAS ACP SAS

Stack 2

IOM A IOM B

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B



Appendix A: Terms and Symbols

A
Terms and Symbols
About this appendix This appendix defines terms and port icon symbols particular to SAS disk shelves and also includes common terms about connectivity.

The terms are organized by relationship to each other instead of alphabetical order.

List of terms and symbols

You should be familiar with the following terms:

◆ Square and circle symbols:

The SAS ports and ACP ports on the disk shelf IOMs are designated by square and circle symbols. All cabling is done in reference to connecting to a square port or a circle port.

NoteSAS connectivity does not use the concept of In ports and Out ports.

◆ IOM:

The SAS shelf I/O module is located in the back of the disk shelf. It connects the individual disk drives to the rest of the storage system and controls the disk shelf operator display panel LEDs. Each disk shelf has two IOMs: IOM A and IOM B. These are also referred to as SBB A and SBB B on the slot map label on the back of the disk shelf.

◆ SAS stack:

Also referred to as stack. A group of one or more SAS disk shelves connected (daisy-chained) together and connected to the controller through the top disk shelf in the stack and the bottom disk shelf in the stack (as needed). The maximum number of disk shelves in a stack of disk shelves and the number of disk shelf stacks in a configuration are dependent on the type of storage system.

◆ ACP:

Alternate Control Path is a protocol that enables Data ONTAP to manage and control the disk shelf storage subsystem. It uses a separate network from the data path, so management communication is not dependent on the data path being intact and available. Use of ACP requires that all disk shelf IOMs and storage system controllers connect through the ACP ports on the IOMs and the designated network interface on each controller.

◆ QSFP:

The standard SAS cable connector for all NetApp® SAS systems. QSFP to QSFP SAS cables are used to daisy-chain SAS disk shelf ports and connect

39

to the quad-port SAS host bus adapter (HBA) or onboard SAS ports. QSFP to mini-SAS cables are used to connect disk shelves to the dual-port mini-SAS HBA.

◆ Software-based disk ownership:

Software-based disk drive ownership is stored on the disk drive rather than determined by the topology of the storage system’s physical connections. It gives you increased flexibility and control over disk drive use. SAS and SATA disk drives use software-based disk drive ownership.

Therefore, the hardware-based disk ownership concept that the local node owns all direct-attached disk drives connected to it through the A channel (the loop attached to the A module on the disk shelf) and its partner owns the disk drives connected to the local node on the B channel, does not apply.

This also means that the hardware-based disk drive ownership terms do not apply: primary path, standby path, redundant path, and redundant standby path. Instead, connectivity between the controller and SAS disk shelf is addressed in terms of: top connections and bottom connections.

◆ Top connection:

The cabling connection from the controller to the top disk shelf in a stack of disk shelves.

References to the top disk shelf in this document are to the logical top of a stack of disk shelves. Your physical disk shelf stack might be arranged differently.

◆ Bottom connection:

The cabling connection from the controller to the bottom disk shelf in a stack of disk shelves.

References to the bottom disk shelf in this document are to the logical bottom of a stack of disk shelves. Your physical disk shelf stack might be arranged differently.

◆ Shelf-to-shelf (daisy-chain):

The cabling connection between disk shelves in a stack of more than one disk shelf. Each stack of disk shelves is daisy-chained through its SAS ports and, if you are using the ACP capability, each disk shelf is also daisy-chained through its ACP ports.

◆ Controller:

The component of your storage system that runs Data ONTAP® software and manages the storage subsystem.

The controller located in the top slot in a storage system chassis can be referred to as Controller 1 or Controller A. The controller located in the bottom slot in a storage system chassis can be referred to as Controller 2 or Controller B.

40 Terms and Symbols

In this document controllers are referred to as Controller 1/A and Controller 2/B.

◆ Single-controller configuration:

A storage system having one controller.

◆ HA pair configuration:

A storage system having two controllers. This can be two controllers in one chassis or it can be two controllers, each in a separate chassis.. The two controllers serve data for each other if one of the controllers becomes impaired.

◆ Multipath HA:

Each controller in the HA pair configuration has multiple ways to connect to a disk drive. Multipath HA cabling is the most resilient and preferred configuration for HA pair configurations. This is because it takes full advantage of the resiliency capability of SAS, which means that the storage system continues to have access to disk drives in the event of cable, HBA, and IOM failure. A single failure of a cable, HBA, or IOM does not result in a controller failover.

◆ Dual-path:

The single-controller storage system has two ways to connect to a disk drive. This is the preferred configuration for single-controller configurations.

◆ Single-path HA:

Each controller in the HA pair configuration has one way to connect to a disk drive. This means an IOM or cable failure requires a controller failover.

◆ Single-path:

The single-controller storage system has one way to connect to a disk drive. This configuration has multiple single points of failure and is not a preferred configuration.

◆ Configured systems:

A new storage system that ships with SAS disk shelves and preinstalled SAS HBAs (if applicable).

◆ Add-on disk shelves:

SAS disk shelves that are shipped individually—not shipped as part of a configured system.

Appendix A: Terms and Symbols 41

42 Terms and Symbols

Appendix B: Document Update Record

B
Document Update Record
Document update history

The following table lists and describes the history of changes made to this document. Changes are not dictated by Data ONTAP releases.

Feature updatesFeature release date

◆ Initial release of the SAS and ACP cabling information was part of the DS4243 Installation and Service Guide.

September 2009

◆ Initial release of the SAS and ACP cabling information as a stand-alone document and updates.

❖ The SAS disk shelf illustrations are now generic so that they apply to any NetApp SAS disk shelf. A generic disk shelf is shown as two IOMs side-by-side (no power supplies are shown).

❖ HBAs (1, 2, 3 etc.) are now referred to as slots (1, 2, 3 etc.)

❖ Introduces the new rule for referring to onboard SAS ports as belonging to slot 0 and the 2040 system exception - the single onboard port is 0a.

❖ Introduces the new rule that for single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

❖ The term active/active is updated to HA pair throughout.

❖ The term Multipath is updated to Multipath HA.

September 2010

43

44 Document Update Record

Index

AACP cabling procedure 28ACP capability 26audience v

Ccabling ACP

examples 36procedure 28verifying ACP connections 31

cabling SASexamples 18rules 4verfiying SAS connections 14worksheet 12

worksheet procedure 10conventions

formatting v

Fformatting conventions v

Sspecial messages vi

Tterminology 39

Index 45

46 Index

Documents

SAS Cabling