HSI Copy Offload WP-r12

7/30/2019 HSI Copy Offload WP-r12

1/44

Copy Offload Hypervisor Storage Interfaces

1 of 442010 STORAGE NETWORKING INDUSTRY ASSOCIATION

Revision 1.0

Hypervisor Storage Interfacesfor Storage Optimization White Paper

June 2010


2/44



Revision 1.0

The SNIA hereby grants permission for individuals to use this document for personal use only, andfor corporations and other business entities to use this document for internal use only (includinginternal copying, distribution, and display) provided that:

1. Any text, diagram, chart, table or definition reproduced must be reproduced in its entiretywith no alteration, and,

2. Any document, printed or electronic, in which material from this document (or any portionhereof) is reproduced must acknowledge the SNIA copyright on that material, and mustcredit the SNIA for granting permission for its reuse.

Other than as explicitly provided above, you may not make any commercial use of this document,sell any or this entire document, or distribute this document to third parties. All rights not explicitlygranted are expressly reserved to SNIA.

Permission to use this document for purposes other than those enumerated above may berequested by e-mailing [email protected] please include the identity of the requesting individualand/or company and a brief description of the purpose, nature, and scope of the requested use.

Copyright 2010 Storage Networking Industry Association.


3/44



Revision 1.0

Table of Contents

1 Hypervisor Storage Interfaces.............................................................................................. 71.1 Terms and Concepts ................................................................................................................................ 71.2 Hypervisor Storage ................................................................................................................................... 71.3 Hypervisor Storage Interfaces (HSI)...................................................................................................... 81.4 Offload Use Cases ..................................................................................................................................... 8

2 Copy Offload Model ............................................................................................................... 92.1 Terms and Concepts ................................................................................................................................ 92.2 Overview ..................................................................................................................................................... 92.2.1 Scope of Copy Managers .................................................................................................................. 102.2.2 Copy Manager Identifiers .................................................................................................................. 102.3 Example Operations............................................................................................................................... 122.3.1 External Copy Operation ................................................................................................................. 122.3.2 Internal to External Copy Operation ............................................................................................ 142.3.3 External to Internal Copy Operation ............................................................................................ 162.3.4 Internal Copy Operation .................................................................................................................. 182.3.5 IP Copy Service Operation .............................................................................................................. 192.4 Reservation Considerations ................................................................................................................. 202.5 General Considerations ........................................................................................................................ 212.6 Discovery .................................................................................................................................................. 212.6.1 EXTENDED COPY command discovery...................................................................................... 212.6.2 COMPARE AND WRITE command discovery ........................................................................... 222.6.3 UNMAP operation discovery .......................................................................................................... 222.6.4 UNMAP command discovery .......................................................................................................... 232.6.5 WRITE SAME (16) command discovery ....................................................................................... 242.6.6 Logical unit discovery requirement ................................................................................................ 252.7 Target Descriptors ................................................................................................................................. 252.7.1 Identification target descriptors ...................................................................................................... 252.7.2 Port type descriptors for Fibre Channel Targets ........................................................................ 252.7.3 Port type descriptors for iSCSI Targets ........................................................................................ 25 2.7.4 Port type descriptors for SAS targets ........................................................................................... 252.7.5 Copy Service Descriptor .................................................................................................................. 252.8 Completion .............................................................................................................................................. 25

3 Security Considerations ...................................................................................................... 264 Use Sequence ....................................................................................................................... 27

4.1 Sequence Diagram .................................................................................................................................. 274.2 Gather Copy Target Descriptor Information .................................................................................. 294.2.1 Method 1 Identification Descriptor (E4h) ................................................................................. 294.2.2 Method 2 N_Port_Name target descriptor (E0h) .................................................................. 294.2.3 Method 3 N_Port_ID target descriptor (E1h not recommended) ................................. 29


4/44



Revision 1.0

4.2.4 Method 4 N_Port_ID with N_Port_Name checking target descriptor(E2h non-fabric environments only) ........................................................................................... 30

4.2.5 Method 5 iSCSI target descriptors (E5h and EAh) .................................................................. 30 4.2.6 Method 6 IP COPY Service descriptor (EBh) .......................................................................... 304.2.7 Method 7 SAS Serial SCSI Protocol target descriptor (E9h) ................................................ 304.3

Gather Segment Descriptor Information (02h) ............................................................................... 31

4.4 Build the Extended Copy Parameter list ........................................................................................... 314.5 Copying a Virtual Disk ........................................................................................................................... 314.6 Copying an Entire Logical Unit ............................................................................................................ 34 4.6.1 CDB and Parameter Data Details................................................................................................... 364.7 Copying a Virtual Machine .................................................................................................................... 374.8 Virtual Machine Storage Migration...................................................................................................... 384.9 Dividing Virtual Machines ...................................................................................................................... 404.10 Reclaiming Unused Space.................................................................................................................. 424.10.1 Reclaiming space overview ............................................................................................................... 424.10.2 Reclaiming space on virtual disk deletion ...................................................................................... 424.10.3

Reclaiming Unused Space on behalf of a guest ............................................................................ 43

4.11 Atomic operation offload ................................................................................................................. 44

5 References ............................................................................................................................ 44

List of TablesTable 1 Single Parameter List Example ............................................................................................................... 33Table 2 Parameter List to copy Segment 1 ........................................................................................................ 33Table 3 Parameter List to copy Segment 2 ........................................................................................................ 34Table 4 Parameter List to copy Segment 3 ........................................................................................................ 34Table 5 Whole LUN copy (single segment descriptor) .................................................................................. 35Table 6 Whole LUN copy (multiple segment descriptors) ............................................................................ 36Table 7 Detailed CDB ............................................................................................................................................. 36Table 8 Detailed Parameter Data Example ........................................................................................................ 37Table 9 Copy of multiple segments of a LUN ................................................................................................... 38

List of FiguresFigure 1 Nested Perceived Logical Units .............................................................................................................. 8Figure 2 External Copy Offload Operation ....................................................................................................... 12Figure 3 Internal to External Copy Offload Operation

................................................................................... 14Figure 4 External to internal Copy Offload Operation ................................................................................... 16

Figure 5 Internal Copy Offload Operation ......................................................................................................... 18Figure 6 IP Copy Service Operation .................................................................................................................... 19Figure 7 Copy Manager Sequence Diagram ....................................................................................................... 28Figure 8 Example Copy of a Virtual Disk in a Container ................................................................................ 32


5/44



Revision 1.0

Figure 9 Example of Copying an Entire Logical Unit ........................................................................................ 35Figure 10 Simple example of dividing virtual machine ..................................................................................... 41Figure 11 Reclaiming Blocks .................................................................................................................................. 42


6/44



Revision 1.0

About the SNIAThe Storage Networking Industry Association (SNIA) is a not-for-profit global organization, made up ofsome 400 member companies and 7,000 individuals spanning virtually the entire storage industry.SNIA's mission is to lead the storage industry worldwide in developing and promoting standards,technologies, and educational services to empower organizations in the management of information.

To this end, the SNIA is uniquely committed to delivering standards, education, and services that willpropel open storage networking solutions into the broader market. For additional information, visit theSNIA web site at www.snia.org.

PrefaceThis document is aimed at Hypervisor software providers and providers of storage intended for usewith Hypervisor environments. It uses Hypervisor examples to describe the use of commands fromthe SCSI T10 Standards for performance and utilization enhancements such as copy offload, atomicsequence offload, and space reclamation technology in a blocks environment. This includes using theSCSI EXTENDED COPY command for providing copy offload services, the SCSI COMPARE ANDWRITE command for atomic sequence offload, and SCSI unmap operations for space reclamation

services.

About the SNIA HSI Technical Working GroupThe SNIA Hypervisor Storage Interfaces (HSI) Technical Working Group is a multi-vendor group ofHypervisor vendors, Storage vendors and System vendors. The goal of the group is to define commonmethods for interaction between hypervisor software and storage devices.

AcknowledgementsWe would like to take this opportunity to acknowledge the many contributions that contributed manyhours, dedication, and persistence to make this document possible.

The SNIAs HSI TWG


7/44



Revision 1.0

1 Hypervisor Storage Interfaces1.1 Terms and ConceptsVirtual Disk A set of disk blocks presented to an operating environment as a range of consecutivelynumbered logical blocks with disk-like storage and I/O semantics. In the context of this document, avirtual disk is defined as a set of disk blocks presented by a hypervisor to a guest operatingenvironment as a range of consecutively number logical blocks with a disk-like storage and I/Osemantics.

Virtual Machine (VM) A representation of a physical machine that is used by the guest operatingsystem as the hardware platform on which it executes. A virtual machine is comprised of one or morevirtual disks and associated meta-data.

Hypervisor Software that provides virtual machine environments which are used by guest operating

systems.

Perceived logical unit A unit of storage as seen by the hypervisor.

1.2 Hypervisor StorageHypervisor Storage refers to a storage abstraction layer that is used by a Hypervisor to create VirtualDisks in the Virtual Machine. It is an intermediate storage abstraction layer located between the guestoperating system storage stack running in a VM and the perceived logical units. From the perspective ofa guest operating system executing within a VM, a virtual disk is perceived as a logical unit, and isgenerally indistinguishable from a physical disk. These Virtual Disks are in turn comprised of a

collection of blocks that are located on perceived logical units. Virtual Disks are encapsulated,meaning that they are a self-contained, self-describing resource to the VM. Virtual Disks may berepresented on physical storage arrays as Files, Blocks or Objects. The address space of a virtual disk(i.e., the blocks that comprise the virtual disk) appear to the guest operating system as a contiguousaddress space. These contiguous addresses are mapped by the hypervisor storage layer to theperceived logical unit and may be located on different physical devices/granularities at the discretion ofthe hypervisor storage layer.

Virtual Disks are completely abstracted by the Hypervisor storage layer. There is no visibility into thecomposition of a given virtual disk by the storage array(s) on which the blocks are stored. In addition,the guest operating system has no visibility into the physical storage array that hosts a virtual disk.

Multi-layer, or nested virtualization, may also occur. For example, a guest operating system may itselfbe a hypervisor that uses its perceived logical units to create virtual disks for guest operating systemsof its own. An example of this is shown in Figure 1.


8/44



Revision 1.0

Figure 1 Nested Perceived Logical Units

1.3 Hypervisor Storage Interfaces (HSI)To assist with the management and operation of Hypervisor Storage, the SNIA HSI TWG is clarifyingthe use of, extending, and/or developing standard Hypervisor Storage Interfaces for storage

interactions with hypervisors. These interfaces enable the hypervisor to offload certain storageintensive tasks to the underlying storage systems to achieve performance and functional improvements.

It is the purpose of this document to describe use cases where it is desirable to have the storage arrayperform offload operations and/or have some visibility into virtual disk constructs while maintaining thenecessary and appropriate abstractions and isolation.

The first revision of this document focuses on optimizing copy operations and space reclamationoperations in block storage environments.

1.4 Offload Use CasesThe offload use cases addressed in this first revision include:

Copying a Virtual Disk: Create a copy of an individual Virtual Disk with no burden on theserver/hypervisor to optimize hypervisor performance.

Copying an entire Logical Unit: Create a copy of an entire Logical Unit (which may contain oneor more virtual disks) with no burden on the server/hypervisor to optimize hypervisorperformance.

Copying a Virtual Machine: Create a copy of a Virtual Machine (which may involve multiplelogical units) with no burden on the server/hypervisor to optimize hypervisor performance.

Virtual Machine Storage Migration: Leverage array-based copy/replication function to non-

disruptively migrate the storage for an entire Virtual Machine from one location to another. Dividing Virtual Machines: Enable the non-disruptive split of a Virtual Machine set by providing a

single procedure that simultaneously relocates a subset of virtual machines from one storagelocation to another.

Reclaiming Unused Space After Virtual Disk Deletion: Offload to the array the reclaiming ofblocks after a Virtual Disk is deleted,

VMPerceivedLogical unit

Virtual

Disk

Hypervisor GuestPerceived

logical unit

Array


9/44



Revision 1.0

Reclaiming Unused Space on behalf of a guest: Offload to the array the reclaiming of blockswhen a guest operating system deletes a file or otherwise no longer needs to retain a particularset of data.

2 Copy Offload Model2.1 Terms and ConceptsCopy Manager The entity within a logical unit that manages extended copy operations. The copymanager receives the copy request and performs the necessary operations to execute a copyoperation.

IP Copy Service An IP entity outside the SCSI device server that communicates with the copymanager and performs the necessary operations to transfer data through an IP network.

Copy Source Device The SCSI logical unit from which data is read.

Copy Destination Device The SCSI logical unit to which the data is written.

Copy Target Device Either a copy source device or a copy destination device.

Target Descriptor A data structure used to describe a copy target device.

Segment Descriptor A data structure that describes a segment (as a starting logical block address andlength) to be transferred, and provides an indication of the copy target devices involved in the transfer.

Relative Port Identifier The unique identifier of the port through which the copy manager accesses anexternal logical unit.

2.2 OverviewCopy offload operations are performed by a SCSI copy manager. A copy manager is a process in aSCSI logical unit that accepts copy offload requests from a host system, and performs the requestedtransfers on behalf of the requesting host. The host uses the SCSI EXTENDED COPY command (seeSPC-4) to specify the details of the copy request to the copy manager.

The EXTENDED COPY command is one of the most complex commands in the SCSI command set.

The EXTENDED COPY command contains at a minimum:

a. an identifier of a device from which data is read (a target descriptor);b. an identifier of a device to which data is written (a target descriptor); andc. a description of the data to copy (a segment descriptor).


10/44



Revision 1.0

The complexity of this command results from the multitude of options available through this command(e.g., eleven (11) different types of target descriptors are available, and twenty two (22) different typesof segment descriptors are available, creating a huge number of possible combinations). In addition,multiple copy operations may be described in a single command (multiple segments may be copied

from multiple sources to multiple destinations). This document focuses on the most commoncombinations of these descriptors and how they are used to provide copy offload services forhypervisors.

2.2.1 Scope of Copy ManagersSome copy managers may have the ability to provide copy offload services only for blocks on logicalunits that are contained within the same array that contains the copy manager itself (intra-array copyoffload).

Some copy managers may have the ability to provide copy offload services for blocks on logical units

that span multiple arrays (inter-array copy offload) as well as the ability to provide intra-array copyoffload.

Some copy managers have the ability to use TCP/IP protocols to perform inter-array copy offload usingan IP copy service.

While, copy managers are permitted to provide copy offload services exclusively for logical units thatspan multiple arrays (inter-array copy offload only), these types of copy managers are rare.

This document uses the term internal when referencing a logical unit that is within the same arraythat contains the copy manager. This document uses the term external when referencing a logical

unit that is not within the same array that contains the copy manager. These terms are not used in theSPC-4 standard where the EXTENDED COPY command is described.

2.2.2 Copy Manager IdentifiersInternal logical units are typically identified using a target descriptor known as an Identificationdescriptor target descriptor (type E4h in SPC-4). This type of descriptor contains a designator that ismatched with a designation descriptor in a logical units Device Identification VPD page (page 83h). Thelogical unit with the matching designation descriptor is used for the copy operation.

Internal logical units may also be identified using target descriptors (e.g., types E0h-E3h, E5h-EAh)

containing port identifiers and the logical unit number (or LU Identifier). In this case however, the portidentifier in the target descriptor must match a port identifier of a port within the array that is allowedaccess to the specified internal logical unit. Due to the additional complexity of this type of identifier,use of target descriptors of this type for internal logical units is not recommended.

External logical units may be identified in three ways:


11/44



Revision 1.0

a. Using an Identification descriptor target descriptor (type E4h)b. Using a port type target descriptor (e.g., type E0h-E3h, E5h,-EAh)c. Using an IP copy service target descriptor (type EBh).

While other port type target descriptors may be used, this document discusses the following port typetarget descriptors:

a. N_PORT_NAME (i.e., type E0h),b. N_PORT_ID (i.e., type E1h),c. N_PORT_NAME and N_PORT_ID (i.e., type E2h),d. IPv4 address (i.e., type E5h),e. SAS address (i.e., type E9h),f. IPv6 address (i.e., type EAh).

When using a port type target descriptor, the host specifies the identifier (e.g., N_PORT_NAME,

N_PORT_ID, IPv4 address, IPv6 address. SAS address) of the port on the external array that is to beused to access the specified logical unit number (LU Identifier). The host may optionally specify whichport (i.e., the relative port identifier) on the array containing the copy manager to use to initiate theoperation with the external array. The host has the ability to totally control which path to use for thecopy operation. The copy manager performs just the operations specified by the host.

When using an identification descriptor target descriptor, the host specifies the designator that is to bematched with a designator in the Device Identification VPD page of the logical unit selected for theoperation. The host may optionally specify which port (i.e., the relative port identifier) on the arraycontaining the copy manager to use to initiate the operation with the external array, but the hostcannot specify which port on the external array to use for the copy operation. With the identification

descriptor target descriptor, the copy manager is responsible for examining the external logical unitsto which it has access, and finding a logical unit that has a Device Identification VPD page designatorentry that matches the designator specified by the host. This examination of external logical units tofind a matching Device Identification VPD page entry requires additional operations from the copymanager beyond just the copy operation requested by the host.

When using an IP copy service target descriptor, the host specifies the IP address of the copy serviceindicated by the external logical unit in its Management Network Addresses VPD page (85h), and adesignator that is to be matched with an entry in the Device Identification VPD page of the logical unitselected for the operation. The host cannot specify any portion of the path used for communicationwith the external array. The IP copy service is responsible for examining external logical units to

which it has access, and finding a logical unit that has a Device Identification VPD page entry thatmatches the designator specified by the host. The copy manager performs just the operations specifiedby the host.


12/44



Revision 1.0

2.3 Example OperationsFigure 2 shows an example of a fully external copy offload operation. Figure 3 shows an example of acopy offload from an internal logical unit to an external logical unit. Figure 4 shows an example of acopy offload operation from an external logical unit to an internal logical unit. Figure 5 shows a copy

offload between logical units within the same array as the copy manager. Figure 6 shows a copy offloadusing an IP copy service to perform the transfers over an IP network connected logical unit to aninternal logical unit.

2.3.1 External Copy Operation

Figure 2 External Copy Offload Operation

When a host wants to copy blocks from an external source device to an external destination device(see Figure 2) these procedures are used. To perform copy offload, one or more target descriptorsand one or more segment descriptors are created to have the copy manager perform this operationon behalf of the requesting host:

A. A source target descriptor containing:

either:a. the relative port identifier of the port over which to issue the READ command (RP3)(or zero);

b. the port identifier (see 2.2.2) of the port on the source device to process the READcommand (TP1); and

c. the logical unit number of the logical unit from which to read the data.

TP4TP3TP2TP1

Hypervisor

Host

Copy Manager

CopySource

Device

CopyDestination

Device

RP1

RP3

RP2

RP4

EXTENDEDCOPY

Command

READ WRITE


13/44



Revision 1.0

ora. the relative port identifier of the port over which to issue the READ command (RP3)

(or zero);b. an identification descriptor target descriptor containing the Device Identification

designator of the logical unit from which the data is to be read.

B. A destination target descriptor containing:either:a. the relative port identifier of the port over which to issue the WRITE command (RP4);b. the port identifier (see 2.2.2) of the port on the copy destination device to process the

WRITE command (TP4); andc. the logical unit number of the logical unit to which the data is written.

ora. the relative port identifier of the port over which to issue the WRITE command (RP3)

(or zero);b. an identification descriptor target descriptor containing the Device Identification

designator of the logical unit to which the data is to be written.

C. Segment descriptor(s) containing:a. the number of logical blocks to be transferred;b. the starting LBA on the source device to begin the READ operation; andc. the starting LBA on the destination device to begin the WRITE operation.

If an identification descriptor target descriptor is used, the copy manager is responsible for determiningthe location of the logical unit with the matching designator. This may involve the copy managermaintaining information about accessible logical units and the paths (e.g., which relative ports and/ortarget ports) via which those logical units may be accessed.

The EXTENDED COPY command is sent to the logical unit containing the copy manager (via RP1).

The copy manager then connects through RP3 to TP1 and sends appropriate READ commands (usingthe starting LBA and length from the segment descriptor) to the specified logical unit. The copymanager then connects through RP4 to TP4 and sends appropriate WRITE commands (using thestarting LBA and length from the segment descriptor) to the specified logical unit.

The data has now been copied from the copy source device to the copy destination device without anymovement of data through the host.


14/44



Revision 1.0

2.3.2 Internal to External Copy Operation

Figure 3 Internal to External Copy Offload Operation

When a host wants to copy blocks from an internal source device to an external destination device(see Figure 3) these procedures are used. To perform copy offload, one or more target descriptors,and one or more segment descriptors are created to have the copy manager perform this operationon behalf of the requesting host:

A. A source target descriptor containing:either:

a. a relative port identifier of zero;b. the port identifier (see 2.2.2) of a port on the source device that has access to the

logical unit from which to read the data (RP1, RP2, RP3, or RP4); andc. the logical unit number of the logical unit from which to read the data.

ora. a relative port identifier of zero;b. an identification descriptor target descriptor containing the Device Identification

designator of the logical unit from which to read the data.

B. A destination target descriptor containing:either:

a. the relative port identifier of the port over which to issue the WRITE command (RP4)(or zero);

TP4TP3

Hypervisor

Host

Copy Manager

Copy

SourceDevice

CopyDestination

Device

RP1

RP3

RP2

RP4

EXTENDEDCOPY

Command

WRITE

READ


15/44



Revision 1.0

b. the port identifier (see 2.2.2) of the port on the destination device to process the writecommand (TP4); and

c. the logical unit number of the logical unit to which the data is to be written.or

a. a relative port identifier of the port over which to issue the WRITE command (RP4)

(or zero);b. an identification descriptor target descriptor containing the Device Identificationdesignator of the logical unit to which the data is to be written.

C. Segment descriptor(s) containing:a. the number of logical blocks to be transferred;b. the starting LBA on the source device to begin the READ operation; andc. the starting LBA on the destination device to begin the WRITE operation.

If the destination target descriptor is an identification descriptor target descriptor, then the copymanager is responsible for determining the location of the logical unit with the matching designator.This may involve the copy manager maintaining information about accessible logical units and the paths

(i.e., the relative ports and/or target ports) via which those logical units may be accessed.

The EXTENDED COPY command is sent to the logical unit containing the copy manager (via RP1).The copy manager then reads from the specified local logical unit (using the starting LBA and lengthfrom the segment descriptor). The copy manager then connects through RP4 to TP4 and sendsappropriate WRITE commands (using the starting LBA and length from the segment descriptor) to thespecified logical unit.



16/44



Revision 1.0

2.3.3 External to Internal Copy Operation

Figure 4 External to internal Copy Offload Operation

When a host wants to copy blocks from an external source device to an internal destination device(see Figure 4) these procedures are used.

A. A source target descriptor containing:either:

a. the relative port identifier of the port over which to issue the READ command (RP3)(or zero);

b. the port identifier (see 2.2.2) of the port on the source device to process the readcommand (TP1); and

c. the logical unit number of the logical unit from which to read the data.or

a. the relative port identifier of the port over which to issue the READ command (RP3)(or zero);

b. an identification descriptor target descriptor containing the Device Identification

designator of the logical unit from which to read the data.B. A destination target descriptor containing:

either:a. a relative port identifier of zero;

TP2TP1

Hypervisor

Host

Copy Manager

CopySource

Device

CopyDestination

Device

RP1

RP3

RP2

RP4

EXTENDEDCOPY

Command

READ

WRITE


17/44



Revision 1.0

b. the port identifier (see 2.2.2) of a port on the destination device that has access to thelogical unit to which the data is to be written (RP1, RP2, RP3, or RP4); and

c. the logical unit number of the logical unit from which to read the data.or

a. a relative port identifier of zero;

b. an identification descriptor target descriptor containing the Device Identificationdesignator of the logical unit to which the data is to be written.C. Segment descriptor(s) containing:

a. the number of logical blocks to be transferred;b. the starting LBA on the source device to begin the READ operation; andc. the starting LBA on the destination device to begin the WRITE operation.

If the source target descriptor is an identification descriptor target descriptor, then the copy manageris responsible for determining the location of the logical unit with the matching designator. This mayinvolve the copy manager maintaining information about accessible logical units and the paths (i.e., therelative ports and/or target ports) via which those logical units may be accessed.

The EXTENDED COPY command is sent to the logical unit containing the copy manager (via RP1).The copy manager then connects through RP3 to TP1 and sends appropriate READ commands (usingthe starting LBA and length from the segment descriptor). The copy manager then writes to thespecified internal logical unit (using the starting LBA and length from the segment descriptor) to thespecified logical unit.



18/44



Revision 1.0

2.3.4 Internal Copy Operation

Figure 5 Internal Copy Offload Operation

When a host wants to copy blocks from an internal source logical unit to another logical unit internalto the same array (see Figure 5). To perform copy offload, multiple descriptors are created to havethe copy manager perform this operation on behalf of the requesting host:

A. A source target descriptor containing:

a. a relative port identifier of zero;b. an identification descriptor target descriptor containing the Device Identification

designator of the logical unit from which to read the data.B. A destination target descriptor containing:

a. a relative port identifier of zero;b. an identification descriptor target descriptor containing the Device Identification

designator of the logical unit to which the data is to be written.C. Segment descriptor(s) containing:

a. the number of logical blocks to be transferred;b. the starting LBA on the source device to begin the READ operation; andc. the starting LBA on the destination device to begin the WRITE operation.

If the copy source device and the copy destination device are the same logical unit, then a single targetdescriptor may be used, with the segment descriptor referencing the same descriptor for both thesource and the destination.

Hypervisor

Host

Copy Manager

CopySource

Device

Copy

Destination

Device

RP1

RP3

RP2

RP4

EXTENDEDCOPY

CommandREAD

WRITE


19/44



Revision 1.0

The EXTENDED COPY command is sent to the logical unit containing the copy manager (via RP1).The copy manager then reads from the specified internal logical unit (using the starting LBA and lengthfrom the segment descriptor). The copy manager then writes to the specified internal logical unit(using the starting LBA and length from the segment descriptor).


2.3.5 IP Copy Service Operation

Figure 6 IP Copy Service Operation

When a host wants to copy blocks from an external source device to an internal destination device(see Figure 6) using an IP copy service, these procedures are used. To perform copy offload, multipledescriptors are created to have the copy manager perform this operation on behalf of the requestinghost:

A. An IP Copy Service target descriptor containing:a. the IP address of the copy service obtained from the Management Network Address

VPD page (85h) of the source logical unit;b. an identification description designator containing the Device Identification designator

of the logical unit from which to read the data.B. A destination target descriptor containing:

a. a relative port identifier of zero;

TP1

IP CopyService

Hypervisor

Host

Copy Manager

CopySourceDevice

RP1

RP2

EXTENDEDCOPY

Command

IP Network

CopyDestination

DeviceWRITE

READ

DataTransfer


20/44



Revision 1.0

b. an identification description designator containing the Device Identification designatorof the logical unit to which the data is to be written.

C. Segment descriptor(s) containing:a. the number of logical blocks to be transferred;b. the starting LBA on the source device to begin the READ operation; and

c. the starting LBA on the destination device to begin the WRITE operation.

The EXTENDED COPY command is sent to the destination logical unit. The copy manager suppliesthe necessary information from the EXTENDED COPY command that describes the source device,and the logical blocks to be transferred to the copy service at the specified IP address. The copymanager and copy service then perform the necessary transfers over an IP network to transfer thedata from the source device to the destination device. The protocol used to perform this transfer isnot standardized, and as such, this sequence may only be used when stated explicitly by an arrayvendor.

The data has now been copied from the copy source device to the copy destination device without any

movement of data through the host.

2.4 Reservation ConsiderationsSCSI Reservations are used to control access to logical units. These reservations may impact theability of the copy manager to access a logical unit.

If the RESERVE command (see SPC-2) is issued to a logical unit, that logical unit cannot be accessed bythe copy manager because the logical unit is reserved for the exclusive use of the single initiator thatissued the RESERVE command.

If reservations are required to restrict access to a logical unit, and a copy manager is also expected toaccess that logical unit, then the PERSISTENT RESERVE IN/OUT commands must be used to createthe reservation. There are two methods available through persistent reservations to enable thiscapability:

A. The host creating the reservation may set the SPEC_I_PT bit (see PERSISTENT RESERVE OUTparameter data in SPC-4), and supply a TransportID list in the PERSISTENT RESERVE OUTcommand that creates the reservation. The Initiator identifier of the relative port in the copymanager that will be used to perform the copy operation must be set in the TransportID list.The relative port identifier used in the segment descriptor of the parameter data of theEXTENDED COPY command must be the relative port identifier of the Initiator identifier thatwas supplied in the TransportID list for the reservation.

B. The host creating the reservation may use the REGISTER AND MOVE service action whencreating the reservation. This allows the host to create the reservation, and transfer accessability created by that reservation to a port within the copy manager (the relative port identifierof the port within the target device and the TransportID of the copy managers initiating port


21/44



Revision 1.0

are required for this operation). This method is described in the Third party persistentreservations section of SPC-4.

2.5 General ConsiderationsHosts typically apply relatively short timers to each I/O operation they issue. In the case of theEXTENDED COPY command, the host must evaluate the request being made and evaluate whatwould be an appropriate timer. For example, an EXTENDED COPY that requested the transfer of100 logical blocks would be expected to complete much faster than an EXTENDED COPY commandthat requested the transfer of 100 million logical blocks.

The EXTENDED COPY command parameter data includes a priority value. This value may be used bythe hypervisor to indicate the priority of the offloaded copy operations relative to the priority of otheroperations being performed on the logical unit. A priority value of zero indicates that the EXTENDEDCOPY command operations have a higher priority than other operations. A priority value of oneindicates that the EXTENDED COPY command operations have the same priority as other operations;

with increasing values indicating lower priorities.

The actual impact of the use of the priority field may vary across different storage vendors.

The List Identifier field of the EXTENDED COPY parameter allows the hypervisor to tag eachEXTENDED COPY operation with a unique identifier. This is needed if the hypervisor wishes to usethe RECEIVE COPY RESULTS command to track the intermediate status of a particular EXTENDEDCOPY command. This may be useful for long lived copy operations, but only 255 of these identifierscan be used at any point in time. If more than 255 operations are expected, then these uniqueidentifiers cannot be used. In addition, when operations are not expected to be long lived, theseunique identifiers are not needed. The discovery process indicates whether this capability is supported

by the device or not.

2.6 DiscoverySeveral methods may be used by a host to determine when the EXTENDED COPY command,COMPARE AND WRITE command, and/or UMAP command are available for use with a particulardevice.

2.6.1 EXTENDED COPY command discoveryTo determine if the EXTENDED COPY command is supported:

A. Issue the INQUIRY command, and examine the 3PC bit in the response. If the 3PC bit is setto one, the device supports the EXTENDED COPY command. The RECEIVE COPY RESULTScommand with a service action code of OPERATING PARAMETERS (03h) will returndescriptive information about support of the EXTENDED COPY command.

B. Issue a REPORT SUPPORTED OPERATION CODES command (see SPC-4):


22/44



Revision 1.0

a. a generic request may be made to determine information about all supported operationcodes (the reporting option set to 000b). This may be useful if a host wishes todetermine the supported status of multiple commands with a single request; or

b. a specific request (the reporting option set to 001b) may be made to determine just thesupported status of the EXTENDED COPY command. Specifying a reporting option of

001b and a requested operation code of 83h will return the supported status of just theEXTENDED COPY command.C. Issue an EXTENDED COPY command specifying a parameter list length of zero. Devices that

support the EXTENDED COPY command will return status of GOOD (while performing nocopy operations), while devices that do not support the EXTENDED COPY command willreturn a status of CHECK CONDITION, a sense key of ILLEGAL REQUEST, and an ASC/Q ofINVALID COMMAND OPERATION CODE.

D. Issue a RECEIVE COPY RESULTS command with a service action code of OPERATINGPARAMETERS (03h). If successful, the command will return descriptive information aboutsupport of the EXTENDED COPY command by the SCSI Device (such as if unique identifiersare required or not, the maximum number of target descriptors supported, maximum number

of segment descriptors supported, maximum transfer size, maximum number of outstandingoperations, etc).

2.6.2 COMPARE AND WRITE command discoveryTo determine if the COMPARE AND WRITE command is supported:

A. Issue a REPORT SUPPORTED OPERATION CODES command (see SPC-4):a. a generic request may be made to determine information about all supported operation

codes (the reporting option set to 000b). This may be useful if a host wishes todetermine the supported status of multiple commands with a single request; or

b. a specific request (the reporting option set to 001b) may be made to determine just the

supported status of the COMPARE AND WRITE command. Specifying a reportingoption of 001b and a requested operation code of 89h will return the supported statusof just the COMPARE AND WRITE command.

B. Issue a COMPARE AND WRITE command specifying the number of blocks to transfer as zero.Devices that support the COMPARE AND WRITE command will return status of GOOD (andperform no actual operation), while devices that do not support the COMPARE AND WRITEcommand will return a status of CHECK CONDITION, a sense key of ILLEGAL REQUEST,and an ASC/Q of INVALID COMMAND OPERATION CODE.

2.6.3 UNMAP operation discoveryTo determine if unmap operations are available:


23/44



Revision 1.0

A. Issue a READ CAPACITY (16) command (see SBC-3) and examine the TPE1

B. The TPRZ

bit in the returnedparameter data. A device that does not support READ CAPACITY (16) (i.e., returns a statusof CHECK CONDITION, a sense key of ILLEGAL REQUEST, and an ASC/Q of INVALIDCOMMAND OPERATION CODE) does not support UNMAP operations. If the TPE bit is setto one, then the device supports the UNMAP command, and/or the WRITE SAME(16)

command with the unmap bit.2

C. For devices that support unmap operations (the TPE bit set is to one):

bit in the returned parameter data is a useful hint for unmap operations(particularly for unmap operations using the WRITE SAME command method).

a. if the device supports the UNMAP command, then the device may or may not supportthe WRITE SAME(16) command with the unmap bit set to one.

b. if the device does not support the UNMAP command, then the device does support theWRITE SAME (16) command with the unmap bit set to one.

c. if the device supports the WRITE SAME(16) command with the unmap bit set to one,then the device may or may not support the UNMAP command.

d. if the device does not support the WRITE SAME(16) command with the unmap bit set

to one, then the device does support the UNMAP command.

2.6.4 UNMAP command discoveryTo determine if the UNMAP command is supported:

A. Issue a REPORT SUPPORTED OPERATION CODES command (see SPC-4):a. a generic request may be made to determine information about all supported operation

codes (the reporting option set to 000b). This may be useful if a host wishes todetermine the supported status of multiple commands with a single request; or

b. a specific request (the reporting option set to 001b) may be made to determine just thesupported status of the UNMAP command. Specifying a reporting option of 001b and a

requested operation code of 42h will return the supported status of just the UNMAPcommand.

B. Issue an UNMAP command specifying a parameter list length of zero. Devices that support theUNMAP command will return status of GOOD (while performing no unmap operations),while devices that do not support the UNMAP command will return a status of CHECKCONDITION, a sense key of ILLEGAL REQUEST, and an ASC/Q of INVALID COMMANDOPERATION CODE.

C. Read the Block Limits VPD page (B0h). Devices that support the UNMAP command willreport a non-zero value in the maximum unmap lba count field.

D. Read the Thin Provisioning VPD page3 (B2h). Devices that support the UNMAP command willreport the TPU4

1 At this time, the name of the TPE bit is being evaluated by the T10 committee, and may be changed. The locationhowever, is bit 7 of byte 14 of the READ CAPACITY (16) returned parameter data.2 The location of the TPRZ bit is bit 6 of byte 14 of the READ CAPACITY (16) returned parameter data.

bit set to one.


24/44



Revision 1.0

2.6.5 WRITE SAME (16) command discoveryTo determine if the WRITE SAME command is supported:

A. Issue a REPORT SUPPORTED OPERATION CODES command (see SPC-4):

a. a generic request may be made to determine information about all supported operationcodes (the reporting option set to 000b). This may be useful if a host wishes todetermine the supported status of multiple commands with a single request; or

b. a specific request (the reporting option set to 001b) may be made to determine just thesupported status of the WRITE SAME (16) command. Specifying a reporting option of001b and a requested operation code of 93h will return the supported status of just theWRITE SAME (16) command.

c. Examine the returned data and examine the CDB USAGE DATA to determine if theunmap bit is supported by the WRITE SAME(16) command.

B. Read the Thin Provisioning VPD page5 (B2h). Devices that support the WRITE SAMEcommand with the unmap bit set to one will report the TPWS6

Unlike the EXTENDED COPY and UNMAP commands, specifying a number of logical blocks of zerocannot be used to determine if the WRITE SAME command is supported. While a device that doesnot support the UNMAP command will return a status of CHECK CONDITION, a sense key ofILLEGAL REQUEST, and an ASC/Q of INVALID COMMAND OPERATION CODE, a device thatdoes support the WRITE SAME command treats a logical block count of zero, as a request to writethe supplied data from the specified starting logical block address to the end of the media. Therefore,a WRITE SAME command with a number of logical blocks field set to zero cannot be used fordiscovery.

This use of a zero value to indicate a length of other than zero is unusual to SCSI. For this reason, it is

possible devices may have non-compliant behavior related to a zero value in the length field (e,g., thecommand may return a status of CHECK CONDITION, with a sense key of ILLEGAL REQUEST whena zero transfer length is requested).

Some devices may support the WRITE SAME command only for the purpose of doing unmapoperations, and therefore may return a status of CHECK CONDITION, with a sense key of ILLEGALREQUEST when the data buffer contains any value other than all zeros.

bit set to one.

3 At this time, the name of the Thin Provisioning VPD page is being evaluated by the T10 committee, and may be changed.The page is identified by page code B2h.4 At this time, the name of the TPU bit is being evaluated by the T10 committee, and may be changed. The location is bit 7of byte 5 of the VPD page.5 At this time, the name of the Thin Provisioning VPD page is being evaluated by the T10 committee, and may be changed.The page is identified by page code B2h6 At this time, the name of the TPWS bit is being evaluated by the T10 committee, and may be changed. The locationhowever, is bit 6 of byte 5 of the VPD page.


25/44



Revision 1.0

2.6.6 Logical unit discovery requirementThe host application making use of the EXTENDED COPY command must still be aware of thetopology of the storage connectivity. Specifically, the application must know which logical units are

within the appropriate domains so a copy manager or copy service may be used to perform the copyoperations between the logical units.

2.7 Target Descriptors2.7.1 Identification target descriptorsThe Identification descriptor target descriptor contains a Device Identification designator from VPDpage 83h of the logical unit the descriptor is referencing. The designators supplied by the host arecopied from INQUIRY VPD page 83h, and should have the association field set to 00b. Thesedescriptors may be used to identify internal logical units, or if the copy manager maintains information

about external logical units, these descriptors may also be used to identify external logical units.

2.7.2 Port type descriptors for Fibre Channel TargetsFor copy offload to or from a Fibre Channel device, when identification descriptor target descriptorsare not used, the target descriptor should use the Fibre Channel N_Port_Name target descriptor(type E0h), or if necessary, the Fibre Channel N_Port_ID with N_Port_Name checking targetdescriptor (type E2h). Fibre Channel N_Port_ID target descriptors (type E1h) are not recommended.

2.7.3 Port type descriptors for iSCSI TargetsFor copy offload to or from iSCSI devices, when identification descriptor target descriptors are notused, the target descriptor may use either the IPv4 target descriptor or the IPv6 target descriptor.

2.7.4 Port type descriptors for SAS targetsFor copy offload to or from SAS devices, when identification descriptors target descriptors are notused, the target descriptor should use the SAS Serial SCSI Protocol target descriptor (type E9h).

2.7.5 Copy Service DescriptorFor a copy offload operation using an IP Copy Service, the IP copy service target descriptor should beused.

2.8 CompletionAt the completion of an EXTENDED COPY command, the copy manager returns status to the host. Ifthe SCSI Status of the operation is GOOD, then the results of the requested transfer are available foruse. If the SCSI Status is not GOOD, then a failure of some kind occurred. Well known SCSI eventssuch as ACA ACTIVE, BUSY, QUEUE FULL, and others should be handled just as they are for other


26/44



Revision 1.0

commands. The status of CHECK CONDITION is returned along with Sense Data to includeadditional information about the specifics of the failure.

Command processing in the Copy Manager begins by validating portions of the command andparameter data. A failure of these checks (described in SPC4 as Errors detected before starting

processing of the segment descriptors) would mean that no data was transferred. Errors that occurafter these checks (described in SPC4 as Errors detected during processing of segment descriptors)would mean that data may or may not have been transferred; and are reported with the sense key setto COPY ABORTED.

Commands that are terminated with a sense key of ILLEGAL REQUEST and an additional sense codeof PARAMETER LIST LENGTH ERROR, TOO MANY TARGET DESCRIPTORS, or TOO MANYSEGMENT DESCRIPTORS must be reformed with a shorter parameter list or fewer descriptors andreissued. The additional sense codes of INVALID FIELD IN CDB and INVALID FIELD INPARAMETER LIST indicate command processing errors that must be corrected and the commandreissued.

Commands that are terminated with a sense key of COPY ABORTED may have transferred somedata, and the remainder of the sense data must be examined for the exact error condition. The sensedata information field should be examined to determine if residual information has been returned for apartial transfer. In addition, if multiple segment descriptors are used, the third and fourth bytes of thecommand-specific information field of the sense data may be used to determine the segment number ofthe segment descriptor being processed at the time the error occurred. See SPC-4 for details.

When a copy operation is aborted (the Sense Key of the Sense Data is COPY ABORTED), the FieldPointer field within the Sense Key specific data of the Sense Data is used to indicate which targetdescriptor or segment descriptor encountered an error (see SPC4).

3 Security ConsiderationsCopy managers are initiators of I/O operations within the storage network. As such, the copymanager is able to read from and write to other logical units within the storage network. This abilitycreates the opportunity for special security considerations. While the SCSI Standard provides theACCESS CONTROL IN/OUT command, these command are typically not implemented. Basicconsiderations are listed here to raise awareness of this topic for copy offload implementers.

When using Fibre Channel connectivity, if external copy operations are to be permitted, FC zoningwithin the fabric, if established, must be defined so that the ports of the storage array containing the

copy manager are permitted access to the ports of the storage arrays from which data will be copied(the ports on the source device), and are permitted access to the ports of the storage arrays to whichdata will be copied (the ports on the destination device). In addition, in arrays that support LUNmapping and masking (the ability to enable or restrict access to a LUN to a subset of the possibleinitiators), the mapping and masking for source and/or destination logical units must be setup toinclude the FC initiators of the array that contains the copy manager.


27/44



Revision 1.0

If only internal copy operations are to be utilized, there are no FC zoning considerations. However,the copy manager is responsible to verify that the requesting application client (the host) is permittedread access to the source logical unit and write access to the destination logical unit. EXTENDEDCOPY commands that attempt access to logical units to which the host does not have appropriate

access will be failed by the copy manager.

When using iSCSI, zoning does not apply, but rather, it is replaced by IP networking (including routing)considerations. The copy manager ports must have connectivity and active network paths to anyexternal source and/or external destination devices. In addition, LUN mapping and maskingconsiderations also apply to iSCSI connectivity.

Usage of an IP copy service requires IP networking (including routing) considerations. The copymanager must have connectivity and active network paths to allow communication with the IP copyservice. There may also be other vendor unique security considerations involved with theseconfigurations.

Extended Copy commands should be sent to the copy destination (rather than the source). Risks ofinappropriate reads (while bad) are less dangerous than risks of inappropriate writes (corruption).

4 Use Sequence4.1 Sequence DiagramFigure 7 is a UML sequence diagram showing the data structures and components associated with acopy offload operation. The interactions of the data structures and components are shown, along withthe basic sequence of operations involved in a copy offload operation. This figure is intended toconvey the interactions with the hypervisor, and the copy related operations performed by the copymanager as a result of processing an EXTENDED COPY command received from the hypervisor. Thisfigure does not show all the necessary transport specific protocol operations required between thecopy manager and the source and/or destination devices such as discovery, login, logout, sessionmanagement, etc.


28/44



Revision 1.0

HypervisorHost Copy Manager Copy Source Device Copy Destination Device

XCOPY Command

Send INQUIRY; read Page 83h

Identification Descriptor

Determine data destination(s)

CDB & XCopy Param listData Structures

Allocate

Fill identification descriptor for copy source device

Fill identification descriptor for copy dest device

Fill in segment descriptor(s)

Fill in CDB for length of parameter list

This represents the Use Sequence forMethod 1 (Identification Descriptor E4h)of gathering the copy target descriptorinformation

XCOPY complete

Send INQUIRY; read Page 83h

Identification Descriptor

Determine what data to copy

Param list

Extract target descriptors for source and destination

Read

Command Complete

Write

Command Complete

Read

Command Complete

Command Complete

Write

All Data copied

Extract segment descriptor(s)

Figure 7 Copy Manager Sequence Diagram


29/44



Revision 1.0

4.2 Gather Copy Target Descriptor InformationThe relative port identifier may be obtained from a port in the SCSI INQUIRY VPD page 83h data thatis available from each port of a device. The relative port identifier descriptor has the ASSOCIATIONfield set to 01b and the DESIGNATOR field set to 4h.

When supplying the relative port identifier in the copy target descriptor, a value of zero indicates thatthe copy manager is not restricted to using a specific port, and may use any port to access the copytarget device.

Method 1 identifies a logical unit using the Device Identification designator and for external logicalunits, requires that the copy manager maintain information about how to access those external logicalunits.

Methods 2, 3, 4, and 5 identify a logical unit using its logical unit number and for external logical units,the path to the external logical unit (by specifying the remote port name through which the logical unit

is accessed).

Method 6 identifies a logical unit using the Device Identification designator and for external logicalunits, includes an IP address descriptor that identifies how to access those external logical units.

4.2.1 Method 1 Identification Descriptor (E4h)1) Determine the relative port identifier of the initiator Port within the device to be used by the

copy manager for access to the copy target device.2) Examine SCSI INQUIRY VPD page 83h of the copy target device. Search the list of Device

Identification designators for a suitable descriptor.

3) Create an Identification descriptor target descriptor using this information.

4.2.2 Method 2 N_Port_Name target descriptor (E0h)1) Determine the relative port identifier of the initiator Port within the device to be used by the

copy manager for access to the copy target device.2) Determine the Logical Unit Identifier of the logical unit when accessed through the relative port

identifier determined in step 1.3) Determine the N_Port_Name defined by the PLOGI of the VN_Port for the copy target device

used to access the logical unit ID determined in step 2.4) Create a Fibre Channel N_Port_Name target descriptor using this information.

4.2.3 Method 3 N_Port_ID target descriptor (E1h not recommended)1) Determine the relative port identifier of the initiator Port within the device to be used by the

copy manager for access to the copy target device.


30/44



Revision 1.0

2) Determine the Logical Unit Identifier of the logical unit when accessed through the relative portidentifier determined in step 1.

3) Determine the D_ID to be used in the transport frames of the copy target device used toaccess the logical unit ID determined in step 2.

4) Create a Fibre Channel N_Port_ID target descriptor using this information.

4.2.4 Method 4 N_Port_ID with N_Port_Name checking target descriptor (E2h non-fabric environments only)

1) Determine the relative port identifier of the initiator Port within the device to be used by thecopy manager for access to the copy target device.

2) Determine the Logical Unit Identifier of the logical unit when accessed through the relative portidentifier determined in step 1.

3) Determine the D_ID to be used in the transport frames of the target device used to access thelogical unit ID determined in step 2.

4) Determine the N_Port_Name defined by the PLOGI of the VN_Port for the copy target device

used to access the logical unit ID determined in step 2.5) Create a Fibre Channel N_Port_ID target descriptor with this information.

4.2.5 Method 5 iSCSI target descriptors (E5h and EAh)1) Determine the relative port identifier of the initiator port within the device to be used by the

copy manager for access to the copy target device.2) Determine the Logical unit Identifier of the logical unit on the iSCSI device when accessed

through the relative port identifier determined in step 1.3) Determine the iSCSI target device name.4) Create the appropriate (IPv4 or IPv6) iSCSI target descriptor with this information.

4.2.6 Method 6 IP COPY Service descriptor (EBh)1) Examine SCSI INQUIRY VPD page 85h of the copy target device. Search the list of Network

Service Descriptors for a Copy Service (type 06h).2) Examine SCSI INQUIRY VPD page 83h of the copy target device. Search the list of Device

Identification designators for a suitable descriptor.3) Create an IP Copy Service target descriptor with this information.

4.2.7 Method 7 SAS Serial SCSI Protocol target descriptor (E9h)1) Determine the relative port identifier of the initiator port within the device to be used by the

copy manager for acess to the copy target device.2) Determine the Logical unit Identifier of the logical unit on the SAS device when accessed

through the relative port identifier determined in step 1.3) Determine the SAS Address of the target device.


31/44



Revision 1.0

4) Create a SAS Serial SCSI Protocol target descriptor with this information.

4.3 Gather Segment Descriptor Information (02h)1) Determine the starting LBA on the copy source device to begin the read operation.

2) Determine the starting LBA on the copy destination device to begin the write operation.3) Determine the length of the copy (the number of logical blocks to copy). This value is used

both for the number of blocks to read and the number of blocks to write.

4.4 Build the Extended Copy Parameter list1) Allocate space for parameter list2) Fill in target descriptor for copy source device3) Fill in target descriptor for copy destination device (if different than the copy source)4) Fill in segment descriptor(s):

a) index of the copy source device target descriptorb) index of the copy destination device target descriptorc) number of logical blocks to read and writed) starting LBA on the copy source devicee) starting LBA on the copy destination device

5) Fill in the CDB with the length of the parameter list6) Send the command to the copy destination device

4.5 Copying a Virtual DiskThe EXTENDED COPY command may be used to create a copy of a virtual disk with no burden onthe server/hypervisor to optimize hypervisor performance. This may be used to provision a newvirtual machine. For example, an administrator may wish to copy a known good golden image.

The steps above are used to build the appropriate copy parameter list to copy a virtual disk. In thefollowing example (see Figure 8), a virtual disk occupies 3 segments of a container that resides on 2logical units (segment 1 and 2 reside on logical unit number 1, while segment 3 resides on logical unitnumber 3). This virtual disk is to be copied to a container that resides on logical unit number 6 andlogical unit number 9.


32/44



Revision 1.0

Figure 8 Example Copy of a Virtual Disk in a Container

A single EXTENDED COPY command parameter list may be built (see Table 1) that contains multiplesegment descriptors. The segment descriptor for segment 1 points to the source target descriptor forLUN 1 and the destination target descriptor for LUN 6. The segment descriptor for segment 2 pointsto the source target descriptor for LUN 1 and the destination target descriptor for LUN 9. Thesegment descriptor for segment 3 points to the source target descriptor for LUN 3 and thedestination target descriptor for LUN 9. Each segment descriptor contains two pointers to copytarget descriptors (one for the source logical unit, and one for the destination logical unit). In addition,

the starting LBA on the source logical unit, the starting LBA on the destination logical unit, and thenumber of logical blocks to transfer are included in the segment descriptor.

LUN 1 LUN 3

LUN 6 LUN 9

1 23

1 2 3


33/44



Revision 1.0

Header

Source Target Descriptor (for LUN 1)


Destination Target Descriptor (for LUN 6)


Segment Descriptor (for segment 1)Starting Read LBA on LUN 1Starting Write LBA on LUN 6

Number of logical blocks in segment 1


Number of logical blocks in segment 2Segment Descriptor (for segment 3)

Starting Read LBA on LUN 3Starting Write LBA on LUN 9


Table 1 Single Parameter List Example

Multiple EXTENDED COPY command parameter lists may also be built where a unique EXTENDEDCOPY command is sent to the copy manager that describes only one portion of the copy offloadoperation per EXTENDED COPY command. This is shown in Table 2, Table 3, and Table 4.

Header





Table 2 Parameter List to copy Segment 1


34/44



Revision 1.0

Header






Header






In general, the more information contained within a single EXTENDED COPY command, the moreefficient and higher performance will be the copy offload operation. However, the more informationcontained within a single EXTENDED COPY command, the more complex the processing, and errortracking should a failure occur. This tradeoff must be made by the application performing the copy

offload operation.

4.6 Copying an Entire Logical UnitThe EXTENDED COPY command may be used to copy an entire logical unit with no burden on theserver/hypervisor to optimize hypervisor performance. This operation may be used, for example,when a new virtual infrastructure is provisioned for a new office, business unit, user group, ortest/development environment.

This sequence may be done via a single EXTENDED COPY command, where the starting LBA is set to

zero, and the length is set to the total number of logical blocks on the logical unit, or this copy may bedone using a single EXTENDED COPY command that contains multiple segment descriptors, whereeach segment is described independently. Alternatively, multiple independent EXTENDED COPYcommands may be used; each copying just a segment of the blocks on the logical unit.


35/44



Revision 1.0

Figure 9 Example of Copying an Entire Logical Unit

In this example, the entire capacity of the logical unit is copied from LUN 1 to LUN 3 (see Figure 9). Asingle EXTENDED COPY command may be used (see Table 5) that describes the entire capacity ofthe logical unit. A multi-segment descriptor example is shown in Table 6. This copy offload could alsobe performed using multiple EXTENDED COPY commands each containing a single segmentdescriptor.

Header



Segment Descriptor (for the entire LUN)Starting Read LBA on LUN 1 (LBA 0)Starting Write LBA on LUN 3 (LBA 0)Number of logical blocks on the LUN

Table 5 Whole LUN copy (single segment descriptor)

LUN 1 LUN 3

1 2

33

1 2


36/44



Revision 1.0

Header



Segment Descriptor (for segment 1)Starting Read LBA on LUN 1 (LBA 0)Starting Write LBA on LUN 3 (LBA 0)

Number of logical blocks in segment 1 = n

Segment Descriptor (for segment 2)Starting Read LBA on LUN 1 (LBA n)Starting Write LBA on LUN 3 (LBA n)

Number of logical blocks in segment 2 = m

Segment Descriptor (for the remainder)Starting Read LBA on LUN 1 (LBA n+m)Starting Write LBA on LUN 3 (LBA n+m)

Number of logical blocks on LUN (n+m)

Table 6 Whole LUN copy (multiple segment descriptors)

4.6.1 CDB and Parameter Data DetailsTable 7 shows a more detailed example CDB used with the parameter data shown in Table 8. Thisexample shows a copy offload from an internal logical unit to another internal logical unit where onesegment of data is copied (note that the CDB contents shown here are specific to this example, andnot to be construed as generic). To copy an entire logical unit, the starting LBA of the source deviceand the starting LBA of the destination device would both be set to zero, and the number of blocks

field would be set to the number of blocks on the logical unit.

BitByte

7 6 5 4 3 2 1 0

0 Operation Code (83h)

1-9 All Zeros (0h)

10-12 Zeros (000000h)

13 Parameter Length (112 / 70h)

14 Zero (0h)

15 Control Byte

Table 7 Detailed CDB

If the application does not wish to allow the length of time that such a large transfer may require, theoperation should be broken into multiple EXTENDED COPY commands with shorter transfers


37/44



Revision 1.0

specified in the parameter data (segment descriptor) of each command. The parameter data shown intable 8 is specific to this example, and should not be construed as generic.

BitByte

7 6 5 4 3 2 1 0

01

2-3 Target descriptors length (64 / 0040h)

4-7 All Zeros (00000000h)

8-11 Segment descriptors length (32 (0020h)

12-15 All Zeros (00000000h)

Target Descriptor 0

16 Descriptor type (E4h)

17 Zero (00h)

18-19 Relative port identifier (0000h)

20-n Device Identification designator from source device page 83h

n+1-47 Zeros (00h)

Target Descriptor 1

48 Descriptor type (E4h)

49 Zero (00h)

50-51 Relative port identifier (0000h)

52-n Device Identification designator from destination device page 83h

n+1-79 Zeros (00h)

Segment Descriptor 0

80 Descriptor type (02h)

81 Zero (00h)

82-83 Length (0018h)84-85 Source target descriptor index (0000h)

86-87 Destination target descriptor index (0001h)

88-89 Zero (0000h)

90-91 Number of Blocks to transfer (xxh)

92-99 Starting LBA on source device (xxxxxxxxxxxxxxxxh)

100-108 Starting LBA on destination device (xxxxxxxxxxxxxxxxh)

109-111 Zeros (000000h)

Table 8 Detailed Parameter Data Example

4.7 Copying a Virtual MachineA common practice to generate a new virtual machine is to create a golden image, then copy thegolden image to provision the new virtual machine. The virtual machine generated with this procedure


38/44



Revision 1.0

is expected to be ready for deployment in production with the least possible configuration steps. Thevirtual machine copy operation may also be used as a possible option for back up.

To copy a virtual machine, all of the information (e.g., metadata, virtual disks, if any) associated withthat virtual machine must be copied. All meta-data and virtual disks data are stored on one or more

logical units. On each logical unit, the data are stored in one or more ranges of contiguous LBAs. Thevirtual machine copy may be composed of multiple copy operations, as required to transfer allcorresponding data segments.

The hypervisor, that has knowledge of the data layout, should identify the LBA addresses and rangeson the logical unit, The hypervisor then constructs the appropriate requests specifying the appropriateLBAs to be copied, and passes the request(s) to the underlying storage.

This may be done via one or more EXTENDED COPY command(s), the choice being dependent onperformance considerations and complexity. Each EXTENDED COPY command may contain multipleindependent segment descriptors where each segment descriptor may be associated with the metadata

or with a virtual disk associated with the virtual machine.

Header



Segment Descriptor (for segment 1)Starting Read LBA on LUN 1 (LBA a)Starting Write LBA on LUN 3 (LBA x)


Segment Descriptor (for segment 2)Starting Read LBA on LUN 1 (LBA b)Starting Write LBA on LUN 3 (LBA y)


Segment Descriptor (for segment 3)Starting Read LBA on LUN 1 (LBA c)Starting Write LBA on LUN 3 (LBA z)

Number of logical blocks in segment 3 = o

Table 9 Copy of multiple segments of a LUN

4.8 Virtual Machine Storage MigrationThis use sequence enables virtual machine storage migration by providing an offload function to copyall of the storage associated with the virtual machine from one array to another. The basic operationis similar to the Copying a Virtual Machine sequence above, except that the value in the destinationcopy descriptor contains an identifier of a destination that is in a different array. This use sequence


39/44



Revision 1.0

may also be performed manually (through means outside the scope of this document) by leveraging anarrays mirroring capabilities.

First the entire virtual machine must be copied from the source target array to destination targetarray. The location of the Copy Manager is immaterial: it may be external to both arrays or contained

within either the Source or Destination Array.

To copy an entire virtual machine, multiple virtual disks or entire logical units may need to be copied.The hypervisor, that has knowledge of the data layout, should identify the LBA addresses, ranges, andport/array identifiers for the storage. The hypervisor then constructs the appropriate requestsspecifying the appropriate LBAs to be copied, and passes the request(s) to the underlying storage.

The requests are passed to the underlying storage system via one or more EXTENDED COPYcommand(s), the choice being dependant on performance and complexity considerations. EachEXTENDED COPY command may contain multiple independent segment descriptors, where eachsegment descriptor may be associated with the metadata or with a virtual disk associated with the

virtual machine.

After the virtual machines have been copied, the previously used space on the source storage may nowbe reclaimed as described in section 4.10 below.

The operation details in section 1 describe how to construct the Source Target Descriptors andDestination Target Descriptors for the EXTENDED COPY command, such as in the example below:


40/44



Revision 1.0

Header

Source Target Descriptor (for LUN 1)- Array Port & LUN Identifiers

Destination Target Descriptor (for LUN 3)- Array Port & LUN Identifiers

Segment Descriptor (for segment 1)Starting Read LBA on LUN 1 (LBA a)Starting Write LBA on LUN 3 (LBA x)


Segment Descriptor (for segment 2)Starting Read LBA on LUN 1 (LBA b)Starting Write LBA on LUN 3 (LBA y)


Segment Descriptor (for segment 3)Starting Read LBA on LUN 1 (LBA c)Starting Write LBA on LUN 3 (LBA z)

Number of logical blocks in segment 3 = o

Table 8 Copy of multiple segments of a LUN across arrays.

4.9 Dividing Virtual MachinesMultiple virtual machines may be grouped together because they are associated with the same

application, the same set of users, the same security context, or require a specific service level. Forthese and other reasons, the administrator may need to simultaneously relocate, a subset of virtualmachines from one storage location to another using a single procedure. When this operation isperformed, virtual machines may be viewed to be in one of two groups: the one group that is left inthe original location, or a second group that is relocated to a new location.

Virtual machine division may be performed between storage belonging to the same storage array oracross different storage arrays.

This procedure is actually composed of multiple virtual machine storage migration operations, asdescribed in section 4.8. After the virtual machines have been migrated, the unused space on the

source storage may be reclaimed as described in section

Documents

HSI Copy Offload WP-r12