Alcoa SRDFA ConfigGuide(YT)v1.2 031606

EMC Corporation Technology Solutions

SRDF/A Config GuideSRDF/A Config Guide

Alcoa CorporationAlcoa Corporation

Date: Feb 08, 2006

Prepared for:

Alcoa CorporationAlcoa Technical Center

100 Technical Drive

Alcoa Corporation Center, PA 15069-0001

USA

Prepared by:

EMC Corporation

1160 Omega Corporate DrivePittsburgh, PA 15205USA

Date: February 08, 2006 EMC SOW Number: ALCOA06272005

Copyright © 2002 - 2006 EMC® Corporation. All rights reserved.

1



2


Table of Contents

Introduction.......................................................................................................................4Purpose..............................................................................................................................4Descriptions of SRDF/A...................................................................................................4Symcli Commands............................................................................................................4Split....................................................................................................................................6Full Restore.......................................................................................................................6Failover..............................................................................................................................7Failback..............................................................................................................................7Suspend.............................................................................................................................8Resume..............................................................................................................................8SRDF Control Operations.................................................................................................9SRDF Pair States.............................................................................................................10Switching from SRDF/A to Another Mode....................................................................10Enabling and Disabling Consistency............................................................................11Recovering after Loss of Links......................................................................................11SRDF/A and Rolling Disasters.......................................................................................12How to create Device Group..........................................................................................13How to import or export Device Group.........................................................................13Device group verification...............................................................................................13Various Remote Configurations....................................................................................14Alcoa SRDF/A Planned Outages procedures where validate R2 data with no impact to R1 production.............................................................................................................15Alcoa SRDF/A True DR Procedures: How to Failover and Failback where R2 data will propagate back to R1...............................................................................................20Database Recoverable Copy From BCV (Apply to Oracle only).................................23REFERENCES.................................................................................................................32


3


IntroductionThis document is prepared for customers who have little knowledge of EMC products (Hardware and Software) with systems administration background.

SRDF/A is one of the EMC products in the SRDF family that works on top of the base SRDF modes. The features of SRDF/A are:

No performance impact on the production systems Unlimited distance Minutes of data exposure

PurposeThe purpose of this document is to provide details of the scripts, installation and configuration (for more details, refer the Operational Guide) related to the Alcoa SRDF/A project.

The intentions of the scripts are to provide the functionality of SRDF/A commands, so that command line entry will not be required, and thus reduce risks when executing the commands.

Descriptions of SRDF/A In the SRDF/Asynchronous mode (SRDF/A), the Symmetrix array provides a consistent

point-in-time image on the target (R2) device, which is a short period of time behind the source (R1) device.

Managed in sessions, SRDF/A transfers data in predefined timed cycles or delta sets to ensure that data at the remote (R2) site is dependent write consistent. This mode requires an SRDF/A license.

The Symmetrix array acknowledges all writes to the source (R1) devices as if they were local devices. Host writes accumulate on the source (R1) side until the cycle time is reached and are then transferred to the target (R2) device in one delta set. Write operations to the target device can be confirmed when the current SRDF/A cycle commits the data to disk by successfully de-staging it to the R2 storage devices.

Because the writes are transferred in cycles, any duplicate tracks written to can be eliminated through Symmetrix ordered write processing, which transfers the changed tracks over the link only once within any single cycle.

Symcli CommandsInitialize the SYMAPI database that maintains configuration and status information about all devices in the symmetrix units:symcfg discover

On a UNIX system, when you run symcfg discover, a Symmetrix host database file, symapi_db.bin, is created in:/var/symapi/db


4


On Windows, the database file is found under:C:\Program Files\EMC\SYMAPI\db

Verfiy SRDF linkFrom Both hosts, issue a SYMRDF PING command to verify that the SRDF link is communicating properly.

Display the status of the Remote Link Directorssymcfg –RA all list

Finding SRDF DevicesYou can find all Symmetrix arrays that are reachable via the SRDF links. For example, to view how Symmetrix arrays are attached to your host, enter:symcfg list

Device GroupsTo view all device groups that have been created in your host database file, enter:symdg list

Listing SRDF DevicesYou can list the SRDF devices that are visible to your host, or SRDF devices that are configured on a given Symmetrix array.symrdf list pd

Listing SRDF/A Capable DevicesThe SRDF/A-capable device option (-rdfa) allows you to list devices that are SRDF/A-capable.Note: Beginning with Enginuity 5671, all devices are SRDF/A-capable and the command will display all devices.

Listing RDF Group DevicesThe RDFG (-rdfg) option applies a Symmetrix RDF (RA) group number to the command to restrict the list to just the RDF devices of an RDF group. When used with control, verify, and query actions, this option targets a specific RDF group number or all groups when devices are configured RDF concurrent. The RDF group syntax for all the composite commands is -rdfg nn|All. Here is the command for listing all rdf groups defined on Symm:symcfg –sid <sid> -rdfg <rdf_group|ALL>


5


Full EstablishTo initiate a full establish, all RDF pairs in the group must already be in one of the following states:

Split Suspended and Write Disabled or Not Ready at the source Invalid, R1 and R2 are Not Ready and the link is Ready R1 Updated or Failed Over and the R1 is not visible to any host

symrdf -g DgName establish -full

Incremental EstablishTo invoke this operation, the RDF pair(s) must already be in one of the following RDF states:

Split Suspended Invalid, R1 and R2 are Not Ready and the link is Ready

symrdf -g DgName establish

SplitTo invoke a split, the RDF pair(s) must already be in one of the following states:

Synchronized Suspended R1 Updated SyncInProg and the -force option is specified

symrdf -g DgName split

Full RestoreTo invoke this operation, the RDF pair(s) must already be in one of the following RDF states:

Split Suspended and Write Disabled at the source Suspended and Not Ready at the source Invalid, R1 and R2 are Not Ready and the link is Ready

symrdf -g DgName restore –full

Incremental RestoreTo invoke this operation, the RDF pair must already be in one of the following states:

Split Suspended and Write Disabled at the source Suspended and Not Ready at the source Suspended and the force (-force) option is specified Invalid, R1 and R2 are Not Ready and the link is Ready


6


symrdf -g DgName restore

FailoverTo invoke a failover, the RDF pair(s) must already be in one of the following states:

Synchronized Suspended R1 Updated Partitioned (when you are invoking this operation from the target side)

symrdf -g DgName failover

Failover operation will be rejected if any of the device pairs are in the following states without specifying the -force option:

Split SyncInProg R1 UpdInProg Invalid

FailbackTo invoke a failback, the RDF pair(s) must already be in one of the following states:

Failed Over Suspended and Write Disabled at the source Suspended and Not Ready at the source R1 Updated R1 UpdInProg

symrdf -g DgName failback

Update R1 MirrorWhile the target (R2) device is still operational (Write Enabled to its local host(s)), an incremental data copy from the target (R2) device to the source (R1) device can be initiated in order to update the R1 mirror with changed tracks from the target (R2) device.To invoke this operation, the RDF pair must already be in one of the following states:

R1 Updated Failed Over Suspended and Write Disabled at the source Suspended and Not Ready at the source

symrdf -g DgName update

Continuous R1 UpdatesThe until (-until) option when used with the update argument checks the number of invalid tracks that are allowed to build up from the active R2 local I/O before another update (R2 to R1 copy) is retriggered. The update sequence loops until the invalid track count is less than the number specified for the -until value.


7


Note that these update sequences start with an immediate update once this command is started as follows:

1. Update R1 mirror.2. Changed tracks build up on R2.3. Check invalid track count.

For example, to update the R1 mirror when track changes are in excess of 3000 on the R2, enter:symrdf -g DgName update -until 3000In this example, the R1 mirror will be continuously updated until the number of tracks to be copied is below 3000.

SuspendTo invoke a suspend, the RDF pair(s) must already be in one of the following states:

Synchronized R1 Updated

symrdf -g DgName suspendWhen the suspend has completed successfully, the devices will be suspended on the SRDF links and their link status set to Not Ready (NR).This operation will be rejected if any of the following occur:

If the source has invalid local (R1) tracks If any of the device pairs are in one of the following states and the -force option is not

specified:o SyncInProg stateo UpdInProg stateo Invalid stateo Split state and the link status is Write Disabledo Suspended state and the link status is Write Disabledo Device Domino modeo When the source has invalid remote (R2) trackso When the target has invalid local (R2) trackso When consistency is enabled and the -force option isn’t specified

ResumeThe resume action argument resumes I/O traffic on the SRDF links for all remotely mirrored RDF pairs in the group or device file.symrdf -g DgName resume


8


SRDF Control Operations


9


SRDF Pair States

Switching from SRDF/A to Another Mode It is sometimes necessary to change from SRDF/A mode to Adaptive Copy Disk mode due

to temporary operational necessities. To accomplish this, consistency protection must be first disabled as shown earlier.

If mode switching is performed while the links are intact and traffic is flowing across the links, the commands:

symrdf –g DgName set mode acp_disk symrdf –g DgName set mode async

will cause the current Apply and Transmit/Receive delta sets (N-2 and N-1 cycles) to be completed and the outstanding I/Os in the Capture delta set (N cycle) to be marked as R2 invalids on the R1 side. As long as the two sides are not fully synchronized, consistency is no longer preserved on the R2 side. It’s recommended to change acp_disk mode to async when R1 and R2 only have <30k invalid tracks.


10


Enabling and Disabling Consistency The primary value proposition of SRDF/A is to provide a consistent restartable image of user data at the remote site without affecting application performance. Be sure there are 0 invalid tracks between R1 and R2 before enable consistency. To enable consistency protection, it is necessary to issue the command: symrdf –g DgName enable

Enabling consistency guarantees consistency of the R2 copy of data at all times. If the logical connection between the two sides is interrupted, either due to a planned or unplanned event, consistency must be explicitly enabled after the resumption of link traffic. When an exit from SRDF/A mode is planned, consistency can be disabled by using the command: symrdf –g DgName disable

Recovering after Loss of Links Prior to starting a resynchronization after link recovery, EMC recommends that a BCV copy of the R2 volumes be split off and set aside to guarantee the availability of a consistent image of data, to guard against an unforeseen interruption of the resynchronization process.

If logical connection between the two sides is lost for a long period of time allowing for the buildup of a large number of R2 invalids on the R1 side, resumption of the links with SRDF/A enabled (the default behavior in open systems) can cause a heavy surge of link traffic, created by the backlog of invalid tracks being added to the cycles generated by the production traffic. This can lead to an unacceptably high demand being placed on Symmetrix cache, causing SRDF/A to drop again.

To avoid this risk, it is best to briefly disable SRDF/A when restarting after the build up of invalid tracks. It is suggested that resynchronization be undertaken at a time when there is a relatively low amount of incoming host writes. Recommended methods of getting back to steady state in such an event are:

o Set the SRDF mode to Adaptive Copy Disk mode by using the command: symrdf –g DgName set mode acp_disk

In this mode, the invalid tracks will be synchronized and new writes will be buffered in cache and sent across without impacting the host application. As long as the rate of incoming writes is less than the link bandwidth, this mode of operation will allow production to continue while continuing to reduce the backlog of invalid tracks. The resynchronization process may be relatively slow. While invalid tracks are being applied during the resynchronization process, the R2 data is not consistent.

o Set the SRDF mode to Asynchronous and allow the resynchronization to occur when invalid tracks <30k. The data on the R2 is not consistent until the invalid tracks have been reduced to zero.

o If the potentially extended resynchronization time and the consequent exposure to data loss in the event of a complete source site failure during resynchronization are deemed unacceptable, then the production application will have to be halted while the resynchronization process completes.


11


It is recommended that SRDF/A be enabled only after the two sides are synchronized and 0 invalid tracks.

SRDF/A and Rolling Disasters EMC defines a rolling disaster as two successive failures, where the second failure occurs

before recovery procedures related to the first one were completed. SRDF/A ensures that the data on the R2 side is consistent at all times, even when the source site is unavailable. As mentioned previously, once the source and target have lost connectivity, it is important to split off a BCV copy of the R2 data, before a resynchronization is attempted. The scenario described next is one that could happen with SRDF/A.

Let us presume that there is a buildup of invalid tracks because the R1 was being written to while the target was unreachable. This can be the result of a planned or unplanned logical suspension of links. Next, the links are resumed and a resynchronization is initiated from the R1 to the R2. Immediately after the start of resynchronization, the tracks to be copied from the R1 to the R2 get marked as local invalids on the R2 side—meaning that they need to be brought across from the R1. If SRDF/A is enabled prior to both sides reaching synchronization (not recommended), the invalid tracks are sent across in delta sets along with the write I/Os. If an unexpected loss of links now occurs before the state of the SRDF volumes is consistent, the R2 will now contain local invalids (caused during the prior resynchronization) and remote invalids (caused by the tracks belonging to the incomplete receive delta set). In SRDF, this is a problem requiring human intervention. In SRDF/A, this is an acceptable, though not desirable, state of affairs. This is why it is recommended that SRDF/A be enabled only after both sides are synchronized.


12


How to create Device GroupTo check the configuration of SRDF devices before adding them to a device group or composite group, you can use the symrdf list command to list SRDF devices configured on Symmetrix arrays attached to your host:symrdf listWhen you add a device to a device group, a logical device name is assigned to it either by your specifying a logical name on the command line or by default.The following sequence creates an RDF1 type device group and adds an R1 device to the group:1. Create a device group named DGName:symdg create DGName –type rdf12. Add an R1 device (Symmetrix device name 0027) to the device group on Symmetrix numberSA287890027. A default logical name of the form DEV001 is assigned to the R1 device in rdf group 5:symld -g DGName –sid 0027 add dev 085 –rdfg 5or add a range of devicessymld -g DGName -sid 0027 add dev -range <start:end>3. If you want to add devices (needed to append few devices in future groups) to the existing

device group, use the following commands.symld -g DGName -sid 0027 add dev <DEV #>

How to import or export Device GroupTo issue the same SRDF commands from the target-side host as from the source-side host, it is necessary to build an RDF2 target-side device group that has the same definitions as the RDF1 source-side device group. The symdg export command creates a text file (Rdf1Grp.txt) that contains the RDF1 group definitions. You then use rcp (or ftp) to transfer that file to the target host.src# symdg export Rdf1Grp -f Rdf1Grp.txt -rdfsrc# rcp Rdf1Grp.txt target-side_host:/.

On the target host, the symdg import command builds the RDF2 device group using the definitions from the text file.tgt# symdg import Rdf2Grp -f Rdf1Grp.txt

Device group verificationTo verify the device group, the following commands can be issued:

symdg list (all the device groups created on the that system) symld –g <group name> list (gives the members of the group) symdg show <group name> (gives all the details including the RDF modes and RDF state)


13


Various Remote Configurations


14


Alcoa SRDF/A Planned Outages procedures where validate R2 data with no impact to R1 production

1. Validate host config in EMC’s Open System Support Matrix (OSSM) From the latest OSSM:

o Host, host components (drivers, patches, HBAs, etc.) are supported in accordance with the OSSM

2. Validate current level of Symmetrix® microcode (5670 or higher).3. Install Solutions Enabler 6.0.2. Ensure Licenses to run SRDF/A.

Default installs solution enabler v6.02 on hosts that control SRDF/A command and operations.

To check version of symcli install on hostC:\Documents and Settings\root>symcli

Set Path= c:\program files\emc\symcli\bin To install licenses use command: symlmf

Note: License file resides on c:\program files\emc\symapi\config\symapi_licenses.datC:\Documents and Settings\root>symlmf

4. Check Symmetrix Configuration and Status. Initialize the SYMAPI database that maintains configuration and status information about all devices in the symmetrix units. Verify SRDF link. From Both hosts, issue a SYMRDF PING command to verify that the SRDF link is communicating properly.

C:\Documents and Settings\root>symcfg discoverThis operation may take up to a few minutes. Please be patient...

To view how Symmetrix arrays are attached to your hostC:\Documents and Settings\root>symcfg list

Display the status of the Remote Link Directors by using command:symcfg –RA all list

5. Ensure proper BIN file configuration. Check with CE to be sure of this task is completed.

6. On Windows, the database file is found under: C:\Program Files\EMC\SYMAPI\db

7. Displays a listing of R1, R2 and BCV devices. List the R1, R2, and BCV devices by running commands:

symdev list –sid xx -R1symdev list –sid xx -R2symdev list –sid xx –bcv


15


8. Create a device group for SRDF/A devices Use command, add devices in device group, show groupname, group type, SymmID, number of devices, a list of them, status of RW, device group RDF info, R1 to R2 relationship, list of devices, link state and RDF pair state.

symdg list (Symdg list will show the device group on that symm)symdg create DgName –type RDF1 or RDF2, [RDF1 is source, RDF2 is target])symld –g DgName add dev xxx –sid xx -rdfg 5 (add one device at a time for an rdfgroup )symld –g DgName addall dev –range xxx:xxx –sid xx -rdfg 5 (add multiple devices with one command)symrdf - g DgName query –rdfa

Add remote BCVsymbcv –g <groupname> -rdf addall dev –range nnn:nnn –sid xxxx (add multiple devices with one command)

9. Set operational mode Adaptive Copy Disk Modesymrdf –g DgName set mode acp_diskC:\Documents and Settings\root>symrdf -g p599_srdfa set mode acp_disk

Execute an RDF Set 'ACp Disk Mode ON' operation for devicegroup 'p599_srdfa' (y/[n]) ? y

An RDF Set 'ACp Disk Mode ON' operation execution is inprogress for device group 'p599_srdfa'. Please wait...

The RDF Set 'ACp Disk Mode ON' operation successfully executedfor device group 'p599_srdfa'.

10. Start SRDF operations, Resynchronize R1 & R2. Then Validate establish is complete between RDF pairs.

symrdf –g DgName establish (If this is an initial replication, please use –full option)symrdf -g DgName query –rdfa

Run query to see the state of RDF Pairs and make sure invalid tracks <30k before set mode async

C:\Documents and Settings\root>symrdf -g p599_srdfa query -i 30

11. Set operational async mode and when invalid tracks is 0, Enable consistency protectionsymrdf –g DgName set mode asyncsymrdf –g DgName enable

C:\Documents and Settings\root>symrdf -g p599_srdfa set mode async

Execute an RDF Set 'Asynchronous Mode' operation for devicegroup 'p599_srdfa' (y/[n]) ? y

An RDF Set 'Asynchronous Mode' operation execution is inprogress for device group 'p599_srdfa'. Please wait...

The RDF Set 'Asynchronous Mode' operation successfully executedfor device group 'p599_srdfa'.


16


C:\Documents and Settings\root>symrdf -g p599_srdfa enable

Execute an RDF 'Enable' operation for devicegroup 'p599_srdfa' (y/[n]) ? y

An RDF 'Enable' operation execution isin progress for device group 'p599_srdfa'. Please wait...

The RDF 'Enable' operation successfully executed fordevice group 'p599_srdfa'.

C:\Documents and Settings\root>symrdf -g p599_srdfa query -rdfa Split devices so we can validate R2 and not impact R1 production

C:\Documents and Settings\root>symrdf -g p599_srdfa split

********At this time, R1 and R2 are both R/W. To change back to SRDF/A normal operations where R1 R/W, R2 W/D and Link is back to RW, please follow below procedures:C:\Documents and Settings\root>symrdf -g p599_srdfa set mode acp_disk -nopC:\Documents and Settings\root>symrdf -g p599_srdfa establish -nopC:\Documents and Settings\root>symrdf -g p599_srdfa set mode async



The RDF Set 'Asynchronous Mode' operation successfully executedfor device group 'p599_srdfa'.




The RDF 'Enable' operation successfully executed fordevice group 'p599_srdfa'.

12. To display the total cache utilization for all SRDF/A sessions that are active (or have non-zero cache usage)

symstat –sid 1229 –type cache -RepType rdfa –i 30

13. To display cycle information for all SRDF/A sessions that are active (or have non-zero cache usage):

symstat –sid 1229 –type cycle -RepType rdfa –i 30C:\Documents and Settings\root>symstat -sid 1229 -type cycle -RepType rdfa –i 30


17


14. Simulate Unexpected Link Failure by force a link suspension to simulate a link drop.symrdf -g DgName suspend –immediate –force

C:\Documents and Settings\root>symrdf -g p599_srdfa suspend -immediate -force

Execute an RDF 'Suspend' operation for devicegroup 'p599_srdfa' (y/[n]) ? y

An RDF 'Suspend' operation execution isin progress for device group 'p599_srdfa'. Please wait...

Suspend RDF link(s).......................................Started. Suspend RDF link(s).......................................Done.

The RDF 'Suspend' operation successfully executed fordevice group 'p599_srdfa'.

To check event with start date of 03/11/06, start time at 15:47:00, end date 03/11/06 and end time at 15:59:00

C:\Documents and Settings\root>symevent list -start 03/11/06:15:47:00 -end 03/11/06:15:59:00 -sid 0027 15. Determine cause of failure

symevent list –start mm/dd/year – end mm/dd/year –sid xx

C:\Documents and Settings\root>symevent list -start 11/10/05:15:47:00 -end 1 1/11/05:15:59:00 -sid 0027

16. Recovery from Unexpected Link Failure. Set mode to adaptive copy disk instead of SRDF/A mode to avoid surge of heavy traffic. Once you have completed the procedures where SRDF/A and Consistency are enable, now you can establish between R2 and Remote BCV.

symrdf –g DgName set mode acp_disksymrdf –g DgName establishsymrdf –g DgName set mode asyncsymrdf –g DgName enable

C:\Documents and Settings\root>symrdf -g p599_srdfa set mode acp_disk -nopC:\Documents and Settings\root>symrdf -g p599_srdfa establishC:\Documents and Settings\root>symrdf -g p599_srdfa set mode async



The device is already in the requested RDF mode




18



The device is already in the requested RDF stateC:\Documents and Settings\root>

Full establish of the remote BCVs and use query to ensure BCV is fully synchronized with R2

symmir –g <groupname> –rdf –full est –exactsymmir –g <groupname> –rdf querysymmir –g <groupname> –rdf verify -synchronized

Split the BCVs from the R2 using the command symmir -rdf -consistent split. symmir -g groupname –rdf split –consistent


19


Alcoa SRDF/A True DR Procedures: How to Failover and Failback where R2 data will propagate back to R1

1. View Symm, links, and RDF pairs info from source or target host. During SRDF/A normal operation, R1 R/W, R2 W/D, Link R/W.

symcfg discoversymcfg listsymrdf pingsymrdf -g DgName verify -consistentsymrdf -g DgName query –rdfa

2. Invoke a checkpoint. Checkpoint waits 2 cycles to ensure last write is issued before the checkpoint is committed to the other side. Split R2 and Remote BCV before failover.

symrdf -g DgName checkpoint3. Disable consistency protectionsymrdf -g DgName disable4. Perform a failover. Switches data processing from the source (R1) to the target (R2) side.

a. Expected Results: RDF ‘Failover’ operation in progress. b. Write disable devices at source (R1)-done. Suspend RDF links – done. c. Read/Write enable devices at target (R2) Operation successful

symrdf -g DgName failover


20


*********Once failover is completed, R1 W/D, R2 R/W, Link N/R. R2 is now ready to use for production. When it’s time to failback, please follow below procedures:5. Check datasymrdf -g DgName query6. Perform a failback. Switches data processing back to the source (R1) side …..@@@

a. RDF ‘Failback’ operation in progress.b. Write disable devices at target (R2)c. Suspend RDF linksd. Merge device track tables startede. Shows device listf. Merge device track tables doneg. Resume RDF linksh. Read/Write enable devices at source (R1).i. Operation successful

symrdf –g DgName failback7. Check datasymrdf –g DgName query8. Be sure Device Group is Async Mode. If necessary, change modesymrdf –g DgName set mode async9. Enable consistency protectionsymrdf -g DgName enable


21


@@@If for reason there are too many invalid tracks to failback, option are to use:1. update until 30000 invalid trackssymrdf -g DgName update -until 300002. When invalid tracks reach 30000, please shutdown all applications that mount to R2

devices3. Be sure that R2 and Remote BCV is synchronized state. Split Remote BCV with R2 to

keep a good copy. Then Place mode into ACP_DISK Modesymrdf –g DgName set mode acp_disk4. Perform a failbacksymrdf –g DgName failback5. Check datasymrdf –g DgName query6. Place mode into Async Modesymrdf –g DgName set mode async7. Enable consistency protectionsymrdf -g DgName enable


22


Database Recoverable Copy From BCV (Apply to Oracle only)

1. Establish 2 sets of BCVs to the target (R2) devicesa. First holds data files

i. If group includes all data, control and redo log files – remote database image restartable

b. Second for archive logs2. Wait until BCVs are fully synchronizedsymmir verify -i 303. Recommended to perform database log switch

a. Minimizes dirty blocks in buffer cache b. alter system switch logfile

4. Place the database in hot backup mode 5. Perform symrdf checkpointsymrdf -g DgName checkpoint

a. Wait for R2 data files to be updated with begin hot backup mark6. Remote consistent split data file BCVs 7. Take database out of hot backup mode8. Force the database to archive current redo log

a. alter system archive log current9. Create a backup control file

a. alter database backup controlfile to …b. NOTE: recommended to create a binary backup controlfile and place it in the

archive logs mountpoint. Otherwise make sure the backup controlfile is available at the remote site for backup/recovery

10. Perform symrdf checkpoint a. Wait for R2 archive logs to be updated with end hot backup mark

11. Remote consistent split archive logs BCVs

EMC SRDF and TimeFinder Script Documentation:EMC SRDF and TimeFinder Script Documentation:


23


Table of Contents

Introduction........................................................................................................25Purpose..............................................................................................................25Environment.......................................................................................................25Installation..........................................................................................................26Details of the Scripts and its functionality......................................................27Script/Function/usage details...........................................................................28Useful Commands.............................................................................................29Troubleshooting................................................................................................29


24


Introduction:This document is prepared for the customers who have little knowledge about the EMC products (Hardware and Software) with systems administration background.

SRDF/A is one of the EMC products from the SRDF family and provides the data replication from the production to the the remote site.

The features of SRDF/Asynchronous are:

1. Less data exposure, i.e., the target is 100% is in consistent with the Source devices.2. Less band width compare to the “Synchronous” data transfer

TimeFinder/BCV is one of the EMC products from the TimeFinder family provides an extra copy of data for DR, backup, development or testing purpose.

The features of TimeFinder/BCV are:

1 Provides an extra copy with out disturbing the production copy for backup or testing purpose.

2 Easy data recovery from the BCV copy in case of data corruption on the production set.

Purpose:

The purpose of this document is to provide details of the scripts used in this environment.

The purpose of the scripts are to monitor the health of SRDF/A and to take a copy of the production data on Gold BCVs.

Environment:

There are two symmetrixs in this environment; one is at Alcoa center in Pittsburgh (0027 – DMX 3000) and another is at Tennessee (1239 – DMX 2000).

Mcode Cache Num Phys Num Symm SymmID Attachment Model Version Size (MB) Devices Devices 000287890027 Local DMX3000 5671 98304 288 4175 000187751239 Remote 2000S-M2 5671 98304 0 2433

The control server in Alcoa center is “adedd46p”

Operating System - HP-UX 11.11EMC software - SYMCLI/V6.0.1


25


Installation:

To run the SRDF/A and TimeFinder/BCV, the following software and the licenses are required;

SYMCLI (V6.0.1) Base License TimeFinder SRDF SRDF/A

We need to create the EMC Device group(s) to make use of these SRDF/A and TimeFinder/scripts (one time only) and also export the Volume groups that contains the application/database/archive logs from the application/database server to the disaster and recovery server (one time only) to verify the copy of R2 and GBCVs.

Note: All the SRDF/A devices are bound to a RDF group number and devices can not swapped from one device group to another device unless changing the RDF group number on the device.

Details of the Scripts and its functionality:

There are four device groups in this environment;

Name Type Valid Symmetrix ID Devs GKs BCVs VDEVs

rdfg01 RDF1 Yes 000287890027 61 0 61 0 rdfg03 RDF1 Yes 000287890027 15 0 15 0 rdfg04 RDF1 Yes 000287890027 3 0 3 0 rdfg02 RDF1 Yes 000287890027 38 0 38 0

All the scripts expecting command line arguments; a Configuration files with all the details and the task that needs to be performed. All these scripts are the wrappers for the “symrdf and symmir” commands and for more options and features of these commands refer to the SYMCLI documentation.

During the script execution time, all the errors related to the individual process will go in to the default log /var/symapi/log/symapi-<date>.log and the SRDF/A and TimeFinder process details in log (grp_name.log) that is defined in the configuration file (/opt/emc/lib/grp_name.conf). The error log contains the details of the process, i.e. hostname, date, time and cause of the problem.


26


How to create the device Groups:

1. Prepare the device list (have two columns (R1, RBCV); Find the R1 s underneath the application/database file systems Find the available GBCVs that were not associated with any R2s on the remote

Symm.symbcv list –sid <SID #> Select the BCVs that has the same size/configuration as the R1/R2s

2. Run the SYMCLI commdns to create the device groups on the control server; symdg create <grp_name> -type RDF1 (source side) symld -g <group_name> -sid <SID #> add dev <DEV #> symbcv -g <group_name> associate dev <BCVDEV #> -rdf

How to find the EMC devices under the file systems (HP-UX): Find all the file systems belongs to the database/application Get the corresponding logical volumes

# bdf <file systems> Get the physical device underneath the logical volume

#lvdisplay –v <logical Volume> | grep –v current | grep /dev/dsk Query the physical volume from the EMC devices list (syminq output)

#syminq >>inq.out#grep </dev/rdsk/cxtxdx> inq.out

Device group verification: symdg list (show all the device groups created on that system) symld –g <group name> list (gives the members of the group) symdg show <group name> (gives all the details including the BCV devices and

device “state” details)

/opt/emc/ is the main directory for the “SRDF and TimeFinder/BCV” scripts, configuration and logs and the structure is as follows;

scripts - Scripts related to SRDF and TimeFinderlib - Configuration files/parameterslogs - SRDF and TimeFinder status logs

/opt/emc/ /scriptsrdfamain

/opt/emc/libmaillist<grp_name>.conf


27


/opt/emc/logs<grp_name>_gbcv.log<grp_name>_mon.logportdown.loglinkdown.logrdfmode.logrdfstate.logavgcyltime.logwpcache.log

Script/Function/usage details:

rdfamain: To monitor/check the link, ports on Source and Target symms, rdf mode and rdf states between R1 and R2 devices and also takes copy of R2 on to GBCV devices using TimeFinder/mirror.

Running “rdfamain” script without any options provides the usage of the script.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

ERROR10: The "configuration" file has not been supplied

USAGE:./rdfamain <config_file> <TASK>Specify the exact path of the configuration file

The valid TASKs are "rdfamonitor" or "gbcvcopy".

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++rdfamain <configuration file> rdfamonitor:

Note: Expecting to run this script in “cron” continuously with an interval defined by the client.

1. This script will check for the connectivity between the Source and Target symmetrix, if they are online, it will check for the individual port status and if they is an error it will sent page/mail to the personal listed in the “maillist”.

2. Check for the RDF mode for “Asynchronous” and the state of the R1 and R2 devices defined in the device group for “consistent” and also the group Consistency Protection. If any one these are not met, it will generate an error sent a page/mail listed in maillist.

3. Once all are working fine, it will check the percentage of Cache utilization and the Cycle time that is taken for each rotation.


28


rdfamain <configuration file> gbcvcopy:

1. The “gbcvcopy” option checks for the initial state of the GBCV devices, if they are in “split” state, it will initiate an “establish” operation otherwise, it will generate an error and “page/mail” the error to the concern listed in “maillist” supplied in the configuration file.

2. After the refresh (“establish”) it will run “srdf/a checkpoint” (to flush the data that is cache to the disk) run the TimeFinder split. Any errors during the process will be logged and also sent to the concern listed in the “maillist”

Note: Expecting to run this script in “cron” and make sure that the GBCVs are un-mount after the backup for the next days “syncup”

Configuration files details:

SRCDIR=/opt/emcLOGDIR=${SRCDIR}/logsETCDIR=${SRCDIR}/libgrp_name=rdfg01LOG=${LOGDIR}/${grp_name}_mon.logBCVLOG=${LOGDIR}/${grp_name}_gbcv.logSSID=0027TSID=1239email_list=${ETCDIR}/maillistmaxpcache=80intvalue=60sync_time=60split_time=60

SRCDIR: Source directory where all the scripts, configuration and log files are available.

LOGDIR: Error and status log filesETCDIR: Configuration files for all the device groupsgrp_name: Device group for which the SRDF/A and TimeFinder operation

carried out.LOG: log file for the monitoring operationBCVLOG: Log file for Glod BCV copy operation.SSID: Souce Symmetrix IDTSID: Target Symmetrix IDemail_list: List of the e-mail/pager address of the concernmaxpcache: Percentage of cache for SRDF/A before raising a flagintvalue: interval value (frequency to check the internal process of “sync” or

split” operations)sync_time: Waiting time for “sync” operation to finishsplit_time: Waiting time for “split” operation to finish


29


Useful Commands:Some useful commands to troubleshoot the “symmir” problems;

<command> -h (Any SYMCLI command follows with “-h” will give you the help/usage)symcfg –h (Shows all the usage and options of symcfg

command)symcfg sync (refresh the local database with respect to the Symmetrix)symcfg discover (to create/recreate symapi_db.bin on the local system)symdg list (to find the device groups existing on the local

system)sympd list –sid <SID #> (to find all the devices that are visible to the system)symdev list –sid <SID #> (to find all the devices that are configured on the Symmetrix)symmir –g <grp_name> query -rdf (to find the details of the R2/GBCVs)symmir –g <grp_name> query –bg -rdf (to find the details of the split that is

happening in the back ground related to R2/GBCVs)

symmir –g <grp_name> verify <state>symrdf –g <grp_name> query (to find the details of the R1/R2s)symrdf –g <grp_name> query –rdfa (to find the details of RDFA devices)

(For more details refer the SYMCLI manual)

Troubleshooting:

All the “SRDF/TimeFinder” related error messages go to either the default log file in /var/symapi/log/symapi.<date>.log or /opt/emc//logs/<grp_name>_gbcv.log and <grp_name>_mon.log

Based on the condition of the devices, take appropriate action (for more details of the conditions, please refer to the manuals)

Recreating the device group:The device groups can be recreated in case of an accidental delete using the import option available with “symdg” command

1. Create an “ascii” file by exporting the existing device group#symdg –f <ascii_file_name> export <device_group>#symdg –f <grp_name>.txt export <grp_name> (to create groups on Source side)#symdg –f <grp_name>_r2gbcv.txt export <grp_name> -rdf (to create groups on Target side)

2. Import the devices from the above file taken when everything is in a working condition.


30


#symdg –f <grp_name>.txt import <grp_name>

Adding new devices in to the existing groups:

The below procedure is useful to add new devices in to the existing groups (RDFA);

SRDF/A operations;

1. Create a file with two columns, R1 and R2 devices (based on the app).2. Execute the RDF commands on the file by specifying the RDFG number where RDFG

number is the one that was given at the time of device creation.symrdf –sid <SSID> -file <r1r2file> -RDFG <#> <task>

a. symrdf –sid 0027 –file <abc123> set mode acp_diskb. symrdf –sid 0027 –file <abc123> -RDFG 2 est –full

Note: By default the new devices are in “sync” mode change them to “acp_disk”3. After the above R1 and R2 devices are in complete “sync”, then

Disable the consistency protection of the device group for which the above R1 and R2 are going to join.

And change the RDF mode from “Asynchromous” to “Acp_disk” Add the above devices to the above group using “symld” Again change the RDF mode from “acp_disk” to “async” (make sure the invalid

tracks are less than 30K) Enable the group Consistency protection.

Working on few devices in an Existing group:

1. Take off the protection (consistency)2. remove those devices from the group3. Execute the tasks 4. Then turn the group into ACP_disk mode and join the devices back5. When the tracks are less than 30K, switch the mode “async”6. Enable the Protection.

Error Codes:

Script name Exit code Description

rdfamain Exit 10 Script usage errorExit 20 Device group parameter is empty in the configuration fileExit 30 Device group specified in Configuration file is not available on the systemExit 50 Configuration file supplied is not available at specified locationExit 21 RDF Link is DownExit 22 RDF ports are OFFLINE


31


Exit 31 Device group is not in "Async" modeExit 32 RDf device pairs are not in "consistent" stateExit 33 Group device protection is not "enabled"Exit 34 Cache used for the SRDF/A is excceded the limit set in the *.conf fileExit 35 Average cycle time exceeded the limit (10 minutes)Exit 36 Checkpoint operation failedExit 12 GBCV devices are not in the "split" stateExit 13 Establish operation failedExit 14 Establish operation "exceeded" the time set in the configuration fileExit 15 "Foreground split" operation failedExit 16 "Background split or exceeded the time set in the configuration file

REFERENCES

Solutions Enabler Release Notes Solutions Enabler Symmetrix CLI Command Reference Solutions Enabler Symmetrix CLI Quick Reference Solutions Enabler Symmetrix Base management CLI Product Guide Solutions Enabler Symmetrix SRDF Family CLI Product Guide Solutions Enabler Symmetrix Timefinder Family CLI Product Guide

Note: The above Product Guides can be downloaded from https://powerlink.emc.com

FOR HARDWARE/SOFTWARE PROBLEMS OPEN SAC CASE

https://powerlink.emc.com

OR

CALL: 800-SVC4-EMC 800-7824-362


32

https://powerlink.emc.com/

https://powerlink.emc.com/