„Tape configuration for TSM“ - exstor.de · Working with nonIBM devices Which driver? TSM passthru driver, „a forensic analysis ... Tivoli Storage Manager: Future Expectations

TSM Symposium 2013: Tivoli Storage Manager: Future Expectations – Partners and Exhibitors

17.-20. September 2013, Hilton Hotel Gendarmenmarkt, Berlin, Germany

„Tape configuration for TSM“

Bruno Friess



„Tape is dead.“Why do I have to give this presentation?

Agenda:Working with IBM devices

● Atape driver, Install, configure and Atape log● Lin_tape driver, compile, configure, understand● Path failover

Working with nonIBM devices● Which driver?● TSM passthru driver, „a forensic analysis“● ACSLS connectivity

Problem with persistent binding● Linux and udev rules

Hardware device changes



IBM Atape

Where to download?http://ftp.software.ibm.com/storage/devdrvr/ is unfortunately empty.You have to use IBM FixCentral.

You can download one binary file, generate the .toc file and start installation.

> installp -acYX Atape.12.3.7.0.bin

> lslpp -l |grep -i atape Atape.driver 12.3.7.0 COMMITTED IBM AIX Enhanced Tape and

Reboot or not? Bosboot verification.Be careful: An Atape update will unconfigure/configure the drives.

Never use doublezeros!11.2.0.0 or 12.6.0.0

http://ftp.software.ibm.com/storage/devdrvr/



IBM Atape

What is where?> lslpp -f Atape.driver Fileset File ---------------------------------------------------------------------------- Atape.driver 12.3.7.0 /usr/lpp/diagnostics/bin/uAtape /usr/lpp/Atape/install.files/Atape.trcdel.trc /usr/lpp/Atape/install.files/Atape.errmsg_del /usr/lpp/diagnostics/da/d7332 /usr/lpp/diagnostics/da/dAtape /usr/lpp/Atape/install.files/Atape.errdel /usr/lpp/Atape/install.files/Atape.odmdel /usr/include/sys/Atape.h /usr/include/sys/Atape_pr.h /usr/lpp/Atape/instAtape

Is a good tool for various things:syntax: instAtape -a | -c | -d | -q | -u | ? | -ton/off [-D]

-a Configures alternate pathing to yes for all devices -c Configures all devices -d Deletes all devices -q Queries all devices and returns totlal number of devices -u Unconfigures all devices -? Displays this help information -ton Sets trace logging on for all devices -toff Sets trace logging off for all devices -D Used with -ton to set debug trace logging on for all devices



IBM Atape

What is the result after installation?> lsdev -Cc tapesmc0 Available 02-08-02 IBM 3584 Library Medium Changer (FCP)smc1 Available 02-09-02 IBM 3584 Library Medium Changer (FCP)rmt0 Available 02-08-02 IBM 3580 Ultrium Tape Drive (FCP)rmt1 Available 02-09-02 IBM 3580 Ultrium Tape Drive (FCP)rmt2 Available 02-08-02 IBM 3580 Ultrium Tape Drive (FCP)rmt3 Available 02-09-02 IBM 3580 Ultrium Tape Drive (FCP)

Details of the drives:> lscfg -vl rmt0 rmt0 U78A0.001.DNWH9CA-P1-C4-T1-W50050763004FD001-L0

IBM 3580 Ultrium Tape Drive (FCP)

Manufacturer................IBM Machine Type and Model......ULT3580-TD2 Serial Number...............1110012395 Device Specific.(FW)........73V1

> lsattr -EHl rmt0attribute value description user_settablealt_pathing no Enable Alternate Pathing Support Trueautoload no Use Autoloading Feature at End-of-Tape Trueblock_size 0 Block Size (0=Variable Length) Truecompress yes Use Hardware Compression on Tape Truedebug_trace no Debug Trace Logging Enabled Truemax_log_size 500 Maximum size of log file (in # of entries) Trueprimary_device rmt0 Primary Logical Device Falsetrace_logging yes Trace Logging Enabled Trueww_name 0x50050763004fd001 World Wide Port Name False

Serial number

Firmware

Drive type

WWNAIX FC Adapter

Device name



IBM Atape

Same thing for the changer / library:> lscfg -vl smc0 3584LTO4-A U78A0.001.DNWH9CA-P1-C3-T1-W50050763004FD00B-L1

IBM 3584 Library Medium Changer (FCP)

Manufacturer................IBM Machine Type and Model......03584L32 Serial Number...............0000000137920402 Device Specific.(FW)........8980

and attributes:> lsattr -EHl smc0attribute value description user_settablealt_pathing no Enable Alternate Pathing Support Truebusy_retry no SCSI Status Busy Retry Truedebug_trace no Debug Trace Logging Enabled Truedevtype 03584L32 Device Type Falselocation 07-00-02-PRI Location Truenew_name New Logical Name Trueprimary_device smc0 Primary Logical Device Falsetrace_logging yes Trace Logging Enabled Truetsm_barcode_len 6 Default TSM Barcode Length for Ultrium 1/2 Media Trueww_name 0x50050763004fd00b World Wide Port Name False

tsm_barcode_len● Default is 6 after installation, and sometimes after an update !!!● Change this value with chdev -l smc0 -a tsm_barcode_len=8● If the setting is wrong on AIX, TSM will not find the tapes● This setting is for all IBM devices, including LTO 16 and 3592 media



IBM Atape

Dumps and Traces are located in /var/adm/ras/var/adm/ras > ls -l *dump*-rw-r--r-- 1 root system 1306726 Apr 24 10:08 Atape.rmt1.dump1-rw-r--r-- 1 root system 1306726 Jul 03 08:08 Atape.rmt1.dump2-rw-r--r-- 1 root system 1306726 Jan 24 2013 Atape.rmt4.dump1

/var/adm/ras > ls -l *trace*-rw-r--r-- 1 tsm0 tsmsrv 282999 Jul 23 17:52 Atape.rmt0.trace1-rw-r--r-- 1 tsm0 tsmsrv 127760 Jul 03 11:42 Atape.rmt2.trace1-rw-r--r-- 1 tsm0 tsmsrv 127760 Jul 03 11:42 Atape.rmt2.trace2-rw-r--r-- 1 tsm1 tsmsrv 400096 Jul 16 11:45 Atape.rmt1.trace1

Recommendation:1. Switch Tracing off

instAtape -toff chdev -l rmt0 -a trace_logging=no

2. Change device names (use same names in Fabric, AIX and TSM)chdev -l rmt0 -a new_name='lto4-dr01'

> lsattr -EHl lto4-dr01debug_trace no Debug Trace Logging Enabled Trueprimary_device lto4-dr01 Primary Logical Device Falsetrace_logging no Trace Logging Enabled True

The new names will stay with next Atape update.But 15 characters maximum.Think before you type (type mismatch, several paths to the same device...)

Max. 3 dumps per drive.

Message in AIX errorlog:dev_dump_retrieved



IBM Atape

Hints and tricks:1. An Atape update will keep changes of device attributes, including new names. But be aware, that an Update or the use of the tool instAtape will unconfigure/configure the drives No TSM Mount please!→

2. Persistent namesAs long as no target device or no adapter has changed, the devices will stay in the same order. They will be there after each reboot or cfgmgr.Do not use TSM SANdiscovery.

3. ITDT – IBM tape diagnostic toolIs a replacement for tapeutil and offers various functions: health check, firmware update,compression/performance verification. But please not in parallel to TSM!!

4. TSM driver tools: lbtest, mttest, optestAre working with IBM devices as well. When you open a drive, it will have a SCSI reservation and TSM cannot access that drive. The tools are unsupported.

5. When you (or IBM) are updating the library / drive firmware you should plan to update Atape as well. Downgrades are possible.



IBM lin_tape

Download via IBM FixCentral.The package is a linux source package and must be rebuilded for the specific kernel level.Read the installation requirements (missing linux modules!)

Install procedure:● Copy the rpm source package into any folder● Invoke rpmbuild –-rebuild lin_tape-1.1.76.0-1.src.rpm● In the output you will see „Wrote: /usr/src/redhat/RPMS/i386/lin_tape-1.1.76.0-1.i386.rpm“● Run rpm -ivh /usr/src/redhat/RPMS/i386/lin_tape-1.1.76.0-1.i386.rpm● Run rpm -ivh /usr/src/redhat/RPMS/i386/lin_taped-1.1.76.0-1.i386.rpm for the lin_tape monitoring Daemon● That's it!

The lin_tape daemon is located in /usr/bin and you can start/stop it with: /usr/bin/lin_taped start/stop/restart/status

The Log files of lin_taped are in /var/log/lin_tape.errorlog.The configuration is done with its config file /etc/lin_taped.conf.

The lin_tape daemon is needed for:Device driver tracing, obtaining drive data, automatically storing drive dumps, saving log senseinformation and path failover decision.



IBM lin_tape

After the installation the lin_tape driver will be (re)startet via reboot or:1. lin_taped stop (stops the lin_taped daemon)2. rmmod lin_tape (unload of lin_tape)3. Time for changes in the config e.g. /etc/modprobe.conf4. depmod5. modprobe lin_tape (reload of lin_tape)6. lin_taped start (restart lin_taped daemon)

Important: Create a new file in /etc/modprobe.d/etc/modprobe.d > vi lin_tape.confoptions lin_tape tape_reserve_type=persistentoptions lin_tape lin_tape_debug=0

Save this file and restart lin_tape (see SWG 21610778)./sys/module/lin_tape/parameters # ll-r--r--r-- 1 root root 4096 Dec 12 10:44 alternate_pathing-r--r--r-- 1 root root 4096 Dec 12 10:44 auto_logging-r--r--r-- 1 root root 4096 Dec 12 10:44 changer_open_reserve-r--r--r-- 1 root root 4096 Dec 12 10:44 default_sys_encryption_proxy-r--r--r-- 1 root root 4096 Dec 12 10:44 default_sys_encryption_write-r--r--r-- 1 root root 4096 Dec 12 10:44 disable_density_on_open-r--r--r-- 1 root root 4096 Dec 12 10:44 dynamic_attributes-r--r--r-- 1 root root 4096 Dec 12 10:44 ibm3592_barcode-r--r--r-- 1 root root 4096 Dec 12 10:44 lin_tape_debug-r--r--r-- 1 root root 4096 Dec 12 10:44 lto_barcode-r--r--r-- 1 root root 4096 Dec 12 10:44 tape_reserve_type-r--r--r-- 1 root root 4096 Dec 12 10:44 wait_on_config

And as result: /sys/module/lin_tape/parameters # less tape_reserve_type

persistent

Modprobe will fail, when:/etc/modprobe.d/unsupported_modules is 0



Path Failover

The IBM path failover feature (DPF – data path failover and CPF – control path failover) requiresa separate license in the libraries. If the license is not available and you configure failover, the devicewill go offline immediately.

Switch on DPF / CPFWith instAtape -a (when nothing is mounted)chdev -l smc0 -a alt_pathing=yes (CPF)chdev -l rmt0 -a alt_pathing=yes (DPF)lsdev -Cc tape | awk '{print $1}' | xargs -n1 -I dr chdev -l dr -a alt_pathing=yes

You will get:smc0 Available 02-08-02-PRI IBM 3584 Library Medium Changer (FCP)smc1 Available 02-09-02-ALT IBM 3584 Library Medium Changer (FCP)

Edit the file /etc/modprobe.d/lin_tape.confoptions lin_tape alternate_pathing=1

Enables DPF and CPFoptions lin_tape alternate_pathing=0

Switch it off.cat /proc/scsi/IBMchanger*lin_tape version: 1.76.0Attached Changer Devices:Number model SN HBA SCSI FO Path0 03584L22 0000078A04250401 lpfc 7:0:1:1 NA1 03584L32 0000078143890401 lpfc 7:0:2:1 NA

NA = not availableON = good



NonIBM devices

You have to use the TSM drivers, included in the TSM server package,Fileset tivoli.tsm.devices.rte and tivoli.tsm.devices.acsls, installed in /opt/tivoli/tsm/devices/bin and /usr/lib/drivers directory.

Smooth integration into AIX and smit (through /usr/lib/methods/cfgtsmdd).lsdev -Cc librarylsdev -Cc adsmtape

This will list the lb and mt devices. These device files are real ones, not just a link.

Persitent binding works similiar as for IBM devices: no change, then the devices will stay with reboots or cfgmgr.



NonIBM devices

You have to use the TSM passthru driver on Linux.Included in the TSM server package, installed in /opt/tivoli/tsm/devices/bin directory:

/opt/tivoli/tsm/devices/bin # ls -l-rw-r--r-- 1 root root 12462 May 1 04:24 README.htm-rwxr-xr-x 1 root root 63915 May 1 04:24 autoconf-rwxr-xr-x 1 root root 25722 May 1 04:24 ddinfo-rwxr-xr-x 1 root root 1178 May 1 04:24 kill.acs_ssi-rw-r--r-- 1 root root 1357 May 1 04:24 lb.conf.smp-rwxr-xr-x 1 root root 1214262 May 1 04:24 lbtest-rwxr-xr-x 1 root root 17896 May 1 04:24 mini_el-rw-r--r-- 1 root root 1344 May 1 04:24 mt.conf.smp-rwxr-xr-x 1 root root 1250570 May 1 04:24 mttest-rw-r--r-- 1 root root 1350 May 1 04:24 op.conf.smp-rwxr-xr-x 1 root root 1221497 May 1 04:24 optest-rwxr-xr-x 1 root root 11924 May 1 04:24 rc.acs_ssi-rwxr-xr-x 1 root root 317049 May 1 04:24 ssi-rwxr-xr-x 1 root root 9707 May 1 04:24 t_parent-rwxr-xr-x 1 root root 73986 May 1 04:24 trace_decode-rwxr-xr-x 1 root root 2482 May 1 04:24 tsmscsi

Passthru? Because there is no tape driver in the package.Two methods for using the „driver“: via autoconf or manually with tsmscsi

for ACSLS

for tsmscsi

for autoconf



NonIBM devices

Working with autoconf on Linux:

Supports three params: a adds read/write permissions for all sg devices for all usersg adds read/write permissions for users in the group of roott turn on tracing (trace output is in autoconf.tar)

What does it do?1. Deletes the directory /dev/tsmscsi2. Scans all sg devices and tries to recognize them as supported tape device3. Writes links into /dev/tsmscsi, example:

lrwxrwxrwx 1 root root 8 Feb 24 18:18 mt0 -> /dev/sg1lrwxrwxrwx 1 root root 8 Feb 24 18:18 mt1 -> /dev/sg2

4. Write the mtinfo file into /dev/tsmscsi, needed for TSM SANdiscovery, example:Index Minor Host CHN ID LUN Type Vendor_ID Device_Serial_Number Product_ID Rev.000 001 003 000 000 000 001 HP HUE11242V3 Ultrium 4-SCSI H64S001 002 003 000 001 000 001 HP HU10569TH7 Ultrium 4-SCSI H58S

Problem with autoconf:● Should be started during each reboot to recognize changes● If there are changes in the fabric or new/changed devices, autoconf must be used again● autoconf doesn't care about persistent binding and TSM SANdiscovery doesn't work at all



NonIBM devices

Working manually with tsmscsi on Linux:

What does it do?1. tsmscsi is a script, you can look into this file :)2. tsmscsi checks if sg is running, if not starts sg, if sg is running it reloads sg, why?3. Uses /proc/scsi/sg/devices as input file and compares it with mt.conf, lb.conf and op.conf4. If config files are missing / empty nothing happens→5. generates the link files in /dev/tsmscsi (=deletion and new write) 6. (over)writes the file mtlist, lblist, oplist in /opt/tivoli/tsm/devices/bin

Example:The mt.info file is: will create links: writes mtlist:

HOST[0]=4 /dev/tsmscsi/mt0 → /dev/sg296 296:4 0 0 0 CHANNEL[0]=0ID[0]=0LUN[0]=0

HOST[1]=4 /dev/tsmscsi/mt1 → /dev/sg307 307:4 0 1 0CHANNEL[1]=0ID[1]=1LUN[1]=0

Problem with tsmscsi:● Does not set the access permissions● Type mismatch in mt.conf = it could be a disk drive or … There is no check!!● tsmscsi doesn't care about persistent binding and TSM SANdiscovery cannot work

because of missing mtinfo file in /dev/tsmscsi



ACSLS attachement

The ACSLS (Automated Cartridge System Library Software) drivers are included in the TSM server package, installed in /opt/tivoli/tsm/devices/bin

To start the drivers just use rc.acs_ssi.The only param is „quiet“. Without quiet set, the program will prompt for the ACSLS host.With the quiet param, the variable ACSLS_SERVER_HOSTNAME must be set.Two programs are started via rc.acs_ssi (the pid's are stored in the file kill_ssi_pid):

ssimini_el

To stop the driver, just use kill.acs_ssi (the file kill_ssi_pid with the pid's must exist).

Start included in the /etc/inittab:ssi:23:once:/opt/tivoli/tsm/devices/bin/rc.acs_ssi quiet

Doing this, you must have the variable set in the rc.acs_ssi file itself or via export in the environment.Problems:

● With the next TSM driver update the file rc.acs_ssi will be overwritten● Where to set the export command? Because rc.acs_ssi is in the /etc/inittab

In TSM you must set: ACSACCESSID name_of_tsm_userACSLOCKDRIVE NOACSQUICKINIT NOACSTIMEOUTX 1RESOURCETIMEOUT 15



Persistent binding

What is the problem?The OS is scanning devices in a different order and therefore the device names will change.This happens especially after a reboot. If this change will happen, then TSM is not able totalk with the right device file.

What does IBM recommend?1. Use the „persistent binding“ setting on the FC adapters. This may work on one adapter, but it will not control the scan order of the adapters in the OS.2. Use of the „SAN device mapping“ function (=SDM) implemented through the SNIA API.More details: IBM support article swg21193154.A specific firmware level on the adapter is required (Qlogic is 8.01 or higher):

Check with modinfo /lib/modules/*default/kernel/drivers/scsi/qla2xxx/qla2xxx.ko→ version: 8.03.07.07-k

This is a combination of TSM SANdiscovery and adapter firmware.....it doesn't work!

One solution which is always working: Build some udev rules and create special, special device files.



Persistent binding

Building udev rules:First you must gather the information of the device files with udevadm or udevinfo (on older Linux releases).For IBM devices you can also use /dev/lin_tape/by-serial

Example (shortened):# udevadm info --attribute-walk --name /dev/IBMtape0Udevadm info starts with the device specified by the devpath and thenwalks up the chain of parent devices. It prints for every devicefound, all possible attributes in the udev rules key format.A rule to match, can be composed by the attributes of the deviceand the attributes from one single parent device.looking at device '/devices/pci0000:00/host2/rport-2:0-2/target2:0:0/2:0:0:0/lin_tape/IBMtape0': KERNEL=="IBMtape0" SUBSYSTEM=="lin_tape" DRIVER=="" ATTR{serial_num}=="000007811620" ATTR{ww_node_name}=="0x500506830F076F22" ATTR{ww_port_name}=="0x500506830F476F22" ATTR{primary_path}=="NA" ATTR{sys_encryption_proxy}=="1" ATTR{sys_encryption_write}=="2"

# udevadm info --attribute-walk --name /dev/IBMchanger1 | grep serial ATTR{serial_num}=="0000078244910401"

# udevinfo -a -p `udevinfo -q path -n /dev/IBMtape0` | grep -e serial_num -e ww_port_name* SYSFS{ww_port_name}=="0x500506830F076F22" SYSFS{serial_num}=="000007811620"

WWN

Serial number



Persistent binding

Building udev rules:With udevadm or udevinfo you can also gather the info for NonIBM devices (see IBM swg21425983).In addition you can use the linux generic driver command: sginfo -s /dev/sg234 which will show the SN.

Now we can build our udev rules in a file named /etc/udev/rules.d/98lin_tape.rulesFor each drive put in one line:

KERNEL=="IBMtape*", ATTR{ww_node_name}=="0x500506830F476F22", MODE="0666", SYMLINK="tape/by-name/drive01"

Kernel module Attribut WWN Access rights Desired name

● Any kernel module and any attribut can be used● The new special device files can be located anywhere (in our example it is: /dev/tape/by-name)● The maximum name in TSM is 64 characters● When the changer is managed over a drive (e.g. with the TS3500),

then the device will report no WWN. Then you have to use the serial number:

KERNEL=="IBMchanger*", ATTR{serial_num}=="0000078244910401", MODE="0666", SYMLINK="tape/by-name/3584lib1"

● The new special device files will be generated via udev during the next reboot or manually with:udevadm trigger –-sysname-match="*IBM*"

Do not use trigger without any option.



Physical changes of the tape environment

When a library is physically upgraded, for example, when drives, cells or columns are added to a physical library to expand it, the TSM server needs to be notified of these changes.

There are two possibilities:

1. The most common way:● Delete all path, drives and redefine them (if the SAN has changed)● Delete all path, drives and the library (if the number of cells has changed), but consider

the TSM volumes before doing this. And redefine everything.● With some TSM Server versions you have to restart the server.● In some cases it is necessary to delete/define the special device file: rmdev/mkdev on AIX● Put these commands into TSM scripts (def_libr, del_libr, def_path_dr, del_path_dr)

2. The new TSM parameter (with 6.1.5, 6.2.3 or 6.3 and higher):● audit library TS3500 checklabel=barcode refreshstate=yes● This will update the number of cells, will check if drives are deleted or added and for

changes in the element address without reboot (see IBM article swg21203271).● It doesn't work with TS3500 and ALMS you must restart the TSM server. →



Bruno [email protected]: 52.511992,13.392902

mailto:[email protected]

Documents

„Tape configuration for TSM“ - exstor.de · Working with nonIBM devices Which driver? TSM passthru driver, „a forensic analysis ... Tivoli Storage Manager: Future Expectations