Chicago Tivoli Storage Manager Users Group Meeting April 2010 © 2009 IBM Corporation © 2010 IBM Corporation Chicago Tivoli Storage Manager Users Group

Chicago Tivoli Storage Manager Users Group Meeting April 2010

© 2009 IBM Corporation


Chicago Tivoli Storage Manager Users Group Meeting

April 14, 2010




Agenda Introductions 10:00 - 10:15

TSM 6.2 update 10:15 – 11:30

Customer 6.2 upgrade experience 11:30 – 12:00

Lunch 12:00 – 13:00

FastBack and TSM 13:00 – 14:00

Break 14:00 – 14:15

STORServer 14:15 – 15:00

– FastBack offering

– STORServer reporter

– Live data gathering

– Data mining

– Capacity planning

– Troubleshooting

Getting to the New DB2 Database © 2010 IBM Corporation2




TSM Version 6 Server Database- Getting to the New Database

Chicago Tivoli Storage Manager Users Group Meeting




Agenda

Current Field Status

Storage requirements for the new database and logs

Memory requirements

CPU requirements

User ID requirements

Upgrade Considerations

Examining database and log operations

Backing up the database

© 2010 IBM Corporation4 Getting to the New DB2 Database



Agenda



Memory requirements

CPU requirements








TSM 6.1 Deployments Are Growing

TSM 6.1 GA’d March 2009 with significant new code and functionality

Early input from technical sales teams, support teams, and customers suggests:– TSM 6.1 adoption is growing, but slightly less than the 5.5 rate

– TSM 6.1 field reported problems are in line with prior releases

– Customers are hungry for more information about the new database• what’s new• how to configure it




Critical Issues Fixed in 6.1.2.1

As of November 10, 2009, current fixes address key known issues and continue to drive greater stability– We encourage all customers to install maintenance level 6.1.2.1

• Most current maintenance available as of this date• Available Now

We are excited about TSM 6.1’s enhancements, and remain focused on making our customer base and sales teams successful




Agenda



Memory requirements

CPU requirements








TSM V6 Storage Components

TSM Server

Log Mirror (optional)

Archive Log

Failover Archive Log

(optional)

TSM DB

TSM STGPools (disk, tape)

Active Log




The TSM Database

Implemented via DB2

Specified as a set of directories– DB2 spreads the database across all directories

• Tries to maintain equal distribution across the directories

Be generous in size estimates – plan for growth– If you under allocate, DB2 may need to reorganize the database

• Done transparently, but time consuming

– To add space, you can either• Add directories• Make existing directories bigger

– Suggest you start with many smaller directories and make them bigger as necessary




Size Estimate for V6 Database

Assume 600-1000 bytes per object stored

Deduplication adds 250 bytes per extent per storage pool– Number of extents is approximately

((( Average file size + 256KB ) / 256KB ) + 1) * (Number of Files)

• If average size is 2 MB, this would be:( 2,097,152 + 262,144 ) / 262,144 = 9( 9 + 1 ) * ( Number of Files )

– Calculations based on:• Average extent (“chunk”) size is 256KB• In addition to “data” extents, each file has 1 “metadata” extent• Files between 2KB and 50KB will have 2 extents

– Difficult to determine “average” file size across storage pool




Possible Database Layout – DB2 Managed

LUN 1VG1/db1

50GB

LUN 2VG2/db2

50GB

LUN 3VG3/db3

50GB

LUN 4VG4/db4

50GB

Assume you estimate 200GB for disk space

4 LUNs, each 50GB, assigned from Disk Array

Each LUN is assigned its own Volume Group on host

Each Volume Group has one File System

DB2 will use separate I/O threads for each directory




Extending the DB – Option 1

LUN 1VG1/db1

50GB

LUN 2VG2/db2

50GB

LUN 3VG3/db3

50GB

LUN 4VG4/db4

50GB

Create 2 new 50GB LUNs

Assign to Volume Group and create File System

Assign to TSM

LUN 5VG5/db5

50GB

LUN 6 VG6/db6

50GB

Caution! – DB2 will perform online reorganization

LUN 1VG1/db1

50GB

LUN 2VG2/db2

50GB

LUN 3VG3/db3

50GB

LUN 4VG4/db4

50GB

LUN 5VG5/db5

50GB

LUN 6 VG6/db6

50GB




Extending the DB – Option 2

Create 4 new 25GB LUNs

Extend each file system by 25GB– Can also be done on Windows via “Disk Management”

No need to assign to TSM

No need for DB reorganization

LUN 1LUN 5VG1/db1

75GB

LUN 2LUN 6VG2/db2

75GB

LUN 3LUN 7VG3/db3

75GB

LUN 4LUN 8VG4/db4

75GB




Possible Database Layout – Array Managed

Let the DS8K or other Disk Array manage the LUNs

Assign 1 single 200GB LUN to the host

One Volume Group and one File System

DB2 will use separate I/O threads for each directory

Disk 150GB

Disk 250GB

Disk 350GB

Disk 450GB

Disk Array

LUN 1200GB

Host




DB2 Usage of Array Managed Storage

DB2 has parameter to control parallel I/O– db2set –i <instance> db2_parallel_io

TSM sets this to “*”– Tells DB2 to assume this directory can handle multiple requests

– DB2 default is to use 6 I/O threads• If your hardware can handle 12 concurrent I/O requests, then

– Login as instance user ID, and issue– db2set db2_parallel_io=12

• You may need to restart the TSM server for this to take effect




The TSM Log Directories

Log space is more complicated than with TSM V5– Required Log Directories

• Active Log Directory– Contains all currently active transactions

• Archive Log Directory– Contains all transactions required for DB restore to last backup point

– Optional Log Directories• Active Log Mirror Directory

– Mirror copy of the active log• Failover Archive Log Directory

– Failover directory for archive log

All log directories should be tuned for sequential I/O– No log directory should reside on the same disk or LUN as any

other log or database directory




Log Size Estimates

Active Log Directory– Must be large enough to hold all active transactions

– 2GB minimum, but start with at least twice size of V5 log• If the log fills up, the server halts – don’t underestimate• You can reduce log size with server restart if you over allocate

– This is the highest priority of any log directory• Should be on its own dedicated LUN or disk

Archive Log Directory– Must be large enough to hold all log files generated in 2 days

• Assuming you back up the database daily• DB Backup removes log files from archive directory

– Performance not as critical




Log Size Estimates

Mirror Log (Active Log Mirror) Directory– Same size as active log

– Same performance requirements as active log

Failover Archive Log Directory– Large secondary storage in case archive log directory fills

• Also used if archive log directory unavailable

– Can be a directory in a shared file system (NFS/CIFS)

– If this directory gets used frequently, then archive log too small

– Tune archive log to avoid using failover




Log Size Examples

This techdoc has numerous examples for log sizing:– 1389352: Tivoli Storage Manager V6.1 server might crash if log

is not sized appropriately• http://www-01.ibm.com/support/docview.wss?uid=swg21389352




Preparation - Estimating Disk RequirementsItem Type Same system

MediaSame system

NetworkNew system

MediaNew system

Network

Active Log Disk 2GB (Min) 2GB (Min) 2GB (Min) 2GB (Min)

Log Mirror Disk Log Size Log Size Log Size Log Size

Archive Log (1) Disk Log Size + Log Size + Log Size + Log Size +

V5 DB Disk Current DB Current DB 0 0

V5 Rcvylog Disk Current Log Current Log 0 0

DB2 DB (2) Disk DB Util%+ 50%

DB Util%+ 50%

DB Util%+ 50%

DB Util%+ 50%

DB Backup (2) Seq Media DB Util% DB Util% DB Util% DB Util%

Extract(2) Seq Media DB Util% 0 DB Util% 0

Total Disk Disk

Total Seq Seq Media

Note 1: Archive log is a function of daily activity

Note 2: V6 DB, DBB, and Extract are a function of current DB utilization




Preparation – Sample for 100GB database 80% Utilized

Item Type Same systemMedia

Same systemNetwork

New systemMedia

New systemNetwork

Active Log Disk 32GB 32GB 32GB 32GB

Log Mirror Disk 0 0 0 0

Archive Log(1) Disk 100GB 100GB 100GB 100GB

V5 DB Disk 100GB 100GB 0 0

V5 Rcvylog Disk 13GB 13GB 0 0

DB2 DB Disk 145GB 145GB 145GB 145GB

DB Backup Seq Media 200GB 200GB 120GB 120GB

Extract Seq Media 80GB 0 80GB 0

Total Disk Disk 390GB 390GB 277GB 277GB

Total Seq Seq Media 280GB 200GB 200GB 120GB

Note 1: Archive log consumption changes in V6.1.2




Agenda



Memory requirements

CPU requirements








Memory Requirements in V5

DB buffer pool largest memory consumer– Contained within dsmserv/dsmsvc process

– TSM’s buffer pool not always efficient

Linux and Unix– 1-2G is reasonable

Windows– 1G is reasonable

– Problems with non-paged pool memory in Windows 32-bit




Memory Requirements in V6

DB buffer pool moved outside dsmserv process– Managed by DB2 (db2sysc/db2syscs)

8GB RAM Recommended– DB2 buffer pool is larger

– DB2 more efficient buffer pool management

– DB pages are larger

IPC facilities used for dsmserv/db2sysc communication– Make sure you have high system limits on

• Shared Memory regions• Message Queues




Agenda



Memory requirements

CPU requirements








CPU Requirements for V6

CPU requirements slightly higher than V5

External Functions requiring more CPU include– Deduplication

• SHA-1 and MD5 cryptographic functions highly CPU intensive

– Multi-process Expiration• Trade-off between time and CPU

Internal Functions requiring more CPU include– DB reorganization

– DB runstats• Used to optimize database lookups




Agenda



Memory requirements

CPU requirements








The Instance Model (V5 and V6)

An Instance is everything required to run a server– Database files

– Log files

– Configuration files (dsmserv.opt, volume history, etc.)

– Storage pool volumes

Each instance requires it own directory– dsmserv.dsk is always located in current working directory

– Database and log files usually stored separately

You can have more than one instance per system– Each instance is separate from every other instance




The Instance Model (V6) – Unix

On Unix, each TSM instance requires a unique user ID– Doesn’t need to be dedicated to TSM use, but not a bad idea

• Cannot be root

– Each instance contains 1 TSM database

If you require more than 1 server on a system, you need more than 1 unique instance user ID– TSM Instance name = db2 instance name = instance user ID




The Instance user ID and root – Unix

Instance user ID controls database access– Primary group of instance user ID is administrative group of DB

– Only members of this group can start/stop the database manager• Including root – root must be a member of this group, too

# id tsmuseruid=203(tsmuser) gid=1(staff) groups=0(system),2(bin)# id rootuid=0(root) gid=0(system) groups=2(bin),3(sys),7(security),8(cron)

As tsmuser:

$ db2 get dbm cfg | grep SYSADM SYSADM group name (SYSADM_GROUP) = STAFF

Note: root is not authorized in this example




The Instance Model (V5 and V6) – Windows

Windows has had an instance model for years– dsmserv –k <key>

• Default is “server1”• HKLM\Software\IBM\ADSM\CurrentVersion\Server\<key>

– Model not changed for V6

TSM instances can share a user ID– TSM Server Key = db2 instance name

• For example, “Server1”• Instance user can be any user ID




TSM V6 Storage Components

Log Mirror (optional)

Archive Log

Failover Archive Log

(optional)

TSM DB

TSM STGPools (disk, tape)

Active Log Database Manager(tsmuser)

Configuration Filesdsmserv.opt, trace, devconfig, volhist

TSM Server(tsmuser, root,

orAdministrator)

DBRequests




A Look at the Processes – Unix

db2sysc is the main database manager process

db2acd is the DB2 health monitor

Processes communicate via IPC

IPC

Shared MemMsg Queues

Server

dsmserv

DatabaseManager

db2sysc

tsmuser 365186 209986 0 11:13:36 - 0:00 db2acd 0

tsmuser 246764 209986 0 11:13:35 - 0:25 db2sysc 0

root 328786 107714 0 11:13:34 pts/0 0:03 dsmserv




A Look at the Service List – Windows




A Look at the Service List – Windows




Agenda



Memory requirements

CPU requirements








Preparing for the Upgrade

Read the DB Upgrade Utility README

Obtain the DB Upgrade Utility– Separate from TSM V6 Product

– Always get the latest available from the FTP site

Obtain the TSM V6 DVDs– Back up V5 database before installing V6

References:– DB Server Upgrade Guide (SC23-9554)

– Storage Technical Exchange Website:– http://www-01.ibm.com/software/sysmgmt/products/support/supp_tech_exch.html




TSM V6 Restrictions

Cannot switch platforms using Upgrade Utility– Following architecture upgrades supported

• Windows x86 to Windows x64• Windows IA64 to Windows x64• HP PA-RISC to HP IA64• 32-bit to 64-bit on same platform with same endianness

– For example, AIX 32-bit to AIX 64-bit

Cannot merge multiple V5 databases during upgrade

Cannot alter the underlying DB2 settings– Pre-configured by TSM during installation and format




Agenda



Memory requirements

CPU requirements








The Log Directories (Revisited)

Active Log Directory– Linear (non-circular) fixed-size log in a single directory

• Performance and availability are very important

– Broken up into 512MB files

Archive Log Directory– Contains archives of active log files for database restore

Mirror Log Directory– Mirror of the active log directory – same size

Failover Archive Log Directory– To handle overflow of archive log directory




How the logs flow

Transaction starts and stops are written to active log

Once an active log file is full, it is immediately copied to an archive log directory– If the archive log directory is not writeable, it is copied to the

failover archive log

– If the failover archive log is not writeable, it is not copied

When an active log file has no more active transactions within it, it is eligible for deletion– Cannot be deleted until it is archived




How the logs flow

Txn 1

S0000099.LOG S0000100.LOG S0000101.LOG S0000102.LOG

CurrentFullFullFull

Active

S0000098.LOG S0000099.LOG S0000100.LOG

S0000101.LOG

…

Archive (Now Full)

Failover Archive

Txn 2

Txn 3

Txn 4




Keeping the logs flowing

To keep the active log from filling:– Make sure the active log is large enough to contain active txns

• Watch out for slow clients pinning the log• In TSM V6, there is much more log activity than in TSM V5• If your active log is too small, it will fill and the server will halt

– Make sure the archive log directories have space

To keep the archive log directories from filling:– Perform regular FULL DB backups

• Only FULL DB backups will clear the archive log directories

It may take some time, but if ANY of the log directories becomes full, the server may halt– The reason for the halt will be out of active log space




Database Maintenance Tasks

Table Reorganization– Performed when CPU and I/O activity low

– DB2 optimizes database tables for efficiency

– Generates a lot of active log records• Can interfere with long-running transactions

ANR0293I Reorganization of table <table_name> started.

ANR0294I Reorganization of table <table_name> ended.

Statistics Updates (runstats)– Used by DB2 to optimize TSM’s SELECT statements to the DB

• Improves DB2’s ability to use indices to avoid table scans

ANR0136I Table updating statistics performed successfully for n of n.




Agenda



Memory requirements

CPU requirements








Configuring V6 for DB Backup

V6 database backup uses the TSM client API– API installed automatically with V6 server

– Uses a special nodename “$$_TSMDBMGR_$$” for backup• Password MUST be TSMDBMGR• This node is hidden and can perform only DB backup and restore

– Note: Be careful when canceling sessions. It is possible to cancel the API session doing the DB Backup

V6 database backup and restore require volumehistory and devconfig files

If possible, use either instance config or upgrade wizard– Set the password in TSM.PWD or registry entry

– Sets up dsm.opt, dsm.sys, and DB2 parameters




Processing Flow of DB Backup

TSMDB2

Database

TSMServer

Sequential DataStream(Seq Disk or Tape)

1

1

Intercept Inbound Session from DB2

Stream DB Backup to Sequential DataStream

TSMDB2

Database

TSMServer

Sequential DataStream(Seq Disk or Tape)

2

3

2

3

Initiate DB Backup

4

Volhistory

Volume history file is written4




DB Backup Types

Full Backup– Backs up entire database and all archive logs

• If using FILE device class, stores archive logs on separate volume• If using TAPE device class, appends logs to DB backup volume

– Prunes the archive log directories

Incremental Backup– Backs up all archive logs since last DB backup

• Does not back up the database itself

– Does not prune the archive log directories




Differences in V6 Incremental Backup

Different than V5 incremental DB backup

For V6 DB restore, need– FULL backup,plus

– LAST incremental backup

Sun Mon Tue Wed Thur Fri Sat Sun

Full Inc Inc Inc Inc Inc Inc Full

V5

V6





IMPORTANT FLASHESAs of November 10th, 2009




Important Flashes

1408717: The Tivoli Storage Manager patch 6.1.2.1 is available and recommended for use by all customers– http://www-01.ibm.com/support/docview.wss?uid=swg21408717

1389352: Tivoli Storage Manager V6.1 server might crash if log is not sized appropriately– http://www-01.ibm.com/support/docview.wss?uid=swg21389352

– Contains important log sizing examples

1406035: Active log fills when a transaction runs for a long time– http://www-01.ibm.com/support/docview.wss?uid=swg21406035




Important Flashes

1406032: Tivoli Storage Manager V6.1 server might crash in DB2 operations– http://www-01.ibm.com/support/docview.wss?uid=swg21406032

1394341: Failure during client backup or archive operation when storage destination is DISK storage pool with CACHE=YES enabled– http://www-01.ibm.com/support/docview.wss?uid=swg21394341

1408547: MOVE DATA and MOVE NODEDATA with RECONSTRUCT=NO might cause data integrity issues– http://www-01.ibm.com/support/docview.wss?uid=swg21408547





BONUS SECTIONExample Upgrade Scenario



Upgrade Example

Library Sharing Environment– 2 OS Instances

– Library Manager with a shared tape library

– 2 Library Clients

LANFree– 2 Storage Agents using 1 Library Client

Preparation– Upgrade Servers and Storage Agents to minimum supported

levels to work with New TSM

– Install / Upgrade OS and Hardware as appropriate

– Update SAN Zoning to include new paths




Example: Starting Configuration

Shared Library

SAN Paths

LIBR MGRLIBR CLIENT 1

Server to Server

LIBR CLIENT 2Server to Server

Storage Agents

OS1 OS2

V5 V5 V5




Example Upgrade Scenario – Library Manager

Move Libr Mgr to new OS instance (OS3)

– Update SAN Zoning

– Update Libr Mgr Paths

– Update Libr Client connections

– Validate configuration / paths / connectivity

Upgrade Libr Clients and Storage Agents

Upgrade Libr Mgr to V5.5.2


Upgrade Libr Mgr to New TSM in place

– Small DB, should be fairly easy and fast


!! This is not the only way to upgrade !!




Shared Library

Paths


Server to Server

LIBR CLIENT 2

Server to Server

Upgraded Storage Agents

OS1 OS2OS3

V5 V5New

Example: Library Manager Upgraded

© 2010 IBM CorporationGetting to the New DB2 Database58



Example Upgrade Scenario – Library Client 1

1. Build Libr Client 1 Instance on OS3

2. Upgrade Libr Client 1 using Media Method

– Shutdown Libr Client 1

– Extract Database

– Start Libr Client 1

3. Create Libr Client 3 on OS3

– Insert DB into Libr Client 3 (from Libr Client1 extract)

4. At this point Libr Client 3 is identical to Libr Client1

– Rename new Libr Client1 to Libr Client3





Example Upgrade Scenario – Library Client 1 …

5. Update Connectivity

– Add Libr Mgr Paths for Libr Client 3

– Update Libr Client 3 connections


6. Define connectivity between Libr Client 1 and Libr Client 3

7. Update B/A client connectivity

– Either update clients or update TSM Server and host OS

8. Export/Import from Libr Client 1 to Libr Client 3 using date/time

9. Shutdown Libr Client 1





Example: Library Client 1 Upgraded

Shared Library

Paths


Server to Server

LIBR CLIENT 2

Server to Server

UpgradedStorage Agents

OS3 OS2

New NewV5




Example Upgrade Scenario – Library Client 2

1. Install Upgrade utility on OS2

2. Shutdown Libr Client 2

3. Install New TSM on OS2

4. Create Libr Client 2 DB instance

5. Upgrade Libr Client 2 in place using network method

6. Start Libr Client 2

7. Validate configuration / paths / connectivity

– Libr Mgr

– Storage Agents

– B/A Clients





Example: Library Client 2 Upgraded

Shared Library

Paths


Server to Server

LIBR CLIENT 2

Server to Server

UpgradedStorage Agents

OS3 OS2

New New New



Tivoli Storage Manager TSM V6.2 Technical Overview

David LarimerSenior IT Specialist for Tivoli [email protected]



Functions / Topics Windows Backup/Archive Client Support for GPFS 3.3

Higher Limits on TXNBYTELIMIT and MOVESIZETHRESH parameters

Windows Passthru Driver

Automatic Deployment for Windows Clients

TSM for ERP: Improve TSM Handling of Very Large Objects

Microsoft Hyper-V full Guest Backup Using Volume Shadow Copy Services (VSS)

Simultaneous Write During Storage Pool Migration

Concurrent Access to Centera Volumes

Auto-discovery of New VMware Guests

VCB Gen II - VMware vStorage for Data Protection API integration

Security: In Flight Data Encryption Using SSL

Client-side Data Deduplication

Incremental Backup of Windows System State



Windows Backup/Archive Client Support for GPFS 3.3

TSM already supports GPFS on AIX and Linux

NEW: TSM Windows support for GPFS:– File system recognition

• for Q FILESPACE • for client GUI and client command line

– Backup, Restore, Archive, Retrieve

– Backup and Recovery of security permissions

– Backup and Recovery of GPFS storage pool information

– Support only for Windows Server 2008 x86-64 (no 32 bit support)



Windows B/A Client Support for GPFS 3.3

TSM Windows support for GPFS (cont.):– LAN-Free data movement

– Backupset

– Adaptive Sub-file backup

– Support for non-Administrator credentialed users • must be member of Backup-Operators group

– Cross file system restore • i.e. backup on GPFS and restore to NTFS file system and vice-versa

Not Supported:– Image Backup, Journal Based Backup, Open File Support, HSM,

Cluster Failover, Cross platform restore, Multiuser support, Case Sensitivity



Limits on TXNBYTELIMIT and MOVESIZETHRESH

Limits increased to 32 GB

TXNBYTELIMIT set in dsm.sys (unix) or dsm.opt (Windows)

MOVESIZETHRESH set in dsmserv.opt or with SETOPT command

Can specify TXNB units (default is Kbytes):

– TXNB 300M (option file)

– -txnb=32G (command line)

MOVESIZETHRESH is specified in Mbytes

Use larger settings for large backups with fast tape drives




Why make this change:– Windows Hardware Qualification Lab (WHQL)

• Some companies have security policies which require certified drivers

– Install of TSM device driver generates pop-ups • Driver cannot be installed silently

– TSM device driver cannot be certified with WHQL• Supports non-IBM devices – cannot submit a request with any

non-IBM devices• Uses TSM defined IOCTLs that are not known to Microsoft• Does not support all standard IOCTLs• Supports many old devices – would be impossible to test all




Native device drivers – – A device driver that supports all standard IOCTLs for a

particular device type

– TSM device driver is not a native device driver

– IBM device driver for IBM devices like LTO & 3592 is a native device driver

– Microsoft driver or operating system device driver is a native driver



TSM Server communicates with devices via SCSI passthru interface in device driver

– IOCTL_SCSI_PASS_THROUGH

– IOCTL_SCSI_PASS_THROUGH_DIRECT

TSM device driver now supports these SCSI Passthru IOCTLs

– Native drivers also support these

Users can use either the native device driver or the TSM device driver

TSM Server communicates the same way with devices via both drivers

When upgrade to TSM 6.2, customers can switch to the native driver

– No changes to the existing device definition necessary




Windows Passthru Driver Considerations:

– Users will need to find the native driver for each device• Either use the Microsoft built-in driver• Or get the driver from the device vendor

– Some devices no longer supported by Windows 2003 or Windows 2008 • TSM will continue support of TSM Device Driver • Driver selection can be done a per device basis

Externals Considerations:– New parameter in DEFINE PATH command: GENERICTAPE=YES|NO

(Defaults to No)• Allows users to define a GENERICTAPE type device for either supported or non-

supported drives• Users can define an unsupported drive as GENERICTAPE

– If that drive becomes a supported device, users can continue to use it as GENERICTAPE

• For supported devices, TSM format (such as LTO) is recommended– Use GENERICTAPE=Yes only when the device is not supported by TSM

• The native device driver is required for GENERICTAPE



Windows Passthru Driver Externals Considerations:

GENERICTAPE Value

Device Drive DevType

Yes Supported GENERICTAPE plus a Warning

No Supported Non-IBM Device 4MM, 8MM, DLT, DTF, ECART, LTO, or QIC

No Supported IBM Device 3570, 3590, 3592, LTO

Yes Unsupported Device GENERICTAPE

No Unsupported Device Invalid with New Error Message




Externals Considerations:– Device Configuration File includes GENERICTAPE parameter in

DEFINE PATH commands

– TSMDLST.EXE will change to show the device type for supported drives regardless of which driver is in control. • Unsupported devices will continue to show as GENERICTAPE• TSMDLST output will include a column for Driver type: TSM, IBM, NATIVE, or

N/A

– TSM not able to reset a drive for failover support when using a native device driver• For cases where the TSM Library Manager is on Windows and LUN reset is

required, perform a workaround or use the TSM device driver for at least one device




Addresses need to centrally manage and perform the distribution and upgrade of TSM client code

For TSM 6.2:– TSM administrator able to configure & schedule automatic client

maintenance

– For Windows Backup/Archive client only

– For upgrade from V5.4.X.X (or higher) to V6.2.0.0 (or higher)

– It is the goal to provide upgrade for other platforms and releases in the future

– Requires a TSM 6.2 server




Capabilities from the Admin Center for TSM Admins:– Discover client maintenance levels on the TSM FTP site

– Download needed packages and store them on the TSM server

– Manage packages stored on the TSM server (retention, etc.)

– Select a maintenance level and distribute to a list of clients• The code will then be distributed based on a predefined schedule

– Review the client distribution status




Not Supported in TSM 6.2:– Distribution of other TSM components such as Storage

Agent, Data Protection modules, HSM, etc.

– Distributions on non-Windows clients

– Distribution for downgrade or rollback (e.g. from 6.2.1.1 to 6.2.1.0)

– Ability for clients to auto-discover a new level and upgrade w/o admin action

– Distribution for initial install

– Distribution when the client scheduler is not running

– Use of Admin Center versions prior to 6.2







How it works:– Special objects created on TSM server with name prefix of

IBM_CLIENT_DEPLOY

– Admin Center provides several wizards to configure and perform distribution

– Device Class: FILE type that identifies the location where maintenance packages will be imported from • Admin Center automatically moves packages to this location • Or packages can be manually placed in this location




How it works:– Node: special node name will control the Archive packages that are

created as a result of the Import of packages• A separate node will be create for each platform (only

IBM_CLIENT_DEPLOY_WIN in this release)

– Domain: will hold all the nodes and schedules that are created to support distribution• Admin Center will create the policy structure (mgmt classes, etc.) for this

domain

– Storage Pool: Admin Center will create a dedicated FILE type storage pool to hold the imported packages• Alternatively, an administrator can use an existing FILE or DISK type storage

pool


























































The challenge TSM backups see are:– Customer db are growing to multiple terabytes

– Difficult to cancel a running backup, as it takes a long time to “clean up” the cancelled activity

– The TSM Recovery Log can get pinned while processing very large objects

The solution is to:– Break very large objects into smaller pieces

– Perform TSM db commits as each of these pieces is successfully backed up

TSM for ERP with DB2 segments & manages db backup objects– Does not apply to TSM for ERP with Oracle databases

– Done by the TSM for ERP module and is transparent to the TSM server and to DB2

– TSM for ERP BackOM and Administrative Assistant are changed to report logical backup sizes (as opposed to reporting segment sizes)




A new parameter is introduced for the TSM for ERP profile (initSID.utl)

SEGMENTSIZE <size> [GB | TB]– All Backup objects broken into units equal to SEGMENTSIZE

• Doesn’t apply to Log archives

– A TSM db transaction commit issued after each segment is successfully transmitted• Frees resources on the TSM server (particularly Recovery Log entries)

TSM for ERP uses a suffix on the object name– Helps keep track of the all the segments for a given db backup on the TSM server:

<DB2 instance>.<db alias>.<type>.<partition num>.<DB2 backup id>.<session num>:<segment num>

– Works well with DB2’s object delete mechanism which is based on a filter:

<DB2 instance>.<database alias>.<backup type>.<partition number>.<DB2 backup timestamp>.*

TSM for ERP uses a special zero-length object to store metadata about the set of backup segments:

<DB2 instance>.<db alias>.<type>.<partition num>.<DB2 backup id>.<session num>:C<last segment num>– Written after all segments are successfully transmitted– Used to determine # of segments to ensure the integrity of the restore



Microsoft Hyper-V Full Guest Virtual Machine Backup Using Volume Shadow Copy Services (VSS)

Windows Server 2008 Hyper-V

– Hypervisor implementation from Microsoft –

– Follow-on to Microsoft Virtual Server 2005




Windows Hyper-V requirements:– 64-bit processor (no 32-bit or Itanium versions) (Guests can be 32-bit)– Dual-Core (or more) processor– Hardware-assisted virtualization (Intel VT, or AMD-V)– Hardware enforced Data Execution Prevention (Intel XD bit or AMD NX bit)

Windows Server 2008 Core installation:– Server Core is an install with limited subset of functions – can create slim install– Can include Hyper-V function

• Does not include graphical interface (command line administration only)

– TSM command line (no GUI) supports Server Core implementations (including Hyper-V functions)

Windows Volume Shadowcopy Service (VSS)– VSS creates snapshots of the running Hyper-V guest virtual machines – TSM Hyper-V support is built on top of the existing TSM VSS support– VSS snapshot awareness is propagated to apps (i.e. Exchange, SQL…) within the guest virtual

machine• So even the DB’s inside of the virtual guest are backed up properly with VSS




TSM Hyper-V Guest Backup and Restore

– Only full guest virtual machine support • No visibility to individual files within the guest

– Guest virtual machine is basically a single entity • Large .vhd file with a few small supporting files• Can be backed up from the host operating system if the guest virtual machine is shut down

– VSS is used to create a snapshot of the running guest virtual machine• TSM sends commands to a Hyper-V VSS Writer interface • This Writer propagates the requests to all internal VSS Writers (Exchange, SQL-Server, etc.)

– TSM does not interact with the internal Writers directly• This ensures the integrity of the guest virtual machine and all internal VSS applications

– Cluster Shared Volumes often used with Hyper-V to support live movement of a guest virtual machine from one physical machine to another • TSM’s VSS supports allows Hyper-V guest virtual machine backup where Cluster Shared Volumes are involved• TSM B/A client supports the backup and restore of Hyper-V guest virtual machines



Microsoft Hyper-V Full Guest Virtual Machine Backup Using Volume Shadow Copy Services (VSS) TSM Command Line Client:

– New option on Query VM, Backup VM and Restore VM commants:• VMBACKUPTYPE=HYPERVFULL

TSM GUI Client:– When GUI client detects that it is running on a Hyper-V server:

• GUI will display a list of Hyper-V guest virtual machines that can be backed up or restored• TSM will backup or restore all files associated with a guest virtual machine • TSM server uses the file grouping function to maintain integrity of the guest files

– Hyper-V guest virtual machine backups can create versions– Multiple guest virtual machines can be restored

• All the guest virtual machines for given host are stored under one Node name




Backup



Microsoft Hyper-V Full Guest Virtual Machine Backup Using Volume Shadow Copy Services (VSS) Restore



Simultaneous Write for Client Store Operations

C o p y p o o l 2

C o p y p o o l 1

C lie nt

A c tive -d ata p o o l

S e rve r

P rim ary S to rag e P o o ls

1 . D ata se nt

D ata f lo w

2 . S im ultane o us write to m ultip le targ e ts

3 . M ig ratio n

Data written synchronous ly to primary poo l and one or more copy-pool or ac tive-data-poo l destina tions

Avoids need for subsequent copy opera tions to ac tive-data poo l or copy poo l

Requires that suffic ient tape dev ices be ava ilable during c lient backup

Tape de lays may extend c lient backup window

Not compatible w ith c lient-s ide deduplica tionC urre ntly A vailab le



Simultaneous Write During Migration

C o p y p o o l 2

C o p y p o o l 1

C lie nt

A c tive d ata p o o l

S e rve r

P rim ary S to rag e P o o ls

1 . D ata se nt

D ata f lo w 3 . M ig ratio n

2 . D ata s to re d

3 . S im ultane o us c o p y

Combines windows for migra tion, s torage poo l backup, and copy ac tive data– R e d uc e s to tal tim e fo r the s e o p e ratio ns– F re e s s e rve r re so urce s fo r o the r o p e ratio ns

Compared to exis ting s imultaneous write function – R e d uc e s the ne e d fo r tap e d e vic e s d uring c lie nt s to re o p e ratio ns (b ac kup , archive , c lie nt

HS M)– C an re d uc e c lie nt b ac kup wind o w

E ff ic ie nt use o f tim e and re so urce s T S M 6 .2



Simultaneous Write During Storage Pool Migration Simultaneous Storage Pool Backup at client backup time has been available with

TSM since V5.1– Can slow backup and it can significantly increase tape drive usage

Manual Migration and Storage Pool Backup need to be done in separate windows of time– These windows are getting smaller

– It is more difficult to complete these functions separately

These same issues apply for simultaneous write to Active Data Pools

New feature allows simultaneous Migration & Storage Pool Backup and/or Active Data Pool Copy– Admins can specify up to 3 Copy Pools

– Copies are incremental (no change here)

– The source primary pool can be random or sequential, disk or tape• Must use the NATIVE or NONBLOCK format

– Simultaneous write will not support other data movements• Reclamation, Move Data, Move Nodedata, etc.



Simultaneous Write During Storage Pool Migration

Implementation:– Existing COPYSTGPOOLS & ACTIVEDATAPOOLS parameters of the DEFINE STGPOOL

and UPDATE STGPOOL commands are used• STEP 1: Specify a storage pool value (target pool) on NEXTPOOL parameter for the migration source

pool (not new)• STEP 2: For the NEXTPOOL specify the COPYSTGPOOLS and/or ACTIVEDATAPOOLS parameters

(up to three total) for the migration source pool• STEP 3: Specify a value for AUTOCOPY parameter (this is a new parameter) on target pool:

– NONE – disable simultaneous write altogether– CLIENT – perform simultaneous write only on client backup operations

> Also applies to IMPORT for Copy Storage Pools only> This is the default

– MIGRATION – perform simultaneous write only on migration operations– ALL – perform simultaneous write during both client backup operations

> Including IMPORT for Copy Pools> And during migration operations

– The COPYCONTINUE parameter is not used for simultaneous copy during migration



Simultaneous Write During Migration: Example• C o nf ig uratio n

– P rimary storage poo ls F ILE P OOL and TA P E P OOL form a storage poo l hierarchy– TA P E P OOL enabled for s imultaneous write to C OP YP OOL1 and C OP YP OOL2 during m igra tion (A UTOC OP Y=MIGRA TION)

• B e fo re m ig ratio n– F iles A , B and C on F ILE P OOL are e lig ib le to be m igra ted– A current copy o f fi le C a lready exists on copy storage poo l C OP YP OOL1

• W he n f ile s A , B and C are m ig rate d :– F iles A and B are simultaneously written to TA P E P OOL, C OP YP OOL1 and C OP YP OOL2 – F ile C is s imultaneously written to TA P E P OOL, and C OP YP OOL2 (it is not written to C OP YP OOL1 because it is a lready

stored in that poo l)

B

C

A

TAPE POOL

COPYPOOL2

COPYPOOL1

Nex t pool

FILEP OOL

B

C

A

TAPE POOL

COPYPOOL2

COPYPOOL1

Nex t pool

FILEP OOL

A

B

C

A

B

Before m igration During migration

M igration

Copy

C C



Concurrent Access to Centera volumes

Goal: Provide same concurrent access as done for devclass=file devices

Internal changes only, no externals

With concurrent access, you will have more sessions so consider raising

– MAXNUMMP for clients accessing Centera

– MOUNTLIMIT on Centera device classes

– Note: The sum of all mount limit values for all device classes assigned to the same Centera device should not exceed the maximum number of sessions allowed by Centera.



VMWare Support Enhancements

Auto discovery of New VMware guests

VCB Gen II - VMware vStorage for Data Protection API integration



vCenter Structure

Host

Guest

Folder

{DataCenter

Physical machine

Virtual machine




Uses VMware Infrastructure SDK (VI API)

VMCHOST still specifies where to connect– vCenter or ESX server

ESX password encrypted in registry when using:– GUI preferences editor

– Command “dsmc set password type=vcb ...” Note: Not encrypted if coded directly in options file

TSM connects to VMCHOST and obtains list of all VM guests including– Host ESX server

– Owning folder

– OS type




VCB file level backup requires the TSM admin to define a nodename for the guest and setup proxy node authority

– REGister NOde VMGUEST1 password

– REGister Node PROXYNODE password

– GRant PROXynode TArget=VMGUEST1 AGent=PROXYNODE

Backup vm -vmbackuptype=fullvm

– Backup occurs as all backups use the same Nodename

Backup vm -vmbackuptype=file (Windows only)

– Backup of VM fails if Nodename and Proxy have not been definedANS1662I Agent Node: 'backupproxy1' Target Node: 'shuffle'ANS1532E Proxy Rejected: Proxy authority has not been granted to this node.ANS4150E Incremental backup of Virtual Machine 'shuffle' failed with RC 5722

– These errors appear in server activity log• Visible to TSM Admin without searching client error log



DOMAIN.backuptype option

DOMAIN defines guests to be processed for backups– TSM uses the domain when processing a backup vm command

without specifying which virtual machines to process

Only one DOMAIN statement for each backup type– DOMAIN.VMFULL

• Creates list of VMs eligible for full VM backups

– DOMAIN.VMFILE• Creates list of VMs eligible for file-leve backups



DOMAIN Syntax ALL-VM

– Selects all virtual machine guests found on vCenter or ESX – Not valid for DOMAIN.VMFILE

ALL-WINDOWS– Selects all Windows guests

VMHOST=esx1.ibm.com,esx2.ibm.com– Selects all guests on an ESX host

VMFOLDER=foldername1,foldername2– Selects all guests with a folder

Use only one of the above

VM=guest1,guest2– Adds to list of guests

-VM=guest3,guest4– Excludes from list of guests

Examples– DOMAIN.VMFULL ALL-VM; -VM=test1,test2– DOMAIN.VMFILE VMHOST=esx1.ibm.com;VM=guest99– DOMAIN.VMFILE VMFOLDER=PRODUCTION



Auto Discovery requirements

Support– ESX / ESXi – 3.5 and 4.0

– vCenter - vSphere 4 and VI 3.5

Proxy Platforms– Windows Server 2003 (32 bit and 64 bit)

– Windows Server 2003 R2 (32 bit and 64 bit)

– No IA-64bit support

– Windows Server 2008 (32 bit and 64 bit) – new !– Windows Server 2008 R2 (64 bit only) – new !



VCB Gen II - VMware vStorage for Data Protection API Integration VMware Introduced a new interface

– vStorage for Data Protection API

Replaces VCB for file level backups,– No calls to VCBMOUNTER executable

– No need for VCB Framework installation

No mountpoint exposed– Uses a symbolic link in the global name space

– VMBACKDIR not required for filelevel backup

No external changes

vStorage API will detect the best data transfer path - SAN or LAN

VCB still used for FULLVM backups– vStorage and VCB Framework can coexist

– VCB might be withdrawn from market, but will still be available for use



Abstract VMware recently announced end of availability for VMware Consolidated Backup (VCB). Reference: http://app.connect.vmware.com/e/es.aspx?s=524&e=12880125 for VMware's announcement. How does this affect TSM support for backup and recovery of VMware environments?

Solution

This information is provided in reference to VMware's "VMware Consolidated Backup - End of Availability Announcement" which was distributed by VMware through their Partner Newsletter on 02/24/2010. In this statement VMware stated " ... with the release of the next vSphere platform, we will continue to provide the binaries for VCB, but they will not be compatible with the next platform release. We will continue to provide support for VCB on the current vSphere platform ...". Note the following in reference to TSM support for VMware environments:

Tivoli Storage Manager will continue to provide support for backup and recovery for the current VMware Virtual Infrastructure and VMware vSphere platforms. Refer to the following table for details and software prerequisites based on your current TSM and VMware environments and current method for backup and recovery.

VMware VCB



Security: In Flight Data Encryption Using SSL

New for TSM V6.2– Provide SSL security to multiple platforms

– Extend Certificate support to externally signed certificates • Verisign, Thawte, etc.• Or an internal Certificate Authority

Two Supported Certificate types:– Self-signed

• As used in TSM V5.5 and TSM V6.1

– Externally signed• Either Well-Known Certificate Authority or not Well-Known Certificate

Authority• Can create your own with various utilities



Enhanced SSL Support for Deduplicated Data• TSM d ed u p lication an d en cryp tion

– TSM clien t-s id e an d server-sid e d ed u p lication are b oth in com p atib le w ith clien t en cryp tion

– E ith er can b e u sed w ith SSL en cryp tion• Exten d ed SSL p latform su p p ort

– W indows (ava ilable in TSM 5.5)– AIX (ava ilable in TSM 5.5)– L inux– Solaris– HP-UX

• Altern atives for valid ation of TSM server certificates– Manual, secure dis tribution o f se lf-s igned certifica tes (ava ilable in TSM 5.5)– Acquire certifica tes s igned by well-known certifica te authority such as Thawte or

Veris ign– Use certifica te s igned by customer’s own certifica te authority

2 5 6 -b it A E S e nc ryp tio n fo r in-f lig ht d ataC o m p atib le with T S M s e rve r- o r c lie nt-s id e d e d up licatio nS im p lif ie d d e p lo ym e nt and valid atio n o f T S M se rve r c e rtif ic ate s T S M 6 .2

C lie nt S e rve rS S L allo ws the e ntire c lie nt/se rve r s e s s io n

to b e wrap p e d in an e nc ryp te d tunne l



Security: In Flight Data Encryption Using SSLInstallation & Support

V6.2 installation packages include all requirements

– GSKit is included in the distribution

– V7 or V8 – depending on platform

Only Client to Server sessions are supported

Not supported sessions:

– client to client

– web browser to client

– server to server

– stagent to server

– admin center to server



Security: In Flight Data Encryption Using SSLServer Implementation Customize server options for SSL

– Server Options• SSLTCPPORT• SSLTCPADMINPORT• SSL and non-SSL ports must be different

Restart TSM server to generate the keyring file Issue SET SSLKEYRINGPW

– To create a useful password

Obtain Certificate from CA– Or generate your own– Install ‘root certificate’ if not well-known CA

Install Certificate– Set as Default certificate

Issue QUERY SSLKEYRINGPW – To review



Security: In Flight Data Encryption Using SSLClient Implementation

Client option file– SSL YES

– Set TCPPORT to match server SSLTCPPORT

For Well-Known Certificate Authority– No further action (root certificates are installed with the TSM

client)

For not Well-Known Certificate Authority– Install the root certificate on the clients



Comparison of TSM Data Reduction Methods

YesYesNoNoNoR em oves redundant data

for files from different c lient nodes ?

YesYesNoNoNoAvoids s toring identic al

files renam ed, c opied, or reloc ated on c lient node?

R edundant data from any files in

s torage pool

R edundant data from any files in

s torage pool

S ubfiles that do not c hange

betw een bac kups

F iles that do not c hange betw een

bac kups

R edundant data w ithin s am e file on

c lient nodeS c ope of data reduc tion

B ac kup, arc hive, AP I

B ac kup, arc hive, HS M, AP I

B ac kup (W indow s only)B ac kup

B ac kup, arc hive, HS M, AP ID ata s upported

YesNoYesYesYesC ons erves netw ork

bandw idth?

C lient and s erver elim inate

redundant data c hunks

S erver elim inates redundant data

c hunks

C lient only s ends c hanged s ubfiles

C lient only s ends c hanged files

C lient c om pres s es files

How data reduc tion is ac hieved

C lie nt-s id e d e d up lic atio n

S e rve r-s id e d e d up lic atio n

S ub f ile b ackup

Inc re m e ntal fo re ve r

C lie nt c o m p re ss io n

All o f these data reduction methods conserve s torage poo l space

A vailab le prio r to V 5 A vailab le 6 .1 A va ilab le 6 .2



Client-Side Data Deduplication

Overall Goal: Reduce network traffic

– Client-side/distributed deduplication

While providing:– Optimization of the Deduplication processes– Security and data integrity– Activity reporting

Changes: Server, Client, and API* Updated

– Transparent to basic operations– Logical Restrictions– New options / controls

Client-side performs– Data fingerprinting (identification of data extents or chunks)– Data digest generation (hashing)

Client sends hash and unique chunks to server– Chunks can be compressed before sending

*(API client side dedupe coming in 6.2.x)



Client-Side Data Deduplication - Operation

Copy Storage Pool

(non-deduplicated)

File 1A B C

D

E

File 1

File 2

File 3

File 4B EF

F

File 4

1. Client creates chunks

2. Client and server identify which chunks need to be sent

3. Client sends chunks and hashes to server so that it

can represent object in database. Client saves newfound

hash values in local cache.

4. Entire file can be reconstructed during

Backup Stgpool operation to non-

deduplicated stg. pool at a later time

hash

Index

File 4

TSM 6.2 client

Deduplication-Enabled Disk Storage Pool

TSM 6.2 API

Local Cache

Local Cache



Client-Side Data Deduplication - Prerequisites

Client and server must be at version 6.2.0

Client must have the client-side deduplication option enabled

Server must enable the node for client-side deduplication

– DEDUP=CLIENTORSERVER parameter with REGISTER NODE or UPDATE NODE

Storage pool destination for the data must have deduplication enabled

File must not be excluded from client side deduplication processing

– By default all files are included

– See exclude.dedup Client option for details

File must be larger than 2 KB



Client Side Overview

•S e rve r e nab le s e ach c lie nt no d e fo r c lie nt-s id e d e d up lic atio n•C lie nt c o ntro ls whe the r it ac tually use s c lie nt-s id e d e d up lic atio n•C lie nt inc lud e /e xc lud e o p tio ns allo w co ntro l o f c lie nt-s id e d e d up licatio n at the f ile le ve l

S e rve r and c lie nt c o ntro ls o ve r d e d up lic atio n us e d

•C o m p re ss e d and unco m p re ss e d c hunks are inte rchang e ab le (o nly o ne fo rm is s to re d )

•S e rve r e xp and s (d e c o m p re ss e s ) d ata whe n it ne e d s to b e re co ns truc te d , s uc h as fo r b ackup to a tap e c o p y p o o l o r fo r re s to re to a le g ac y c lie nt

C o m p atib ility with c lie nt c o m p re ss io n

•C lie nt m aintains a lo cal cac he o f hash value s•Minim ize s ne two rk c hat and d atab as e lo o kup s d ue to c hunk-ind e x (d e d up lic atio n-ind e x) q ue rie s to the T S M s e rve r

O p tim izatio n

•C lie nt-s id e and s e rve r-s id e d e d up lic atio n share d ata c hunks in p o o l us ing a unif ie d c hunk ind e x in the T S M d atab as e

•C lie nt-s id e and s e rve r-s id e us e the sam e hashing alg o rithm s /p aram e te rs fo r f ing e rp rinting and chunk id e ntif icatio n

D ata c o m p atib ility with s e rve r-s id e d ata d e d up lic atio n

•No p o s t p ro c e s s ing re q uire d af te r d ata is s to re d in d e d up licatio n-e nab le d d isk s to rag e p o o l

In-b and

•R e d uc e s ne two rk traf f ic b y d e d up lic ating d ata b e fo re trans fe r to the se rve r•D ata c an b e se nt f ro m 6 .2 B ackup -A rchive o r A P I c lie nt to 6 .2 se rve r

S o urc e -s id e (c lie nt-s id e )

C om m entsD esign po in t



Client-Side Data Deduplication - Logical Restrictions

LAN-free

Unix HSM

Encryption

– TSM client-side

– Files from known encrypted file systems which are read in their raw encrypted format

– SSL encryption is OK

Sub-file backup

API Buffer copy elimination

Simultaneous Storage Pool Write

If any of these exist, Client-side dedup does not occur



Client-Side Data Deduplication - Options / Controls

Server-side

– Register and Update Node commands include:• DEDUPLICATION=SERVERONLY | CLIENTORSERVER

– Copygroup destination must be• Stgpool with DEDUPLICATE=YES

Client-side

– DEDUPLication YES | NO

– Include.dedup and exclude.dedup

– ENABLEDEDUPCAche YES | NO

– DEDUPCACHEPath directory_name

– DEDUPCACHESize 256 (1 to 2048 in MB)



Client-Side Data Deduplication - Include / Exclude Controls

Include.dedup filter Exclude.dedup filter

Objtype parameter– Include and Exclude may also specify:

• File• Image• SYSTEMObject• SYSTEMState

Examples– include.dedup /Users/Administrator/Documents/.../*– include.dedup objtype=image E:– include.dedup objtype=systemobject ALL– include.dedup objtype=systemstate ALL



Client-Side Data DeduplicationPreferences Editor & Include / Exclude controls



Client-Side Data Deduplication - Optimization

Local cache for hash index

– Client controlled size (around 200mb for 500gb of backup data)

– Cleared when integrity is questioned

– Uses system cache memory

Local cache similar to JBB and SnapDiff implementations

Additional session for digest queries to server

CPU increase of 3x

Elapsed time upto 2x shorter if large % of duplicate data found

I/O and Memory are not significantly different



Client-Side Data Deduplication and Compression

New chunks can be compressed Process:

– Client chunks the data– Client compresses each chunk– Data is stored in compressed format– During a restore operation, compressed chunks sent back to client

Compressed chunks shared across compressed/not compressed objects Server will decompress on way back to lower level client Server Operations

– BACKUP STGPOOL – chunks decompressed first– BACKUPSET and EXPORT

• Data is decompressed during a backup set generation or an export operation, if the node for which the backup set or export being generated is TSM 6.1 or prior.

• For TSM 6.2 and later clients, the chunks in backup set volumes and export volumes are not decompressed.



Client-Side Data Deduplication Reporting

B/A ClientTotal number of objects inspected: 2

Total number of objects backed up: 2

Total number of objects deduplicated: 2

Total number of bytes inspected: 7.22 MB

Total number of bytes transferred: 0 B

Deduplication reduction: 100%

Total data reduction ratio: 100% API Client

– Extended “dsmEndSendObjExOut”– DP Products not exploiting deduplication reporting at GA release

Server– Additional ANEnnnn client messages written to activity log



Client-Side Data Deduplication Reporting



Client-Side Data Deduplication Integrity Hash collisions

– Generate MD5 hash on the entire file– Save on server– Return to client on restore / retrieve– The file is not restored if the 2 digests do not match

Protection against security attacks– Such as:

• Spoof hash values to gain access to chunks• Store hash values that do not match data

– Protect by:• SHOW commands show limited output• Chunk size stored along with hash value• Query must be followed by store, or protocol violation• SET DEDUPLICATIONVERIFICATIONLEVEL nn (nn>0)

– If failure to verify, NODE updated to DEDUPLICATION=SERVERONLY– Message ANR2695E issued



Client-Side Data Deduplication - Considerations

Separate session for chunk lookups on the server– One extra session per TSM Client consumer or per TSM API thread

– Consider increasing server MAXSESSIONS parameter on the server

File space, backup, or archive deletions may result in the deduplication cache being out of synch with the server– TSM Client detects the condition and resets deduplication cache

– Failed transaction is re-tried

The same deduplication cache cannot be used from multiple processes– Second process will not use the deduplication cache and will query the server for

chunks

– Different processes can use separate cache files by using DEDUPCACHEPATH option



Client-Side Data Deduplication - Considerations

ENABLEDEDUPCACHE option defaults:– YES for BA client – BA client can recovery from invalid cache

– NO for API – minimize txn aborts due to hash lookup failure

DP products not updated for client-side dedup– DP for Oracle and DB2 may start multiple sessions

– DP for SQL and Exchange legacy backups can be large transactions

– DP for SQL and Exchange VSS backups use BA client

Initial storage pool must be sequential disk– Should not migrate objects to tape quickly

• Migration can invalidate client dedup cache

– Consider device class mountlimit setting

– Number of volumes must support number of backup sessions



Incremental Windows System State Backup

Current Problem:

– Any change in the system state cause a full backup of the system state

– While Windows 2003 system was relatively inactive, Windows 2008 system state changes much more often

– Windows Vista/7/2008 can generate 50,000+ files and over 7 GB of system state data

What is New:

– TSM only backs up changed files in the “system files” component of system state (typically a very large component)

– TSM will use backup grouping functions to manage the versions of the system files

– No new externals except that customer can assign System State backup to a Management Class with “mode=absolute” to force a full backup

– Can use TSM 6.2 client with TSM 5.5 or TSM 6.1 server



Incremental Windows System State Backup

Backup Versioning:

Backup Day 1 Backup Day 2



V6.2 Summary Performance

– Limits on TXNBYTELIMIT and MOVESIZETHRESH– TSM for ERP: Improve TSM Handling of Very Large Objects– Simultaneous Write During Storage Pool Migration– Concurrent Access To Centera Volumes– Progressive Incremental for Windows System State

Function– Windows B/A Client Support for GPFS 3.3– Automatic Deployment for Windows Clients– Security: In Flight Data Encryption Using SSL– Client-side Data Deduplication– Windows Passthru Driver

Virtualization– Microsoft Hyper-V Full Guest Virtual Machine Backup Using Volume Shadow Copy

Services (VSS)– Auto Discovery Of New Vmware Guests– VCB Gen II - VMware vStorage for Data Protection API integration


© 2009 IBM CorporationGetting to the New DB2 Database © 2010 IBM Corporation146

Thank You