GGS WinUnix Admin 10.0

7/30/2019 GGS WinUnix Admin 10.0

1/374

GoldenGate For

Windows and UNIX

Administrator Guide

Version 10.0

GoldenGate Software, Inc.

Transactional Data Management


2/374

Build Date 1/22/09

Copyright 1995 - October 3, 2008 by GoldenGate Software, Inc.

301 Howard Street, Suite 2100

San Francisco, CA 94105

Phone +1 (415) 777-0200

Fax +1( 415) 777-0201

[email protected]

If this guide is distributed with software that includes an end user agreement, this guide, as well as the software described in it, is

furnished under license and may be used or copied only in accordance with the terms of such license. Except as permitted by any

such license, no part of this guide may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means,

electronic, mechanical, recording, or otherwise, without the prior written permission of GoldenGate Software, Inc. Please note that

the content in this guide is protected under copyright law even if it is not distributed with software that includes an end user

license agreement.

All rights reserved. Printed in the United States of America.

Disclaimer

The content of this guide is furnished for informational use only, is subject to change without notice, and should not be construed

as a commitment by GoldenGate Software Inc. GoldenGate Software Inc. assumes no responsibility or liability for any errors or

inaccuracies that may appear in the informational content contained in this guide.

GoldenGate Software Inc. and its distributors expressly disclaim all implied, express or statutory warranties including, without

limitation, the implied warranties of merchantability, noninfringement of third party rights and fitness for a particular purpose.

Neither GoldenGate Software Inc. nor its distributors will be liable for damages, whether direct, indirect, special, incidental, or

consequential, as a result of the reproduction, modification, distribution, or other use of the software or its documentation even if

advised of the possibility of such damages.

Trademarks

GoldenGate, GoldenGate Veridata, and the GoldenGate logo are trademarks of GoldenGate Software, Inc.

Adobe and Acrobat are trademarks of Adobe Systems Incorporated in the United States and/or other countries.

Guardian, HP, NonStop Himalaya, and Tandem are either trademarks, or service marks of Hewlett-Packard Company in the

United States and/or other countries.

DB2, IBM, Informix, OS/390, Red Brick and z/OS are either trademarks, or service marks of International Business MachinesCorporation in the United States and/or other countries.

Microsoft, Microsoft SQL Server, and Windows are either trademarks, or service marks of Microsoft Corporation in the United

States and/or other countries.

MySQL is a trademark of MySQL AB in the USA and other countries.

Teradata is a trademark of NCR in the United States and other countries.

Oracle is a trademark of Oracle Corporation in the United States and other countries.

Sun is a trademark of Sun Microsystems, Inc. in the United States and other countries.

Sybase is a trademark of Sybase, Inc., Concord, Massachusetts in the United States and other countries.

UNIX is a trademark licensed exclusively through X/Open Company Ltd. in the United States and other countries.

All other marks used herein are trademarks or service marks of their respective owners.

Licenses

The software uses the following libraries which are licensed under the GNU Lesser General Public License, version 2.1: Pthreads-

win32 (only applicable for Windows) and Blowfish encryption library. The software is also provided with a number of other

libraries licensed under the LGPL. All of these libraries are available in source code form, along with a copy of the LGPL, at

http://support.goldengate.com. The software also contains or may be provided with other third party code. Such third party code

is identified at http://support.goldengate.com along with the copyright and other legal and licensing notices applicable to such code.


3/374

3GoldenGate for Windows and UNIX Administrator Guide

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Contents. . . . . . . . . . . . . . .

Preface About the GoldenGate guides. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

Typographic conventions used in this manual. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

Getting more help with GoldenGate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

New features in this release of GoldenGate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Chapter 1 Introduction to GoldenGate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

GoldenGate supported processing methods and databases . . . . . . . . . . . . . . . . . . . . . . . . .22Overview of the GoldenGate architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

Overview of processing methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29

Overview of groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30

Overview of the Commit Sequence Number (CSN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31

Chapter 2 Configuring the Manager process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34

Overview of the Manager process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34

Configuring Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34

Recommended parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35

Starting Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36Stopping Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36

Chapter 3 Getting started with GoldenGate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38

Running the user interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38

Using GoldenGate parameter files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40

Chapter 4 Using GoldenGate for live reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47

Overview of the reporting configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47

Considerations when choosing a reporting configuration . . . . . . . . . . . . . . . . . . . . . . . . . . .48

Creating a standard reporting configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49

Creating a reporting configuration with a data pump on the source system . . . . . . . . . .51Creating a reporting configuration with a data pump on an intermediary system . . . .53

Creating a cascading reporting configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57

Chapter 5 Using GoldenGate for real-time data distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64

Overview of the data-distribution configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64


4/374

Contents


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Considerations for a data-distribution configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64

Creating a data distribution configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66

Chapter 6 Configuring GoldenGate for real-time data warehousing. . . . . . . . . . . . . . . . . . . . . . . . . . .70

Overview of the data-warehousing configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70Considerations for a data warehousing configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70

Creating a data warehousing configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72

Chapter 7 Using GoldenGate to maintain a live standby database . . . . . . . . . . . . . . . . . . . . . . . . . . . .76

Overview of a live standby configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76

Considerations for a live standby configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77

Creating a live standby configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79

Moving user activity in a planned switchover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83

Moving user activity in an unplanned failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86

Chapter 8 Using GoldenGate for active-active high availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90Overview of an active-active configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90

Considerations for an active-active configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90

Preventing data looping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93

Creating an active-active configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .96

Managing conflicts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Conflict detection and resolution examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Chapter 9 Configuring GoldenGate security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Overview of security options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Using encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110Generating encryption keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

Using command security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Using target system connection initiation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

Chapter 10 Handling GoldenGate processing errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Overview of GoldenGate error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Handling Extract errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Handling Replicat errors on DML operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Handling Replicat errors on DDL operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Handling TCP/IP errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Maintaining updated error messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Resolving GoldenGate errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Chapter 11 Creating a data-definitions file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Overview of the data-definitions file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Determining if you need to use a data-definitions file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125


5/374

Contents


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Deciding whether to use a definitions template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Configuring a data-definitions file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Chapter 12 Configuring online change synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

Overview of online change synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130Naming conventions for groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

Creating a checkpoint table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

Creating an online Extract group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

Creating a trail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

Creating a parameter file for online extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

Creating an online Replicat group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

Creating a parameter file for online replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Controlling online processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

Deleting a process group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

Chapter 13 Configuring change synchronization as a batch run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

Overview of batch change synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

Creating a parameter file for batch extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

Creating a parameter file for batch replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

Starting batch change synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

Chapter 14 Configuring DDL synchronization for an Oracle database . . . . . . . . . . . . . . . . . . . . . . . . 150

Overview of DDL synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

Supported Oracle objects and operations for DDL replication . . . . . . . . . . . . . . . . . . . . . 150

Non-supported Oracle objects and operations for DDL replication. . . . . . . . . . . . . . . . . 151

Limitations of GoldenGate DDL support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

Special DDL cases and their treatment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

Configuration guidelines for DDL support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

Understanding DDL scopes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

Enabling and filtering DDL for replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

How GoldenGate handles derived object names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

Using DDL string substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

Controlling the propagation of Replicats DDL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Adding supplemental log groups automatically . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Removing comments from replicated DDL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174Controlling whether renames enter the DDL configuration . . . . . . . . . . . . . . . . . . . . . . . . 174

Replicating an IDENTIFIED BY password. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

How DDL is evaluated for processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

Handling Extract DDL processing errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

Handling Replicat DDL processing errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178


6/374

Contents


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Handling DDL trigger errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

Viewing DDL report information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184

Viewing metadata in the DDL history table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

Tracing DDL processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188

Tracing the DDL trigger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188

Chapter 15 Configuring DDL synchronization for a Teradata database . . . . . . . . . . . . . . . . . . . . . . . 189

About this documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

Overview of DDL synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

Supported Teradata objects and operations for DDL replication . . . . . . . . . . . . . . . . . . . 190

Limitations of GoldenGate DDL support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

Configuration guidelines for DDL support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

Understanding DDL scopes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

Enabling and filtering DDL for replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

How GoldenGate handles derived object names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

Using DDL string substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

Controlling whether renames enter the DDL configuration . . . . . . . . . . . . . . . . . . . . . . . . 210

How DDL is evaluated for processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

Handling Extract DDL processing errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

Handling Replicat DDL processing errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

Viewing DDL report information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

Tracing DDL processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219

Chapter 16 Running an initial data load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

Overview of initial data load methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220Using parallel processing in an initial load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

Prerequisites for initial load. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

Loading data with a copy utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

Loading data with Replicat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

Loading data with a bulk-load utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

Loading data with a GoldenGate direct load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233

Loading data directly to SQL*Loader. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

Loading tables with Teradata table copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

Loading data with Teradata load utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249

Chapter 17 Mapping and manipulating data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251

Overview of data mapping and manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251

Deciding where data mapping and conversion will take place . . . . . . . . . . . . . . . . . . . . . 251

Handling anomalies in data from NonStop systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252

Selecting rows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252


7/374

Contents


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Selecting columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

Selecting and converting SQL operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

Mapping columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

Using transaction history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264

Testing and transforming data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266

Using tokens. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270

Mapping and transforming Unicode and native characters . . . . . . . . . . . . . . . . . . . . . . . . 272

Chapter 18 Customizing GoldenGate processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274

Overview of custom processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274

Executing commands, stored procedures, and queries with SQLEXEC . . . . . . . . . . . . . . 274

Using GoldenGate macros to simplify and automate work. . . . . . . . . . . . . . . . . . . . . . . . . 281

Using user exits to extend GoldenGate capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

Using the GoldenGate event marker system to raise database events . . . . . . . . . . . . . . 290

Chapter 19 Monitoring GoldenGate processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301

Overview of the GoldenGate monitoring tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301

Using the information commands in GGSCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301

Monitoring lag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302

Monitoring processing volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303

Using the error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306

Using the process report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307

Using the discard file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310

Using the system logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312

Reconciling time differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313Sending event messages to a NonStop system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313

Getting more help with monitoring and tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314

Chapter 20 Performing administrative operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315

Overview of administrative operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315

Performing application patches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315

Adding process groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316

Initializing the transaction logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323

Shutting down the system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324

Changing database attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324Changing the size of trail files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328

Chapter 21 Undoing data changes with the Reverse utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329

Overview of the Reverse utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329

Reverse utility restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

Configuring the Reverse utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330


8/374

Contents


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Creating online process groups and trails for reverse processing. . . . . . . . . . . . . . . . . . . 335

Running the Reverse utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

Undoing the changes made by the Reverse utility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338

Chapter 22 Creating target objects with DDLGEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339

Overview of DDLGEN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339

Creating a DDLGEN template file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339

Creating a DDLGEN parameter file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339

Running DDLGEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340

Transferring and applying data definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341

Appendix 1 GoldenGate record format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342

Example of a GoldenGate record . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342

Record header area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

Record data area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345

Tokens area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346

GoldenGate operation types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347

Glossary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363


9/374


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

PREFACEAbout the GoldenGate guides. . . . . . . . . . . . . . .

The complete GoldenGate documentation set contains the following components:

NonStop Server

GoldenGate for HP NonStop Administrator Guide: Introduces GoldenGate components

and explains how to plan for, configure, and implement GoldenGate on the NonStop

platform.

GoldenGate for HP NonStop Reference Guide: Provides detailed information about

GoldenGate parameters, commands, and functions for the NonStop platform.

Windows and UNIX platforms

GoldenGate Installation and Setup guides: There is one such guide for each database

that is supported by GoldenGate.

GoldenGate for Windows and UNIX Administrator Guide: Introduces GoldenGate

components and explains how to plan for, configure, and implement GoldenGate on the

Windows and UNIX platforms.

GoldenGate for Windows and UNIX Reference Guide: Provides detailed information

about GoldenGate parameters, commands, and functions for the Windows and UNIX

platforms. GoldenGate for Windows and UNIX Troubleshooting and Performance Tuning Guide:

Provides suggestions for improving the performance of GoldenGate in different

situations, and provides solutions to common problems.

Typographic conventions used in this manual

This manual uses the following style conventions.

Parameter and command arguments are shown in upper case, for example:

CHECKPARAMS

File names, table names, and other names are shown in lower case unless they arecase-sensitive to the operating system or software application they are associated with,

for example:

account _t ab

GLOBALS

Variables are shown within < >characters, for example:


10/374

About the GoldenGate guidesGetting more help with GoldenGate


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

When one of multiple mutually-exclusive arguments must be selected, the selection is

enclosed within braces and separated with pipe characters, for example:

VI EW PARAMS {MGR | | }

Optional arguments are enclosed within brackets, for example:

CLEANUP EXTRACT [ , SAVE ]

When there are numerous multiple optional arguments, a placeholder such as

[ ] may be used, and the options are listed and described separately, for

example:

TRANLOGOPTI ONS [ ]

When an argument is accepted more than once, an ellipsis character (...) is used, for

example:

PARAMS ( [ ] [ , ] [ , . . . ] )

Getting more help with GoldenGateIn addition to the GoldenGate documentation, you can get help for GoldenGate in the

following ways.

Getting help with the GoldenGate interface

Both GGSCI and the GoldenGate Director applications provide online help.

GGSCI commands

To get help for a GoldenGate command, use the HELP command in GGSCI. To get a

summary of command categories, issue the HELP command without options. To get help for

a specific command, issue the HELP command with the command name as input.

HELP

Example:

HELP ADD EXTRACT

The help file displays the syntax and description of the command.

GoldenGate Director

To get help for either GoldenGate Director Client or GoldenGate Director Web, use the Help

menu within the application.

Getting help with questions and problemsFor troubleshooting assistance, see GoldenGate for Windows and UNIX Troubleshooting

and Performance Tuning Guide. Additional information can be obtained from the

GoldenGate knowledge base on the GoldenGate support site at

http://support.goldengate.com. You can search on a question or word, or do an advanced

search. If you cannot find an answer, you can submit a request to the GoldenGate

Technical Services team.
http://support.goldengate.com/http://support.goldengate.com/


11/374

About the GoldenGate guidesNew features in this release of GoldenGate


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

You will need a GoldenGate Support ID and password to use this web page. To get an ID

and password, send an e-mail to [email protected].

New features in this release of GoldenGate

Database-specific enhancements

DB2 LUW

GoldenGate now supports DB2 LUW version 9.5.

DB2 z/OS

GoldenGate now supports DB2 for z/OS version 9.1.

GoldenGate now supports ASCII and Unicode encoded tables on z/OS. The type of

character format (ASCII, Unicode, EBCDIC) is recorded in the record header. The

mapping ofNCHAR to non-NCHAR, and Unicode to non-Unicode is not supported.

GoldenGate now translates between EBCDIC and ASCII automatically, based on the

format of incoming data (from record header) and the format of the target table. Up to

this point, the translation was controlled by parameter specification with ASCIITOEBCDIC

and EBCDICTOASCII (undocumented parameter). With the addition of the automatic

conversion, GoldenGate can correctly process a set of trail files that contains a mixture

of EBCDIC and ASCII to ensure data integrity. Unicode is never converted. If the

PASSTHRU parameter is specified, the data is not converted by the associated data pump.

NOTE See the GoldenGate for DB2 z/OS Installation and Setup Guide for informationabout GoldenGate upgrades and the order in which they should be performed in aheterogeneous environment.

GoldenGate now supports GRAPHIC, VARGRAPHIC, and DBCLOB data types in Unicode

tables. Mixed data columns are also supported. Graphic and mixed data in tables otherthan Unicode are not supported.

c-tree

The following TRANLOGOPTIONS options were added:

READBUFFER specifies a length, in bytes, for the buffer maintained by c-tree that

contains replication log data. This buffer is used by default as a performance

feature. When Extract requests data that is in the buffer, a noticeable performance

gain can usually be observed.

READTIMEOUT specifies a timeout, in milliseconds, that determines how long the

c-tree replication engine waits to send a change record to the transaction buffer

(controlled by READBUFFER) if one is not present at its current position.

Ingres

GoldenGate now supports extraction from, and replication to, an Ingres Database target.

GoldenGate supports Ingres Database versions 2.6, 9.0, and 9.1. For more information, see

the GoldenGate for Ingres Installation and Setup Guide.


12/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

Oracle

GoldenGate now supports Oracle 11g.

GoldenGate for Oracle 10g and 11g now can be configured to capture data in archived-

log only mode (known asALO), where Extract reads exclusively from the archived logs,

bypassing the online logs completely. This enables you to deploy GoldenGate on adifferent server from the one that contains the online logs and supports more strict

security level agreements. Certain limitations apply. See the GoldenGate for Oracle

Installation and Setup Guide.

The following TRANLOGOPTIONS options were added or changed:

An ARCHIVEDLOGONLY option controls the ALO feature. In this mode, Extract reads

from archived logs that are stored in a location that is specified with the

ALTARCHIVELOGDEST option ofTRANLOGOPTIONS.

The PRIMARY sub-option of the ALTARCHIVELOGDEST option was changed to the default.

PRIMARY prevents Extract from checking the default log location if it does not find

the log in the alternate location. Only the ALTARCHIVELOGDEST path is checked.

An ASMBUFSIZE option tunes the memory pool that contains data that is captured

from an ASM instance. Use this option to set an optimum balance between thecache size and the disk I/O that is incurred to store data that exceeds the cache size.

ALOGSOURCE option specifies the correct operating system when redo and/or

archived logs are stored on a different system from the one that contains the

database.

GoldenGate now supports the replication of Oracle sequences. GoldenGate ensures

that the target sequence values will always be higher than those of the source. To

replicate sequences, use the new SEQUENCE parameter.

A new CHECKSEQUENCEVALUE parameter for Replicat can be used to ensure that the target

sequence value is higher than that of the source.

GoldenGate now supports index-organized tables (IOT) and tables with virtual

columns. For more information, see the GoldenGate for Oracle Installation and Setup

Guide.

A new parameter FETCHMODCOLS(EXCEPT) was added to process transparent data

encryption (TDE) data, instead of having to use SQLEXEC with user tokens.

A new option LOBWRITESIZE was added to DBOPTIONS to reduce the impact on performance

of writing LOB chunks to the target. Replicat now makes use of a buffer and writes the

chunks all at once based on the parameter setting. Another new DBOPTIONS options,

DISABLELOBCACHING, turns off the caching of LOBs.

A new USELASTREADTIME option was added to WARNLONGTRANS to force Extract to use the

last read time as the basis when considering whether a transaction is long-running.

The new default behavior is to use the timestamp of the last record processed.

Additional INCLUDE and EXCLUDE options have been added to support the filtering of DDL

according to words in the DDL syntax or (Oracle only) in comments only. These two

options can be combined. The new options are:

INSTRWORDS

INSTRCOMMENTSWORDS

The ddl_tracelevel script now can be set to a value ofNone. At this level, no DDL tracing

is done except for fatal errors and installation logging. The default value of 0 generates

minimal tracing information. A value of 1 or 2 generates a much larger amount of

information in the trace file.


13/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

GoldenGate introduces a new method for selecting a key when a primary key does not

exist (as first choice). Instead of using all of the columns on which all unique indexes

and unique constraints are defined (except for virtual columns), GoldenGate now

selects a key as follows:

This new method reduces the number of columns that must be logged to the

transaction log. It also reduces the number of primary key updates that have to bemanaged by GoldenGate. A primary key update is a DML change on any column that

is part of the key that is being used by GoldenGate, whether that is a primary or unique

key, a KEYCOLS-defined key, or all of the columns if no other key is present. This

improvement resolves some of the problems that primary key updates caused with

certain functions of GoldenGate, such as HANDLECOLLISIONS and SQLEXEC. As before, you

can always override the priority of key selection by using a KEYCOLS-defined key.

SQL Server

GoldenGate now supports reading from online logs and log backups that are stored in

an alternate location from the default. Use the TRANLOGOPTIONS parameter with the

following options:

ALTARCHIVELOGDEST specifies an alternate location for the log backups. ALTONLINELOGS specifies an alternate location for the online logs.

ARCHIVERESTOREEXECUTABLE () specifies the path to a third-party

program to invoke for restoring log backups.

ARCHIVERESTOREPARAMS specifies the parameters that the third-party executable

specified by ARCHIVERESTOREEXECUTABLE needs.

For SQL Server 2005, GoldenGate no longer requires the use of SQL Server replication

components and a distributor database. GoldenGate now makes use of a new Microsoft

stored procedure that enables extended logging to provide full before and after row

images, which formerly were only available by means of the replication components.

This procedure is part of the Microsoft Cumulative Update Package 6 (CU6) for SQL

Server 2005 Service Pack 2 (or later).NOTE You can continue to use the former GoldenGate support for SQL Server 2005,

which makes use of the replication components. See the GoldenGate for SQL

Server Installation and Setup Guide.

1. Primary key2. First unique key alphanumerically with no virtual columns or UDTs and no

nullable columns

3. First unique key alphanumerically with no virtual columns or UDTs but caninclude nullable columns

4. If none of the preceding key types exists (even though there might be other typesof keys defined on the table) GoldenGate constructs a pseudo key of all columns

that can be used in a WHERE clause, excluding virtual columns, UDTs, and any

columns that are explicitly excluded from the GoldenGate configuration.


14/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

In CU6, Microsoft also has enabled new functionality that allows GoldenGate to

capture from tables that do not have primary keys, as found in commonly used

business applications. If you previously had Microsoft native replication and want to

convert to the CU6 method, see the GoldenGate for SQL Server Installation and Setup

Guide.

The LIMITROWS parameter now supports SQL Server.

GoldenGate now supports tables with non-materialized computed columns. For more

information, see the GoldenGate for SQL Server Installation and Setup Guide.

A new SEND EXTRACT option enables you to send a message to a running SQL Server

Extract process to print a list of open transactions with their transaction ID, start time,

first LSN, and the number of operations they contain. The syntax is:

SEND EXTRACT VAMMESSAGE OPENTRANS

The following GoldenGate environment variables are now exposed to users to control

the number of retries and delay (in milliseconds) between retries when Extract is

unable to access the transaction logs on any supported Windows or UNIX-based

platform. The default values are shown in the example.

SETENV ( GGS_CacheRet r yCount = 10)

SETENV ( GGS_CacheRet r yDel ay = 1000)

The error message that indicates when these parameters should be changed begins as

follows:

GGS ERROR 600 [CFi l eI nfo: : Read] Ti meout expi r ed af t er 10 r et r i es wi t h1000 ms del ay wai t i ng t o read t r ansact i on l og or backup f i l es.

Sybase

GoldenGate now supports Sybase ASE version 15.

GoldenGate now supports the capture of encrypted data from the transaction log.

GoldenGate now supports the use of dynamic SQL by Replicat to improve theperformance of applying data to a Sybase target. With dynamic SQL, Replicat compiles

a given SQL statement once, and then reuses it many times with different bind

variables. Dynamic SQL is enabled by default, but it can be controlled with the DYNSQL

parameter.

NonStop SQL/MX

GoldenGate now can be installed and operated directly on a SQL/MX target system. An

intermediary Windows system is no longer needed. For installation information, see the

GoldenGate for NonStop SQL/MX Installation and Setup Guide.

Informix

Informix is no longer supported by GoldenGate.

Teradata

GoldenGate now supports Teradata version 13.


15/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

GoldenGate now supports DDL replication for Teradata. For information about

configuring replication groups to support this new feature, see the GoldenGate for

Teradata Installation and Setup Guide. For information about how to configure

GoldenGate to support Teradata DDL replication, see the GoldenGate for Windows

and UNIX Administrator Guide.

GoldenGate now supports more Teradata data types, including large objects and user-defined types. See the GoldenGate for Teradata Installation and Setup Guide for the

list of supported types.

Other enhancements

JMS integration

GoldenGate now can publish transactions from source systems to a JMS queue and JMS

topic. The transactions are written with guaranteed integrity. The events are published

real-time with only sub-second latency, providing support for event-driven architecture

(EDA). Service-oriented architecture (SOA) is supported by GoldenGates heterogeneous

capabilities.

Deferred Apply

A new DEFERAPPLYINTERVAL parameter allows you to set an amount of time that Replicat

waits before applying transactions from the trail to the target system, based on the commit

timestamp of the source transaction. You can use DEFERAPPLYINTERVAL to keep the target

system at a consistent state behind the primary system for such purposes as to prevent the

propagation of erroneous changes made to the source data, to control data arrival across

different time zones, and to allow time for other planned events to occur before the data is

applied. Transaction integrity is maintained at all times.

Improved memory performance

To address the need for a more efficient means of handling uncommitted transaction datain memory, GoldenGate introduces a new memory-management system that replaces the

need for the TRANSMEMORY parameter on all databases except DB2 z/OS and NonStop

SQL/MX. For those databases, TRANSMEMORY still applies. The new system is controlled by

the new CACHEMGR parameter, although you should not adjust the settings without

assistance from a GoldenGate support analyst.

The GoldenGate cache manager takes advantage of the memory management functions of

the operating system to ensure that GoldenGate processes work in a sustained and

efficient manner. Within its cache, it makes use of modern virtual memory techniques by:

Allocating and managing active buffers efficiently.

Recycling old buffers instead of paging to disk, when possible.

Paging less-used information to disk, when necessary.

The actual amount of physical memory that is used by any GoldenGate process is

controlled by the operating system, not the GoldenGate program.

The cache manager keeps a GoldenGate process working within the soft limit of its global

cache size, only allocating virtual memory (not physical memory) on demand. System calls

to increase the cache size are made only as a last resort and, when used, are always

followed by the release of virtual memory back to the system.


16/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

Event marker system

A new event marker system enables GoldenGate processes to take a specific action based

on a record in the transaction log or the trail. This record, known as the event record, is so

designated by including a FILTER or WHERE clause in aTABLE or MAP statement, in conjunctionwith the new EVENTACTIONS option, which specifies the action to be taken.

Some supported actions are to stop Extract or Replicat, to log a message, and to issue a

checkpoint. This system can be used to switch over from an initial load to the movement of

change data, to coordinate with end-of-day reporting procedures, and to track processing

lag.

A new TABLE parameter for Replicat also supports this feature. This parameter enables

FILTER, WHERE, SQLEXEC, and EVENTACTIONS to be applied by Replicat without requiring data to

be mapped to a target table.

New recovery modes

Version 10 introduces significant changes to the GoldenGate trails. GoldenGate no longer

overwrites data in a trail file when it restarts after a failure. Formerly, anything that was

written to the trail by the previous instance of Extract was overwritten by the new Extract.

This was sometimes a problem, because the new transaction records usually were different

from what was written previously. New data was often written in the middle of an old

record, causing a record alignment error from Replicat.

By default, Extract operates in append mode, where if there is a process failure, a recovery

marker is written to the trail and Extract appends recovery data to the file so that a history

of all prior data is retained for recovery purposes.

In append mode, the Extract initialization determines the identity of the last complete

transaction that was written to the trail at startup time. With that information, Extract

ends recovery when the commit record for that transaction is encountered in the data

source; then it begins new data capture with the next committed transaction that qualifiesfor extraction and begins appending the new data to the trail. A data pump or Replicat

starts reading again from that recovery point.

The overwrite/append behavior is controlled by the new RECOVERYOPTIONS parameter. Do not

change this parameter from its default (append) unless directed to do so by a GoldenGate

support analyst.

File versioning

To support forward and backward compatibility of the different trail formats between new

versions of GoldenGate (which use the append mode to write a trail) and old GoldenGate

versions (which use the overwrite mode), file versioning has been introduced. With file

versioning, you can use different versions of GoldenGate processes and trail files in

different combinations. If you plan to replicate data between version 10 of GoldenGate andolder versions, it is important that you understand how to configure your processes

correctly.

A trail or extract file must have a version that is equal to, or lower than, that of the process

that reads it. Otherwise the process will abend. Additionally, GoldenGate forces the output

trail or file of a data pump to be the same version as that of its input trail or file. Upon

restart, Extract rolls a trail to a new file to ensure that each file is of only one version

(unless the file is empty).


17/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

The trail file versioning works as follows:

Because all of the GoldenGate processes are decoupled and thus can be of different

GoldenGate versions, each trail file or extract file has a version that is stored in the file

header. By default, the version of a trail is the current version of the process that created

the file. To set the version of a trail, use the FORMAT option of the EXTTRAIL, EXTFILE, RMTTRAIL,or RMTFILE parameter.

To ensure forward and backward compatibility of files among different GoldenGate process

versions, the file header fields are written in a standardized token format. New tokens that

are created by new versions of a process can be ignored by older versions, so that backward

compatibility is maintained. Likewise, newer GoldenGate versions support older tokens.

Additionally, if a token is deprecated by a new process version, a default value is assigned

to the token so that older versions can still function properly. The token that specifies the

file version is COMPATIBILITYLEVEL and can be viewed in the Logdump utility and also by

retrieving it with the GGFILEHEADER option of the @GETENV function.

There is a dependency between the RECOVERYOPTIONS parameter and the FORMAT option of

EXTTRAIL, RMTTRAIL, EXTFILE, and RMTFILE. When RECOVERYOPTIONS is set to APPENDMODE, the

FORMAT option must be set to RELEASE 10.0 or greater. When RECOVERYOPTIONS is set toOVERWRITEMODE, the FORMAT option must be set to RELEASE 9.5 or less.

New Commit Sequence Number

When working with GoldenGate, you might need to refer to a Commit Sequence Number,

or CSN. The CSN can be required to position Extract in the transaction log, to reposition

Replicat in the trail, or for other purposes. It is returned by some conversion functions and

is included in reports and certain GGSCI output.

A CSN is an identifier that GoldenGate constructs to identify a transaction for the purpose

of maintaining transactional consistency and data integrity. It uniquely identifies a

particular point in time in which a transaction commits to the database.

Each kind of database management system generates some kind of unique serial number

of its own at the completion of each transaction, which uniquely identifies that transaction.

A CSN captures this same identifying information and represents it internally as a series

of bytes, but the CSN is processed in a platform-independent manner. A comparison of any

two CSN numbers, each of which is bound to a transaction-commit record in the same log

stream, reliably indicates the order in which the two transactions completed.

The CSN value is stored as a token in any trail record that identifies the beginning of a

transaction. This value can be retrieved with the @GETENV column conversion function and

viewed with the Logdump utility.

New Replicat START options

The following options were added to the START REPLICAT command: SKIPTRANSACTION: Causes Replicat to skip the first transaction from its restart position

in the trail upon startup. All operations from that first transaction are excluded.

ATCSN : Causes Replicat to skip transactions in the trail until it finds a transaction

indicator that contains the specified CSN. All transactions with a CSN less than the

specified one are skipped.


18/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

AFTERCSN: Causes Replicat to skip transactions in the trail until it finds the first

transaction after the one that contains the specified CSN. All transactions whose

begin-transaction record contains a CSN less than, or equal to, the specified one are

skipped.

Limited Unicode and native character encoding

GoldenGate supports the use of an escape sequence to represent characters in a string

column in Unicode or in the native character encoding of the Windows, UNIX, and Linux

operating systems. An escape sequence can be used in the following elements within aTABLE

or MAP statement:

WHERE clause

COLMAP clause to assign a Unicode character to a Unicode column, or to assign a native-

encoded character to a column.

GoldenGate column conversion functions within a COLMAP clause.

GoldenGate supports the following types of escape sequence:

\uFFFF Unicode escape sequence

\377 Octal escape sequence

\xFF Hexadecimal escape sequence

The following limitations apply:

This support is limited to UTF-16 code points from U+0000 to U+007F, the equivalent

of 7-bit ASCII.

The source and target columns must both be Unicode.

The source and target data types must be identical (for example, NCHAR to NCHAR).

Improved initial loadsWhen extracting data from source tables in an initial-load procedure, Extract now captures

the data as arrays, rather than as rows, to improve the performance of the overall initial

load. This capability has existed in GoldenGate for Oracle and SQL/MX but has been

expanded to include the other supported databases. By default, the number of rows that

are fetched as an array is 1000. The FETCHBATCHSIZE option of the DBOPTIONS parameter for

Extract controls the array size and now is valid for all GoldenGate-supported databases.

Note that when a table contains LOB data, Extract reverts to single-row fetch mode.

Additional internal performance enhancements were made to remove unnecessary work

within the framework design. These are internal enhancements that have no configuration

parameters.

New user exit functions and enhancements

The GoldenGate user exit feature on Windows and UNIX-based platforms has been

expanded to include the following capabilities:

A versioning feature has been added to the usrdecs.h file to allow backward compatibility

with existing user exits when enhancements are added in ongoing GoldenGate

releases. The new version available with this release is version 2. The old versions are


19/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

0 and 1. The version is printed in the report file at the startup of Replicat or Extract.

If your routine requires any of the enhancements that are introduced in a new

GoldenGate release, you must recompile it with the new usrdecs.h file.

Version 2 includes the following enhancements.

CUSEREXIT parameter: A new PARAMS option enables you to supply a startup string, such as a properties

file or global startup parameter, which is executed on the first callout to the user

exit code as EXIT_CALL_START.

A new PASSTHRU option adds support for pass-through behavior with an Extract

data pump, wherein PASSTHRU specifies the following: no output trail is required; the

user exit will perform all of the processing; and Extract will skip the record.

A new INCLUDEUPDATEBEFORES option includes the before images of column values to

a user exit. When using this parameter, you must explicitly request the before

image by setting the new requesting_before_after_ind flag to BEFORE_IMAGE_VAL within a

callback function that supports this flag. Otherwise, only the after image is

processed to the user exit.

Functionality improvements: Support was added for primary key updates by allowing the before values to be

returned with a new requesting_before_after_ind field in the GET_COLUMN_VALUE

function.

The GET_COLUMN_VALUE_FROM_INDEX and GET_COLUMN_VALUE_FROM_NAME functions

now support LOBs. A new column_def structure in usrdecs.h includes a more_lob_data

flag to indicate if more LOB data is present beyond the initial 4K that can be stored

in the base record. GoldenGate will access LOB columns up to 8K of data at all

times, filling up the buffer to the amount that the user exit has allocated. If the

LOB is larger than that which was allocated, subsequent callbacks are required to

obtain the total column data, until all data has been sent to the user exit.

GET_STATISTICS was enhanced to handle statistics automatically based on which

process type has requested the data. Previously, it was assumed that the tableprovided was the source table. Now, Extract will always treat the request as a

source, counting that table once regardless of the number of times output, and

Replicat will always treat the request as a set of target tables that includes all

counts to the target regardless of the number of source tables.

Along with those improvements, GET_STATISTICS now has a value for counting

truncates, and a new EXIT_STAT_GROUP_USEREXIT group level of aggregation has been

added.

New EXIT_CALL_RESULT response

EXIT_PROCESSED_REC_VAL instructs Extract or Replicat to skip the record, but update

the statistics that are printed to the report file for that table and for that operation

type.

New ERCALLBACK codes EXIT_FN_RET_NOT_SUPPORTED indicates that the process does not support the function

that is being used.

EXIT_FN_RET_FETCH_ERROR indicates that the record could not be fetched.

EXIT_FN_RET_EXCEEDED_MAX_LENGTH indicates that the metadata could not be

retrieved because the table or column name did not fit into the allocated buffer.


20/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

EXIT_FN_RET_TOKEN_NOT_FOUND indicates that the specified user token could not be

found in the record.

EXIT_FN_RET_INCOMPLETE_DDL_REC indicates that an internal error occurred when

processing a DDL record.

EXIT_FN_RET_ENV_NOT_FOUNDindicates that the specified environment variable could

not be found in the record.

EXIT_FN_RET_INVALID_COLUMN_TYPE indicates that the routine is trying to manipulate

a data type that is not supported by GoldenGate for that purpose.

New callback functions:

GET_COL_METADATA_FROM_NAME and GET_COL_METADATA_FROM_INDEX retrieve column

metadata by specifying the name or index of the desired column

GET_DDL_RECORD_PROPERTIES retrieves information about a DDL operation, including

the statement text.

GET_ENV_VALUE allows GoldenGate environment information to be retrieved by a

user exit. It accepts the same input as the @GETENV column-conversion function and

supplies the same return values.

GET_GMT_TIMESTAMP retrieves the operation commit timestamp in GMT format. Useof this function requires a re-compile with the Version 2 usrdecs.h.

GET_TABLE_METADATA retrieves metadata about the table that associated with the

record that is being processed.

RESET_USEREXIT_STATS gives the user exit control over when statistics are reset,

allowing it to maintain is own aggregation. A user exit can now reset statistics

since the last call to GET_STATISTICS was processed. The reset is processed within the

existing callback to get the statistics.

New demo/sample files:

Column conversion function enhancements

AGGFILEHEADER option has been added to @GETENV to return attributes of a GoldenGate

trail file that are stored in the file header. This new function supports the new trail

versioning feature and also supports the new GET_ENV_VALUE user exit function. Among

GoldenGate installs the following sample user exit files into the UserExitExamples

directory of the GoldenGate installation directory:

exitdemo.c shows how to initialize the user exit, issue callbacks at given exit points,and modify data. The demo is not specific to any database type.

exitdemo_passthru.cshows how the PASSTHRU option of the CUSEREXIT parameter can be

used in an Extract data pump.

exitdemo_more_recs.c shows an example of how to use the same input record multiple

times to generate several target records.

exitdemo_lob.c shows an example of how to get read access to LOB data.

exitdemo_pk_befores.c shows how to access the before and after image portions of a

primary key update record, as well as the before images of regular updates (non-

key updates). It also shows how to get target row values with SQLEXEC in the

Replicat parameter file as a means for conflict detection. The resulting fetched

values from the target are mapped as the target record when it enters the user exit.

Each directory contains the .c file as well as makefiles and a readme.txt file.


21/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

the attributes that can be returned are the trail version, the file sequence number and

size, information about the first and last records in the file, and information about the

local environment, among other attributes.

New GROUPDESCRIPTION, GROUPTYPE, and PROCESSID environment values have been added

to the GGENVIRONMENT option of@GETENV.

DBOPTIONS LOBWRITESIZE parameter

Specifies a fragment size for each LOB that Replicat writes to the target database. The

LOB data is stored in a buffer until this size is reached. Because LOBs must be written to

the database in fragments, writing in larger blocks prevents excessive I/O. The higher the

value, the fewer I/O calls that are made by Replicat to the database server to write the

whole LOB to the database.


22/374


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

CHAPTER 1Introduction to GoldenGate. . . . . . . . . . . . . . .

GoldenGate supported processing methods and databases

GoldenGate enables the exchange and manipulation of data at the transaction level among

multiple, heterogeneous platforms across the enterprise. Its modular architecture gives

you the flexibility to extract and replicate selected data records, transactional changes, and

changes to DDL (data definition language1) across a variety of topologies.

With this flexibility, and the filtering, transformation, and custom processing features of

GoldenGate, you can support numerous business requirements:

Business continuance and high availability.

Initial load and database migration.

Data integration.

Decision support and data warehousing.

1. DDL is not supported for all databases

Figure 1 GoldenGate supported topologies


23/374

Introduction to GoldenGateOverview of the GoldenGate architecture


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

*Supported only as a target database. Cannot be a source database for GoldenGate extraction, the exception being that GoldenGate canextract records from MySQL source tables as part of a GoldenGate direct load.

** A module that communicates with the GoldenGate API to facilitate transactional data management by GoldenGate.*** Only like-to-like configuration is supported. Data manipulation, filtering, column mapping not supported.

Overview of the GoldenGate architecture

GoldenGate is composed of the following components:

Extract

Data pump

Replicat

Trails or extract files

Checkpoints

Manager

Collector

Figure 2 illustrates the logical architecture of GoldenGate for initial data loads and for the

replication of ongoing database changes. This is the basic configuration. Variations of this

model are recommended depending on business needs.

Table 1 Supported processing methods

DatabaseLog-BasedExtraction

VAM Extraction(Vendor AccessModule)**

Replication

c-tree*** X X

DB2 for Linux, UNIX,Windows

X X

DB2 for z/OS X X

Ingres X X

MySQL* X

Oracle X X

SQL/MX X X

SQL Server X X

Sybase X X

Teradata X X

Generic ODBC* X


24/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

Overview of Extract

The Extract process runs on the source system and is the capture mechanism ofGoldenGate. You can configure Extract in one of the following ways:

Initial loads: For initial data loads, Extract extracts a current set of data directly from

their source objects.

Change synchronization: To keep source data synchronized with another set of data,

Extract extracts transactional changes made to data (inserts, updates, and deletes)

after the initial synchronization has taken place. DDL changes and sequences are also

extracted, if supported for the type of database being used.

When processing transactional data changes, Extract obtains the data from a data source

that can be one of the following.

The database transaction logs (such as the Oracle redo logs or SQL/MX audit trails).This method is known as log-based extraction. When Extract is installed as a log-based

implementation, it can read the transaction logs directly.

AGoldenGate Vendor Access Module (VAM). The VAM is a communication layer that

passes data changes and transaction metadata to the Extract process. The database

vendor provides the components that extract the data changes and pass it to Extract.

This method is known as VAM-based extraction.

Extract captures all of the changes that are made to objects that you configure for

synchronization, but it only sends the data from committed transactions to the trail for

propagation to the target system. When Extract captures the commit record of a

transaction, all of the log records for that transaction are written to the trail as a

sequentially organized transaction unit. This design ensures both speed and data

integrity.

Multiple Extract processes can operate on different objects at the same time. For example,

one process could continuously extract transactional data changes and stream them to a

decision-support database, while another process performs batch extracts for periodic

reporting. Or, two Extract processes could extract and transmit in parallel to two Replicat

processes (with two trails) to minimize target latency when the databases are large. To

differentiate among different processes, you assign each one a group name (see Overview

of groups on page 30).

Figure 2 GoldenGate logical architecture


25/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

For the SQL/MX database, the Extract module also includes a program named VAMSERV.

Extract starts VAMSERV, and together they retrieve and process database changes from the

audit trails created by TMF-enabled applications on a NonStop system.

Overview of data pumps

A data pump is a secondary Extract group within the source GoldenGate configuration. If

a data pump is not used, Extract must send data to a remote trail on the target. In a typical

configuration that includes a data pump, however, the primary Extract group writes to a

trail on the source system. The data pump reads this trail and sends the data across the

network to a remote trail on the target. The data pump adds storage flexibility and also

serves to isolate the primary Extract process from TCP/IP activity.

Like a primary Extract group, a data pump can be configured for either online or batch

processing. It can perform data filtering, mapping, and conversion, or it can be configured

inpass-through mode, where data is passively transferred as-is, without manipulation.

Pass-through mode increases the throughput of the data pump, because all of the

functionality that looks up object definitions is bypassed.

In most business cases, it is best practice to use a data pump. Some reasons for using a datapump include the following:

Protection against network and target failures: In a basic GoldenGate configuration,

with only a trail on the target system, there is nowhere on the source system to store

data that Extract continuously extracts into memory. If the network or the target

system becomes unavailable, the primary Extract could run out of memory and abend.

However, with a trail and data pump on the source system, captured data can be moved

to disk, preventing the abend. When connectivity is restored, the data pump extracts

the data from the source trail and sends it to the target system(s).

You are implementing several phases of data filtering or transformation. When using

complex filtering or data transformation configurations, you can configure a data pump

to perform the first transformation either on the source system or on the target system,

and then use another data pump or the Replicat group to perform the secondtransformation.

Consolidating data from many sources to a central target. When synchronizing multiple

source databases with a central target database, you can store extracted data on each

source system and use data pumps on each of those systems to send the data to a trail

on the target system. Dividing the storage load between the source and target systems

reduces the need for massive amounts of space on the target system to accommodate

data arriving from multiple sources.

Synchronizing one source with multiple targets. When sending data to multiple target

systems, you can configure data pumps on the source system for each target. If network

connectivity to any of the targets fails, data can still be sent to the other targets.

Overview of Replicat

The Replicat process runs on the target system. Replicat reads extracted data changes and

DDL changes (if supported) that are specified in the Replicat configuration, and then it

replicates them to the target database. You can configure Replicat in one of the following

ways:

Initial loads: For initial data loads, Replicat can apply data to target objects or route

them to a high-speed bulk-load utility.


26/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

Change synchronization: To maintain synchronization, Replicat applies extracted

transactional changes to target objects using native database calls, statement caches,

and local database access. Replicated DDL and sequences are also applied, if supported

for the type of database that is being used. To preserve data integrity, Replicat applies

the replicated changes in the same order as those changes were committed to the

source database.

You can use multiple Replicat processes with multiple Extract processes in parallel to

increase throughput. Each set of processes handles different objects. To differentiate

among processes, you assign each one a group name (see Overview of groups on page 30).

You can delay Replicat so that it waits a specific amount of time before applying data to

the target database. A delay may be desirable for such purposes as to prevent the

propagation of errant SQL, to control data arrival across different time zones, and to allow

time for other planned events to occur. The length of the delay is controlled by the

DEFERAPPLYINTERVAL parameter.

Overview of trails

To support the continuous extraction and replication of supported database changes,

GoldenGate stores those changes temporarily on disk in a series of files called a trail. A

trail can exist on the source or target system, or on an intermediary system, depending on

how you configure GoldenGate. On the local system it is known as an extract trail (or local

trail).On a remote system it is known as a remote trail.

By using a trail for storage, GoldenGate supports data accuracy and fault tolerance (see

Overview of checkpoints on page 28). The use of a trail also allows extraction and

replication activities to occur independently of each other. With these processes separated,

you have more choices for how data is delivered. For example, instead of extracting and

replicating changes continuously, you could extract changes continuously but store them

in the trail for replication to the target later, whenever the target application needs them.

Processes that write to, and read, a trail

The primary Extract process writes to a trail. Only one Extract process can write to a trail.

Processes that read the trail are:

Data-pump Extract: Extracts data from a local trail for further processing, if needed,

and transfers it to the target system or to the next GoldenGate process downstream in

the GoldenGate configuration.

Replicat: Reads a trail to apply change data to the target database.

Trail maintenance

Trail files are created as needed during processing, and they are aged automatically to

allow processing to continue without interruption for file maintenance. By default, each filein a trail is 10 MB in size. All file names in a trail begin with the same two characters,

which you assign when you create the trail. As files are created, each name is appended

with a unique, six-digit serial (sequence) number from 000000 through 999999, for

example c:\ggs\dirdat\tr000001. By default, trails are stored in the dirdat sub-directory of the

GoldenGate directory.

You can create more than one trail to separate the data of different objects or applications.

You link the objects that are specified in a TABLE or SEQUENCE parameter to a trail that is


27/374



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confidential

specified with an EXTTRAIL or RMTTRAIL parameter in the Extract parameter file. Aged trail

files can be purged by using the Manager parameter PURGEOLDEXTRACTS.

How processes write to a trail

To maximize throughput, and to minimize I/O load on the system, extracted data is sent

into and out of a trail in large blocks. Transactional order is preserved. By default,

GoldenGate writes data to the trail in universal data format, a proprietary format which

allows it to be exchanged rapidly and accurately among heterogeneous databases.

However, data can be written in other formats that are compatible with different

applications.

By default, Extract operates in append mode, where if there is a process failure, a recovery

marker is written to the trail and Extract appends recovery data to the file so that a history

of all prior data is retained for recovery purposes.

In append mode, the Extract initialization determines the identity of the last complete

transaction that was written to the trail at startup time. With that information, Extract

ends recovery when the commit record for that transaction is encountered in the data

source; then it begins new data capture with the next committed transaction that qualifiesfor extraction and begins appending the new data to the trail. A data pump or Replicat

starts reading again from that recovery point.

Overwrite mode is another version of Extract recovery that was used in versions of

GoldenGate prior to version 10.0. In these versions, Extract overwrites the existing

transaction data in the trail after the last write-checkpoint position, instead of appending

the new data. The first transaction that is written is the first one that qualifies for

extraction after the last read checkpoint position in the data source.

If the version of GoldenGate on the target is older than version 10, Extract will

automatically revert to overwrite mode to support backward compatibility. This behavior

can be controlled manually with the RECOVERYOPTIONS parameter.

Trail format

As of GoldenGate version 10.0, each file of a trail contains afile header record that is stored

at the beginning of the file.The file header contains information about the trail file itself.

Previous versions of GoldenGate do not contain this header.

Each data record in a GoldenGate trail file contains a header area and a data area. The

header contains information about the transaction environment, and the data area

contains the actual data values that were extracted. For more information about the

GoldenGate trail record format, seeAppendix 1.

File header and versioning

Because all of the GoldenGate processes are decoupled and thus can be of different

GoldenGate versions, each trail file or extract file has a version that is stored in the file

header. By default, the version of a trail is the current version of the process that created

the file. To set the version of a trail, use the FORMAT option of the EXTTRAIL, EXTFILE, RMTTRAIL,

or RMTFILE parameter.

To ensure forward and backward compatibility of files among different Golden

Documents

GGS WinUnix Admin 10.0