Warehouse Management System

Front cover
DB2 Warehouse Management:

High Availability and Problem Determination Guide
DB2 Warehouse Manager components and consequences of system failures High Availability techniques for data, servers, and media Problem determination methodology
Corinne Baragoin David Bryant Allison Francois Jay Kim Rakesh Ranjan
ibm.com/redbooks
International Technical Support Organization DB2 Warehouse Management: High Availability and Problem Determination Guide March 2002
SG24-6544-00
Take Note! Before using this information and the product it supports, be sure to read the general information in Special notices on page xix.
First Edition (March 2002) This edition applies to IBM DB2 Universal Database V7.2 and IBM DB2 Warehouse Manager V7.2. Comments may be addressed to: IBM Corporation, International Technical Support Organization Dept. QXXE Building 80-E2 650 Harry Road San Jose, California 95120-6099 When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you.
Copyright International Business Machines Corporation 2002. All rights reserved. Note to U.S Government Users Documentation related to restricted rights Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp.
Contents
Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Special notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix IBM trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi The team that wrote this redbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Notice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii Comments welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv Chapter 1. DB2 warehouse management architecture . . . . . . . . . . . . . . . . 1 1.1 Generic ETML subsystem architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 DB2 Warehouse Manager components . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.1 Warehousing tasks and objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.2.2 The Data Warehouse Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.2.3 The warehouse server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.2.4 The warehouse control database . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.2.5 The warehouse agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.2.6 Publishing metadata to Information Catalog Manager . . . . . . . . . . . 20 1.2.7 Accessing data sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 1.2.8 DB2 Warehouse Manager Connectors . . . . . . . . . . . . . . . . . . . . . . . 23 1.2.9 Additional tools and functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Chapter 2. Possible failure scenarios and their consequences . . . . . . . . 31 2.1 Hardware server failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.1.1 Source server failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.1.2 Target server failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.1.3 Warehouse server failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.2 Network failures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.3 Disk failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.4 Failures in DB2 Warehouse Manager server . . . . . . . . . . . . . . . . . . . . . . 34 2.5 Agent failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.6 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Chapter 3. Database backup and recovery . . . . . . . . . . . . . . . . . . . . . . . . . 39
Copyright IBM Corp. 2002
iii
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.2 Planning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.2.1 Speed of recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.2.2 Backup window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.2.3 Recovery points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.2.4 Unit of recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.2.5 Backup of other supporting files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.2.6 Keeping related data together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.2.7 Using different operating systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.3 Recovery concepts and terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.3.1 Unit of work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.3.2 Recovery methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.3.3 Database logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.3.4 Disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.4 Database backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.4.1 Using the db2 backup db command . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.4.2 Invoking the db2 backup db command . . . . . . . . . . . . . . . . . . . . . . . 65 3.4.3 Displaying backup information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 3.4.4 LAN-free backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 3.4.5 Backing up to named pipes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 3.4.6 Optimizing backup performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 3.4.7 Backup restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 3.5 Database restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 3.5.1 Using the db2 restore db command . . . . . . . . . . . . . . . . . . . . . . . . . 70 3.5.2 Invoking the db2 restore db command . . . . . . . . . . . . . . . . . . . . . . . 71 3.5.3 Tablespace redirected restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.5.4 Restoring to an existing database . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 3.5.5 Restoring to a new database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.5.6 Optimizing restore performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 3.5.7 Restore restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 3.6 Rollforward recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 3.6.1 Using the db2 rollforward db command . . . . . . . . . . . . . . . . . . . . . . 77 3.6.2 Invoking the db2 rollforward db command . . . . . . . . . . . . . . . . . . . . 79 3.6.3 Rollforward cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 3.6.4 Tablespace rollforward recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 3.6.5 Point-in-time recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3.6.6 Quiesced state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 3.6.7 Recovering a dropped table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 3.6.8 Rollforward restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 3.7 Backup and restore hints for partitioned databases . . . . . . . . . . . . . . . . . 87 3.8 Useful data movement tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 3.8.1 The db2move tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 3.8.2 The db2look tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
iv
DB2 Warehouse Management: High Availability and Problem Determination Guide
3.8.3 Export and import utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9.1 Transport of database backup . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9.2 Transport of database backup and archived logs . . . . . . . . . . . . . 3.9.3 Standby database via log shipping . . . . . . . . . . . . . . . . . . . . . . . . 3.9.4 Standby database via replication . . . . . . . . . . . . . . . . . . . . . . . . . 3.9.5 Synchronous remote mirroring of all data and log files . . . . . . . . .
.. .. .. .. .. .. ..
92 92 93 93 93 94 95
Chapter 4. High Availability techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 4.2 Clustering and failover support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.2.1 Cluster configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4.2.2 High Availability on AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.2.3 High Availability on the Windows operating system . . . . . . . . . . . . 108 4.3 High Availability of media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 4.3.1 DB2 UDB support for High Availability of media . . . . . . . . . . . . . . . 111 4.3.2 Disk mirroring in AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 4.3.3 RAID technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 4.3.4 High Availability through split mirror . . . . . . . . . . . . . . . . . . . . . . . . 119 4.3.5 Tivoli Storage Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Chapter 5. Warehouse control database recovery scenarios . . . . . . . . . 147 5.1 Before starting to use DB2 Warehouse Manager . . . . . . . . . . . . . . . . . . 148 5.1.1 Configure at least two warehouse servers . . . . . . . . . . . . . . . . . . . 148 5.1.2 Back up the warehouse control database . . . . . . . . . . . . . . . . . . . . 148 5.1.3 Maintain multiple administrative clients . . . . . . . . . . . . . . . . . . . . . . 148 5.1.4 Enable the warehouse control database for rollforward recovery . . 149 5.1.5 Allocate enough logging space . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 5.2 Warehouse control database recovery overview . . . . . . . . . . . . . . . . . . 151 5.3 Backing up the control database using DB2 UDB . . . . . . . . . . . . . . . . . . 152 5.3.1 Full offline backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 5.3.2 Full online backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 5.3.3 Tablespace backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 5.3.4 Incremental backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 5.3.5 Verifying DB2 UDB backup image files . . . . . . . . . . . . . . . . . . . . . . 169 5.3.6 Automating DB2 UDB backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 5.4 Recovering the control database using DB2 UDB . . . . . . . . . . . . . . . . . 180 5.4.1 Restore recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 5.4.2 Rollforward recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 5.4.3 Point-in-time recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 5.4.4 Tablespace redirected recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 5.4.5 Dropped table recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 5.4.6 Restoring your control database into a new DB2 UDB instance . . . 219
Contents
5.4.7 Recovering the recovery history file . . . . . . . . . . . . . . . . . . . . . . . . 221 5.4.8 Restarting restore and rollforward . . . . . . . . . . . . . . . . . . . . . . . . . . 222 5.4.9 Restore without rolling forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 5.5 Backup and recovery using Data Warehouse Center . . . . . . . . . . . . . . . 224 5.5.1 Exporting and importing Data Warehouse Center metadata . . . . . 224 Chapter 6. Problem determination methodology . . . . . . . . . . . . . . . . . . . 229 6.1 A methodology in three phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 6.2 The generic troubleshooting methodology . . . . . . . . . . . . . . . . . . . . . . . 231 6.3 The problem determination phase. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 6.3.1 A problem occurs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 6.3.2 How to start problem determination . . . . . . . . . . . . . . . . . . . . . . . . 234 6.3.3 Understanding DB2 Warehouse Manager architecture . . . . . . . . . 239 6.4 The problem source identification phase. . . . . . . . . . . . . . . . . . . . . . . . . 241 6.4.1 Where to find additional error information . . . . . . . . . . . . . . . . . . . . 242 6.4.2 Other resources/tools to use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 6.5 The problem fixing phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 6.5.1 User errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 6.5.2 Operating system errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 6.5.3 Looking at the problem areas identified . . . . . . . . . . . . . . . . . . . . . 270 6.5.4 What to do when the failure area is not identified . . . . . . . . . . . . . . 286 Chapter 7. DB2 Warehouse Manager hints and tips . . . . . . . . . . . . . . . . 287 7.1 Building a restartable application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 7.1.1 Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 7.1.2 Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 7.1.3 Moving and loading data with DB2 Warehouse Manager . . . . . . . . 292 7.2 Application design hints and tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 7.3 Step and process failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 7.3.1 Step and process definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 7.3.2 What happens when a step/process fails . . . . . . . . . . . . . . . . . . . . 296 7.4 Steps and process hints and tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 7.4.1 Process general design checklist . . . . . . . . . . . . . . . . . . . . . . . . . . 302 7.4.2 SQL steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 7.4.3 UDPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 7.4.4 Transformers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 7.5 Specific application hints and tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 7.5.1 How to set up data exception with DB2 Warehouse Manager . . . . 305 7.5.2 How to update summary records (incremental aggregation) . . . . . 306 7.6 General performance advice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 7.6.1 High resource utilization: CPU, memory, and disk . . . . . . . . . . . . . 307 7.6.2 Overall system-wide performance. . . . . . . . . . . . . . . . . . . . . . . . . . 307 7.6.3 Stored procedures performance considerations . . . . . . . . . . . . . . . 308
vi
7.6.4 Dynamically scale load processing . . . . . . . . . . . . . . . . . . . . . . . . . 310 7.7 Performance checklist when testing applications . . . . . . . . . . . . . . . . . . 311 Appendix A. Using DB2 Warehouse Manager Connector for the Web. . 313 A.1 Using Web Connector: step-by-step procedure . . . . . . . . . . . . . . . . . . . 313 A.1.1 Using WebSphere Site Analyzer administration interface . . . . . . . 313 A.1.2 Using the Data Warehouse Center user interface . . . . . . . . . . . . . 321 A.2 Hints and tips for using the Web Connector . . . . . . . . . . . . . . . . . . . . . . 327 Appendix B. Using DB2 Warehouse Manager Connector for SAP . . . . . 331 B.1 Using SAP R/3 connector: step-by-step procedure . . . . . . . . . . . . . . . . 331 B.2 Hints and tips for using the SAP R/3 Connector . . . . . . . . . . . . . . . . . . . 342 Appendix C. Recovering the Information Catalog Manager database . . 343 C.1 Information Catalog Manager database . . . . . . . . . . . . . . . . . . . . . . . . . 343 C.2 ICM configuration information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 Appendix D. UDP to invoke SQL statements and get SQL codes back . 345 D.1 The UDP source code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 D.2 Procedure to use the UDP under Windows NT/2000 . . . . . . . . . . . . . . . 352 Appendix E. Moving from test to production environment . . . . . . . . . . . 357 E.1 Test to production initial migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 E.1.1 Demoting and copying the steps . . . . . . . . . . . . . . . . . . . . . . . . . . 358 E.1.2 Copying steps and moving to a different subject area . . . . . . . . . . 358 E.1.3 Changing the database definition . . . . . . . . . . . . . . . . . . . . . . . . . . 358 E.1.4 Creating the test environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 E.2 Test to production incremental migration . . . . . . . . . . . . . . . . . . . . . . . . 360 E.2.1 Partial migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 Related publications . . . . . . . . . . . . . . . . . . . . . . IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . Other resources . . . . . . . . . . . . . . . . . . . . . . . . Referenced Web sites . . . . . . . . . . . . . . . . . . . . . . How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . IBM Redbooks collections . . . . . . . . . . . . . . . . . ...... ...... ...... ...... ...... ...... ....... ....... ....... ....... ....... ....... ...... ...... ...... ...... ...... ...... . . . . . . 363 363 363 365 365 365
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
Contents
vii
viii
Figures
1-1 1-2 1-3 1-4 1-5 1-6 1-7 1-8 1-9 1-10 1-11 1-12 3-1 3-2 3-3 3-4 3-5 3-6 3-7 3-8 3-9 3-10 3-11 3-12 3-13 3-14 3-15 3-16 3-17 3-18 3-19 4-1 4-2 4-3 4-4 4-5 4-6 4-7 The population subsystem reference architecture . . . . . . . . . . . . . . . . . . 4 DB2 Warehouse Manager architectural view. . . . . . . . . . . . . . . . . . . . . . 7 Pre-built transformers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Data Warehouse Center GUI Interface . . . . . . . . . . . . . . . . . . . . . . . . . 14 Agent-based architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Agent configuration: all components on the same machine . . . . . . . . . 18 Agent configuration: agent located on the target server . . . . . . . . . . . . 19 Agent configuration: agent located on the source server. . . . . . . . . . . . 20 The Web Connector integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 WebSphere Site Analyzer and DB2 Warehouse Manager . . . . . . . . . . 25 The SAP R/3 Connector integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 MQ Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Crash recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Database status from the Control Center . . . . . . . . . . . . . . . . . . . . . . . 46 Restore recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Rollforward recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Incremental backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Circular logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Online archive logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Offline archive logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Database configuration parameters related to database logging . . . . . 56 Creating and updating the Recovery History File . . . . . . . . . . . . . . . . . 60 Recovery History File from DB2 UDB journal . . . . . . . . . . . . . . . . . . . . 61 Backup Database notebook in the Control Center. . . . . . . . . . . . . . . . . 66 Restore Database notebook in the Control Center . . . . . . . . . . . . . . . . 72 Restore Database notebook with Redirect option . . . . . . . . . . . . . . . . . 73 How to set a quiesce point for a table . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Information about Quiesced state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Standby database via log shipping . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Standby database via replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Remote mirroring for disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . 95 Clusters and DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 High Availability clustering configurations . . . . . . . . . . . . . . . . . . . . . . 101 Idle standby . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Mutual takeover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Example of a AIX hot standby failover configuration . . . . . . . . . . . . . . 106 Example of a AIX mutual instance failover configuration . . . . . . . . . . . 107 Example MSCS configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
ix
4-8 4-9 4-10 4-11 4-12 4-13 4-14 4-15 4-16 4-17 4-18 4-19 4-20 4-21 4-22 5-1 5-2 5-3 5-4 5-5 5-6 5-7 5-8 5-9 5-10 5-11 5-12 5-13 5-14 5-15 5-16 5-17 5-18 5-19 5-20 5-21 5-22 5-23 5-24 5-25 5-26 5-27 5-28
MSCS hot standby configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 MSCS mutual takeover configuration . . . . . . . . . . . . . . . . . . . . . . . . . 111 RAID-0 (Block interleave data striping without parity) . . . . . . . . . . . . . 115 RAID-1 (Disk mirroring) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 RAID-1 (Disk duplexing) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 RAID-5 (Block interleave data striping with skewed parity) . . . . . . . . . 117 Split mirror concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 Implementation of FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 PPRC write cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 StorWatch Expert communicates ESS/ETL through TCP/IP network . 128 DB2 UDB clone database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 DB2 UDB standby database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 DB2 UDB mirror database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 AIX LVM mirroring with three physical copies . . . . . . . . . . . . . . . . . . . 134 Splitcopy read-only filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 Logretain database configuration parameter . . . . . . . . . . . . . . . . . . . . 150 Services window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Offline backup using DB2 Control Center . . . . . . . . . . . . . . . . . . . . . . 156 Backup Database window in DB2 Control Center . . . . . . . . . . . . . . . . 156 DB2 start backup window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 End of DB2 offline backup message . . . . . . . . . . . . . . . . . . . . . . . . . . 157 DB2 Journal shows the result of the backup . . . . . . . . . . . . . . . . . . . . 158 Online backup using DB2 Control Center . . . . . . . . . . . . . . . . . . . . . . 160 Backup database online in DB2 Control Center . . . . . . . . . . . . . . . . . 161 Select Online in Backup Database Options window . . . . . . . . . . . . . . 161 DB2 start backup window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 End of DB2 online backup message . . . . . . . . . . . . . . . . . . . . . . . . . . 162 DB2 Journal shows the result of the backup . . . . . . . . . . . . . . . . . . . . 163 Select tablespace backup menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Select directories or tapes on the backup table space window . . . . . . 166 Select Offline or Online in options window. . . . . . . . . . . . . . . . . . . . . . 166 DB2 start backup window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 End of tablespace backup message . . . . . . . . . . . . . . . . . . . . . . . . . . 167 DB2 Journal shows the result of the tablespace backup . . . . . . . . . . . 168 Select backup database in the DB2 Control Center . . . . . . . . . . . . . . 175 DB2 UDB backup window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 DB2 UDB schedule window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 DB2 UDB message window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 DB2 journal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 DB2 script center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 New command script window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 Restore recovery using the DB2 Control Center . . . . . . . . . . . . . . . . . 183 Backup image selection window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
5-29 5-30 5-31 5-32 5-33 5-34 5-35 5-36 5-37 5-38 5-39 5-40 5-41 5-42 5-43 5-44 5-45 5-46 5-47 5-48 5-49 5-50 5-51 5-52 5-53 6-1 6-2 6-3 6-4 6-5 6-6 6-7 6-8 6-9 6-10 6-11 6-12 6-13 6-14 6-15 6-16 6-17 6-18
DB2 message - Restore job created . . . . . . . . . . . . . . . . . . . . . . . . . . 184 DB2 message window - successful completion . . . . . . . . . . . . . . . . . . 185 Tablespaces window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 Restore using Database drop-down menu . . . . . . . . . . . . . . . . . . . . . 192 Backup image selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 DB2 message warning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 Table Spaces page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Roll-forward page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Options page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 DB2 message - Restore job created . . . . . . . . . . . . . . . . . . . . . . . . . . 195 DB2 message - Successful completion . . . . . . . . . . . . . . . . . . . . . . . . 195 Table Spaces state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 Tablespace point-in-time in the DB2 Control Center . . . . . . . . . . . . . . 204 Backup image selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 Table Spaces window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 Rollforward options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 Options window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Tablespace redirected restore in the DB2 Control Center . . . . . . . . . . 211 Backup image selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 Table Spaces selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 Container selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 Change container. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 Rollforward option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 Options selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 Additional information written in Log and Recovery History FIle . . . . . 217 The troubleshooting generic methodology . . . . . . . . . . . . . . . . . . . . . . 232 Problem determination steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 Error message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 DB2 Information Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 Troubleshooting Tab in DB2 Information Center . . . . . . . . . . . . . . . . . 236 DB2 Messages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 Search in DB2 documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 Error message reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 Warehouse agent role during population . . . . . . . . . . . . . . . . . . . . . . . 240 Problem source identification steps . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 Data Warehouse Center traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 Tools settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 Starting the Control Center trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 Trace output example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 AS/400 Operations Navigator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 Integrated file systems properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 How to turn the transformer trace on . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Debugging procedure for Web Connector . . . . . . . . . . . . . . . . . . . . . . 258
Figures
xi
6-19 6-20 6-21 6-22 6-23 6-24 6-25 6-26 6-27 6-28 6-29 6-30 6-31 6-32 6-33 6-34 7-1 7-2 7-3 7-4 7-5 7-6 7-7 7-8 7-9 A-1 A-2 A-3 A-4 A-5 A-6 A-7 A-8 A-9 A-10 A-11 A-12 A-13 A-14 A-15 A-16 A-17 A-18
DB2 Warehouse Manager Web site . . . . . . . . . . . . . . . . . . . . . . . . . . 260 List of FAQs on the Web site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 Problem fixing steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 Linking to the Data Warehouse Center . . . . . . . . . . . . . . . . . . . . . . . . 263 Logon settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 Logon error example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 db2 list database result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 Warehouse control database initialization . . . . . . . . . . . . . . . . . . . . . . 266 Adding a new user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Metadata configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 Change metadata configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 Warehouse server services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 Event log example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 Resolving error symptoms from the process . . . . . . . . . . . . . . . . . . . . 277 Message type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 Hostname defined on agent site definition . . . . . . . . . . . . . . . . . . . . . . 285 ETML phases and processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 The various steps in the Data Warehouse Center . . . . . . . . . . . . . . . . 295 Step processing options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Work In Progress Start Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 WIP detailed step status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 Run step manually screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 Manual submission step progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 Step statistics screen example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 Information on the execution history of step . . . . . . . . . . . . . . . . . . . . 302 Using WSA user interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314 Defining a WSA project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 Defining a WSA log file data import . . . . . . . . . . . . . . . . . . . . . . . . . . . 316 Defining a WSA database table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Defining a Web Tracker data import . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Scheduling a log file data import . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318 Notification setup for log file data import . . . . . . . . . . . . . . . . . . . . . . . 319 Status for the log file data import extraction. . . . . . . . . . . . . . . . . . . . . 320 Web tracker status monitor window . . . . . . . . . . . . . . . . . . . . . . . . . . . 320 Introducing a new Data Warehouse Center source type . . . . . . . . . . . 321 Defining the WSA interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322 Selecting the data to use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 Storing the data selected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 Introducing the Data Warehouse Center polling step . . . . . . . . . . . . . 324 Defining the polling step parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Define the polling step processing options . . . . . . . . . . . . . . . . . . . . . 325 Using new source and step within the Process Model. . . . . . . . . . . . . 326 Running the process from the Work In Progress window . . . . . . . . . . 327
xii
A-19 B-1 B-2 B-3 B-4 B-5 B-6 B-7 B-8 B-9 B-10 D-1 D-2 D-3 D-4 D-5
Importing Webmart tables flowchart . . . . . . . . . . . . . . . . . . . . . . . . . . 328 Defining SAP R/3 as a new warehouse source . . . . . . . . . . . . . . . . . . 332 Defining application server connection type . . . . . . . . . . . . . . . . . . . . 333 Defining group of server connection type . . . . . . . . . . . . . . . . . . . . . . 334 Selecting SAP R/3 Business Objects. . . . . . . . . . . . . . . . . . . . . . . . . . 335 Defining SAP R/3 as a warehouse source . . . . . . . . . . . . . . . . . . . . . . 336 Defining the parameters of the Business Object selected . . . . . . . . . . 337 Mapping the Business Object parameters . . . . . . . . . . . . . . . . . . . . . . 338 Defining a SAP R/3 process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 Selecting Business Object input parameters . . . . . . . . . . . . . . . . . . . . 340 Selecting SAP R/3 output parameters to be mapped to columns . . . . 341 Define the UDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 Define UDP parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 Define the process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 Process values. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355 Run the UDP process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
Figures
xiii
xiv
Tables
1-1 1-2 3-1 3-2 4-1 4-2 6-1 6-2 6-3 6-4 6-5 6-6 6-7 6-8 6-9 7-1 Platforms and components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Client driver program by system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 The logretain and userexit parameters . . . . . . . . . . . . . . . . . . . . . . . . . 59 Disaster recovery methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 RAID Classifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Summary of RAID performance characteristics . . . . . . . . . . . . . . . . . . 118 Log file locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 UDP trace location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 Levels of trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Logtable column definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Odbctest program location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 Traces to use with Web Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 Traces to use with SAP Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Message categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 iSeries message types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280 UNIX commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
xv
xvi
Examples
3-1 3-2 3-3 4-1 4-2 4-3 4-4 4-5 5-1 5-2 5-3 5-4 5-5 5-6 5-7 5-8 5-9 5-10 5-11 5-12 5-13 5-14 5-15 5-16 5-17 5-18 5-19 5-20 5-21 5-22 5-23 5-24 5-25 5-26 5-27 5-28 5-29 5-30 Directory listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Restore error message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 Rollforward command results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 Splitcopy example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 db2adutl utility example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Example db2chkbk. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 DB2 list history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Sample backup and verification script . . . . . . . . . . . . . . . . . . . . . . . . . 144 Update database configuration parameters for rollforward recovery . . 149 Update database configuration parameter to increase logging space . 151 db2start.exe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Net start command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 List applications command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Force application command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Backup command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Backup online command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Tablespace backup using the DB2 command line window . . . . . . . . . 164 Setting up TRACKMOD parameter ON using update command . . . . . 169 A full backup before the first incremental backup . . . . . . . . . . . . . . . . 169 Incremental backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 Checking backup image files using db2ckbckp . . . . . . . . . . . . . . . . . . 170 The db2 list history backup command . . . . . . . . . . . . . . . . . . . . . . . . . 170 db2adutl query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 db2adutl extract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 db2adutl verify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 backpcdb.cmd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Scheduling backup using AT command. . . . . . . . . . . . . . . . . . . . . . . . 174 Displaying backup schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 Container error in db2diag.log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 The list history command results . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 Restoring the warehouse control database . . . . . . . . . . . . . . . . . . . . . 183 List tablespaces to check the state . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 The list history backup command . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Restore tablespace command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 List tablespaces command to check the state again . . . . . . . . . . . . . . 189 Rollforward command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 List tablespaces command again. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 List tablespaces show detail results . . . . . . . . . . . . . . . . . . . . . . . . 198
xvii
5-31 5-32 5-33 5-34 5-35 5-36 5-37 5-38 5-39 5-40 5-41 5-42 5-43 5-44 5-45 5-46 5-47 5-48 5-49 5-50 5-51 5-52 5-53 5-54 5-55 5-56 5-57 5-58 5-59 5-60 5-61 6-1 6-2 6-3 6-4 6-5 6-6 6-7 6-8 6-9 D-1
The list history command results . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 Restore using the last backup image . . . . . . . . . . . . . . . . . . . . . . . . . . 202 Rollforward to a point-in-time past the minimum recovery time . . . . . . 202 Rollforward stop command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Tablespace in backup pending state . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Tablespace backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 List tablespaces to determine the tablespace ID . . . . . . . . . . . . . . . . . 208 List containers for DATASPACE1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 Restore redirect command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Set tablespace containers command . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Restore continue command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Rollforward to end of logs and stop . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 List tablespace containers for DATASPACE1 . . . . . . . . . . . . . . . . . . . 210 Alter tablespace with dropped table recover option . . . . . . . . . . . . . . . 216 Checking to see if the dropped table recovery is enabled . . . . . . . . . . 216 List history dropped table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 Restore from backup taken before the drop . . . . . . . . . . . . . . . . . . . . 218 Rollforward with Recover Dropped Table option . . . . . . . . . . . . . . . . . 218 Redefine the table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Create a database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 Restore into a new database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 Rollforward stop for a new database . . . . . . . . . . . . . . . . . . . . . . . . . . 221 Recover history file command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 Restore USERSPACE1 with a backup image . . . . . . . . . . . . . . . . . . . 222 Rollforward cancel to use different backup image . . . . . . . . . . . . . . . . 222 USERSPACE1 is in restore pending . . . . . . . . . . . . . . . . . . . . . . . . . . 222 Restore USERSPACE1 with a different backup image . . . . . . . . . . . . 223 Rollforward to end of logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Restore without rolling forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Rollforward query status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 Rollforward with stop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 Using db2level command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 DSPCATRC output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 DSNAOINI file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 Find the corrupted relationships. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 Correct the inconsistency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 Identify the steps that have dangling correlation objects . . . . . . . . . . . 283 Delete the dangling correlation objects . . . . . . . . . . . . . . . . . . . . . . . . 283 Delete the relationships to the dangling objects . . . . . . . . . . . . . . . . . 283 Error message example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 Source program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
xviii
Special notices
References in this publication to IBM products, programs or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM product, program, or service is not intended to state or imply that only IBM's product, program, or service may be used. Any functionally equivalent program that does not infringe any of IBM's intellectual property rights may be used instead of the IBM product, program or service. Information in this book was developed in conjunction with use of the equipment specified, and is limited in application to those specific hardware and software products and levels. IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to the IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785. Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact IBM Corporation, Dept. 600A, Mail Drop 1329, Somers, NY 10589 USA. Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of a fee. The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites.
xix
IBM trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States and/or other countries:
e (logo) Redbooks (logo) AIX AS/400 DataJoiner DB2 DB2 Connect DB2 OLAP Server DB2 Universal Database DRDA Enterprise Storage Server ESCON FlashCopy IBM IMS Informix iSeries MQSeries OS/2 OS/390 OS/400 Perform pSeries QMF Redbooks RETAIN S/390 Seascape SP SP1 StorWatch System/390 Tivoli TME Visual Warehouse WebSphere z/OS zSeries Lotus Lotus Notes Notes Working Together
Other company trademarks

The following terms are trademarks of other companies:
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and/or other countries. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States and/or other countries. UNIX is a registered trademark in the United States and other countries licensed exclusively through The Open Group. Other company, product, and service names may be trademarks or service marks of others.
xx
Preface
The majority of warehouse systems run with little or no special protection or availability measures, but as warehouses get bigger or more crucial to the running of a business, then better and better service levels are expected. This IBM Redbook includes many detailed facets of recovery and High Availability. We do not mean to say that every system needs to employ such measures. Rather, we present what is possible and let the reader decide what is appropriate for their environment. The objective of this book is to deliver guidelines for building a High Availability DB2 warehouse environment and to assist with problem determination when using DB2 Warehouse Manager. This book will help you to understand the DB2 Warehouse Manager architecture and components, and to cope with system failures and their consequences. It also provides a set of documentation on the different High Availability techniques that can be used when facing database, server, or disk failures. Finally, we develop a problem determination methodology to help warehouse administrators and implementors to diagnose failures and understand how to solve them or to restart. Application design recommendations and performance advice are also provided to help in designing and building the warehouse population subsystem when using DB2 Warehouse Manager.
The team that wrote this redbook

This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, San Jose Center. Corinne Baragoin is a Business Intelligence Project Leader with the International Technical Support Organization, San Jose Center. Corinne has over 16 years of experience as an IT specialist on DB2 and related solutions. Before joining the ITSO, she worked as an IT Specialist for IBM France, supporting Business Intelligence technical presales activities and assisting customers on DB2 UDB and data warehouse environments.
xxi
David Bryant is an IBM Consultant IT Specialist working with Business Intelligence and Warehousing systems across Europe. He joined IBM in 1969, and in the first 10 to 15 years worked on many large system installations in the UK, specializing in operating systems and their performance. He then moved into the Business Intelligence and Data Warehousing, an area on which he has concentrated for almost 20 years. David holds a BSc degree in Physics from Southampton University. He has had extensive experience with many customer projects across many different industries. Allison Francois is an IT supporting Business Intelligence technical presales activities in the IBM Americas Business Intelligence Solutions Center in Dallas, Texas. She has over 18 years in the IT field with an emphasize on AS/400 and iSeries. For the past 6 years she has been implementing data warehouse proof-of-concepts on the iSeries. Allison was the coauthor of the Business Intelligence Certification Guide redbook. She holds a degree in Mathematics from Morgan State University, Baltimore, Maryland. Allison expertise includes DB2 UDB, DB2 Warehouse Manager, and DB2 OLAP Server. Jay Kim is an IBM Certified Consulting IT Specialist, currently a member of IBM Global Services Business Metrics and Decision Support organization, located in Southbury, Connecticut. Jay has over 24 years of IT experience, focusing primarily on DB2 across all platforms and Business Intelligence solutions, and he has IBM Certifications in DB2 UDB, including Advanced Technical Expert and Solutions Expert for Application Development and Database Administration. Rakesh Ranjan is a Staff Software Engineer at IBM Silicon Valley Lab in San Jose, CA. He has 8 years of experience working with application development on iSeries (AS/400). Rakesh holds a bachelors degree in Electronics and Communication Engineering from India. His areas of expertise at IBM include DB2 UDB for iSeries, and he currently works as a member of the DB2 Warehouse Manager Agent development team. We would like to especially thank the following people for their contributions in providing contents to be incorporated within these pages: Michio Kikuchi Matthias Tschaffler IBM Silicon Valley Laboratory Also, thanks to the following people for their contributions to this project by providing their support, technical input and/or reviewing this redbook: Rohit Agarwal Lynnette Bakow Aakash Bordia Cathy Drummond
xxii
Julie Jerves Madhu Kochar Jacques Labrie Steve Madsen Cindy Marlow Michele Schwartz IBM Silicon Valley Laboratory Chris Eaton IBM Toronto Laboratory Andy Perkins IBM Advanced Technical Support Business Intelligence IBM Americas Kevin P Jones IBM EMEA Technical Sales Business Intelligence Solutions Simon Woodcock IBM EMEA Technical Sales Data Management Solutions Pierre Carrier IBM CRM and Business Intelligence Services in Canada Karl-Heinz Scheible IBM Technical Sales Data Management Solutions in Germany Ron Fryer IBM WW Business Intelligence Solutions Marketing
Notice
This publication is intended to help warehouse implementors and administrators understand recovery and High Availability techniques to use with DB2 warehouse management and to assist them in problem determination. The information in this publication is not intended as the specification of any programming interfaces that are provided by IBM DB2 Universal Database and DB2 Warehouse Manager products. See the PUBLICATIONS section of the IBM Programming Announcement for the product names above for more information about what publications are considered to be product documentation.
Preface
xxiii
Comments welcome
Your comments are important to us! We want our Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at:
ibm.com/redbooks
Send your comments in an Internet note to:

redbook@us.ibm.com
Mail your comments to the address on page ii.
xxiv
Chapter 1.
DB2 warehouse management architecture

This chapter describes the following topics: Generic Extract Transform Move Load (ETML) architecture DB2 Warehouse Manager components The technical environment of DB2 Warehouse Manager
1.1 Generic ETML subsystem architecture

DB2 Warehouse Manager is the IBM middleware to perform Extract Transform Move and Load (ETML) on a data warehouse (or datamart) based mainly on DB2 UDB from multiple data sources. It is an automation tool to help build and populate the data warehouse or datamart. Note: In the old days, data warehousing broke into two camps: those who advocated a single centralized data warehouse, and those who advocated datamarts. Stand-alone data warehouses offered simplicity in their administration and their ability to unexpected user queries, but had inconsistent performance (due to these same unexpected queries) and difficulty accepting true real-time updates, due to conflicting service level agreements. Stand-alone data marts offered consistent performance, but were expensive to manage and did not respond effectively to new user requests. As the business intelligence market has moved closer to traditional production environments, the need for hybrid "n-tier" architectures has evolved. Only through this architecture can we support the conflicting demands of ever-increasing user communities. In a general way, the tasks to populate a data warehouse are basically the same as in any data conversion project. We must extract data from some data source, change the data to match the warehouse/ datamarts data structures and then load it into the warehouse/ datamarts. However, the tasks are usually many times more complex: First, we may have a number of source structures across many systems. Second, we do many more functions in preparing the data for the warehouse. We not only must worry about functions to get the data into a new structure, but we also do much more. We enhance the data by converting codes into something meaningful, calculating new data, adding data from external sources, geo-coding the data, and so forth. We also have to make sure the data is cleansed and reconciled across sources systems. Third, we must maintain history. Any population project now must be repeatable as we will be adding data to the warehouse/datamarts at some periodic interval. On top of all this, we have to maintain this warehouse application as new sources are added, as new business rules must be applied, as new cleansing requirements arise.
Another issue is synchronization between the core data warehouse and any necessary data marts. The marts need to be kept current and consistent with the warehouse, but in a way which does not require extensive manual intervention: hence, the need for a tool like DB2 Warehouse Manager. Designing a data warehouse/datamart population subsystem is an important phase to build a consistent and maintainable data warehouse. The recommended reference architecture for a data warehouse population subsystem is defined in Figure 1-1. This architecture splits the processing into layers, such that each layer performs a specific function which does not overlay into other layers. The processes in each layer are encapsulated, such that what happens in one layer is of no concern to the other layers. This approach allows us to implement a more modular approach, allows the layers to be modified independently, and allows layers to be moved to other platforms with minimal impact to the other layers. It can be utilized to build an easily recoverable population subsystem. The software architecture and components delivered with DB2 Warehouse Manager represents a framework for the design and construction of some type of run-time system. This allows to make intelligent decisions regarding the sometimes conflicting requirements of: Build-time Adaptability/Reusability/Testability Run-time Efficiency/Reliability/Availability/Security As you can see, this architecture has physical components, each of which provides a certain set of services based on its logical layers. Each physical component will likely be one or more technical functions implemented via user program, database, or file utility provided by DB2 Warehouse Manager or most likely, a combination of all of these.
Chapter 1. DB2 warehouse management architecture
Physical components
Extract Component
Prepare Component
Transform and Build Component
Load Component
E X T R A C T
P R E F O R M A T
F I L T E R
M E R G E
D E L T A
C L E A N
T R A N S F O R M
B U I L D
L O A D
Logical Layers
Figure 1-1 The population subsystem reference architecture
For additional information on this architecture and on the description of each component, please read Building the Operational Data Store on DB2 UDB, SG24-6413.
1.2 DB2 Warehouse Manager components

DB2 warehouse management is built on the DB2 UDB and DB2 Warehouse Manager feature. It provides an integrated distributed, heterogeneous warehouse management infrastructure for designing, building, maintaining, governing, and accessing highly scalable, robust data warehouses, Operational Data Stores, and datamarts stored in DB2 UDB databases. DB2 UDB and DB2 Warehouse Manager help warehouse administrators: To manage data volumes, to move data directly from source-to-target (allowing also packaged and simplified access to popular partner products as SAP R/3), and to control the servers on which transformations take place with distributed warehouse agents.
Data warehouses are continuing to grow. DB2 warehouse management provides point-to-point data movement in distributed environments for DB2 data warehouses on Windows NT/2000, OS/2, AIX (pSeries), SUN Solaris, OS/400 (iSeries), and z/OS and OS/390 (zSeries). To speed warehouse and datamarts deployment with commonly used, pre-built data cleansing and statistical transformations. These capabilities can be expanded by adding your own user-defined transformation functions or by integrating products from other vendors, like ETI*Extract or Vality INTEGRITY. Information is also deployed across corporations, between corporations, and to consumers. Managing their access is becoming increasingly important. Packaged as part of the Query Manager is a Web-based query and reporting tool called the Query Management Facility (QMF). It also includes an Information Catalog that allows users to find, understand, and access relevant and accurate information about their data warehouses and datamarts, as well as other sources like files, charts, spreadsheets, and reports. To build and manage from a central point of control, integrated in DB2 UDB, utilizing the Data Warehouse Center graphical user interface. Some of the bits of DB2 warehouse management are chargeable. DB2 warehouse management consists of: An administrative client to define and manage data warehousing tasks and objects, and warehouse or datamart operations: the Data Warehouse Center A manager to manage and control the flow of data: the warehouse server Agents residing on DB2 UDB server platforms (that could be also SUN, HP, Dell and so on) to perform requests from the manager or warehouse server: the local or remote warehouse agent A warehouse control database storing the warehouse management metadata on a DB2 UDB database on UNIX or Intel server A metadata administrative and publishing tool with its own administration graphical user interface (GUI): Information Catalog Manager to manage and present both technical and business metadata. The Information Catalog data may be stored on any licensed DB2 UDB platform. Users may access the Information Catalog through a Web browser.
Data sources:
Any IBM DB2 family databases may be used as data sources. Oracle, Sybase, IBM Informix, and Microsoft SQL Server are databases that require installation of the appropriate ODBC client software.
Flat files: Flat files can be accessed via ODBC on UNIX and Windows NT/2000 platforms. Sequential data sets can be first loaded in DB2 UDB using DB2 load utility, specifically in z/OS and OS/390 environment. They can also be sent through File Transfer Program (FTP) to other platforms (UNIX and Windows/NT) and accessed via ODBC there.
Lotus 1.2.3 spreadsheets and Lotus Notes databases through Data Warehouse Center generic ODBC driver Any data access through DataJoiner Any changes propagation using IBM Data Replication Any OLE-DB source IMS and VSAM data through the Classic Connect drivers that are shipped as part of the DB2 Warehouse Manager feature Classic Connect is separate and chargeable. The zSeries (OS/390) warehouse agent can access IMS and VSAM through the Classic Connect ODBC driver. With Classic Connect, you can set up a DB2-like definition of IMS and VSAM data sets, and then access them using ODBC. You must purchase and install Classic Connect separately from the warehouse agent.
Any application source access through: DB2 Warehouse Manager Connector for Web logs DB2 Warehouse Manager Connector for SAP R/3 data WebSphere MQ (MQSeries) queues
Data target or data warehouse (datamarts) on any DB2 UDB database as:
DB2 UDB for OS/400 DB2 UDB for z/OS and OS/390 DB2 UDB for UNIX or Windows NT/2000
Data target or multidimensional datamart integration that are built on IBM DB2 OLAP Server or Hyperion Essbase
Additional data administrator tools and complementary products may be used to enhance the warehousing solution for: Warehouse design and data modeling: Erwin Query reports: QMF for Windows, Brio Enterprise, Business Objects Long running queries monitoring and tuning: Query Patroller OLE DB sources support
Name and address cleansing: Support for INTEGRITY from Vality Support for Trillium Batch System
Additional more complex extraction and transformation than DB2 Warehouse Manager itself can achieve, the following products can also be integrated: ETI*Extract from ETI: development runs on Windows and UNIX environment and it generates code that can be run on many platforms DataStage from Ascential
The different components of DB2 Warehouse Manager are shown in Figure 1-2.
DB2 Warehouse Manager Architecture: Agent Based

Clients
Warehouse Server
Warehouse Agents
Data Data
Databases
End Users
Data Warehouse Center
Message Message
Relational Source
Message xx Windows 95/98, NT, 2000, AIX, Solaris

NT/2000 NT/2000 Agent
Data
DB2 Target
Data Metadata Message Data
NonRelational Source
Metadata yy Control Database Log Editions Configuration Data
Non-DB2 Target
Data
Type title
DB2
NT, 2000, OS/2, AIX, iSeries Sun, zSeries
1.Typ te e xt
Included with DB2 UDB
Flat Files, Web or SAP R/3
Figure 1-2 DB2 Warehouse Manager architectural view
The DB2 warehouse management functionality has been packaged in two parts. Part 1: This basic part is delivered with DB2 UDB servers V7.2 for any edition. It is visualized through the Data Warehouse Center, which is the graphical user interface for warehouse management, integrated under DB2 UDB Control Center. Installing DB2 UDB on Windows NT or 2000 platforms provides: Data Warehouse Center Warehouse server (Windows NT/2000 only) Default Windows NT/2000 warehouse agent, which can only be installed on the warehouse server machine DB2 OLAP Starter Kit, to start building multidimensional applications Note: DB2 UDB set up on AIX and SUN Solaris platforms provides also the Data Warehouse Center graphical user interface on these platforms. Part 2: This extended part is obtained by purchasing DB2 Warehouse Manager additional licences that include support for remote native platform execution, Data Warehouse Center extensions and additional data administrator tools. There are three DB2 Warehouse Manager offerings: DB2 Warehouse Manager V7.2 for UNIX, Windows and OS/2 is a priced option of a licensed copy of DB2 UDB Enterprise Edition V7.2 or DB2 UDB Enterprise - Extended Edition V7.2. Note: For each DB2 Warehouse Manager V7.2 ordered, there must also be a corresponding license of DB2 UDB Enterprise Edition or DB2 UDB Enterprise - Extended Edition V7.2 to which DB2 Warehouse Manager is licensed. DB2 Warehouse manager for UNIX, Windows and OS/2 provides: Data Warehouse Center extensions such as these: Additional distributed warehouse agents for AIX, Solaris, Windows NT/2000 and OS/2 platforms Note: Classic Connect ODBC drivers are provided with the Windows NT/2000 agent for access to IMS or VSAM sources on OS/390 and z/OS, but the Classic Connect product must be purchased separately. Extended transformers that are prebuilt stored procedures and User Defined Functions for warehouse and statistical transformations WebSphere MQ support ODBC targets
Information Catalog Manager to publish metadata DB2 Query Patroller, which may only be used on DB2 UDB Enterprise Edition or DB2 UDB Enterprise - Extended Edition, to which DB2 Warehouse Manager is licensed QMF for Windows, now integrated with Query Patroller and QMF queries, can be submitted to Query Patroller to take advantage of its additional capabilities. Add-on priced features called Connectors: DB2 Warehouse Manager Connector for Web, DB2 Warehouse Manager Connector for SAP R/3 DB2 Warehouse Manager V7.2 for AS/400 (iSeries) is a stand-alone product that provides: Data Warehouse Center extensions as: Additional distributed warehouse agent for iSeries platform that requires at least AS/400 V4.3 or later Extended AS/400 transformers that are prebuilt stored procedures and User Defined Functions for warehouse and statistical transformations that require AS/400 V4.5 with Java Stored Procedure environment Information Catalog Manager to publish metadata QMF for Windows that may only be used to access warehouse data stored on the iSeries machine DB2 UDB V7.2 Enterprise Edition with restricted use to the DB2 warehouse databases management DB2 Warehouse Manager V7.2 for z/OS and OS/390 (zSeries) is a feature code of DB2 UDB Server for OS/390 Version 7 and provides: Data Warehouse Center extensions such as: Additional distributed warehouse agent for zSeries (OS/390) that requires UNIX System Services and TCP/IP Extended zOS and OS/390 transformers that are prebuilt stored procedures and User Defined Functions for warehouse and statistical transformations that require OS/390 V7.2 with Java Stored Procedure support Information Catalog Manager to publish metadata DB2 UDB V7.2 Enterprise Edition with restricted use to the DB2 warehouse databases management The systems and platforms where these components reside are described in Table 1-1. We will use DB2 Warehouse Manager term generically to discuss components in both DB2 UDB and DB2 Warehouse Manager products.
Table 1-1 Platforms and components

W I N 9 5 / 9 8 W I N / N T W I N / 2 0 0 0 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y O S /2 A I X H P U X S O L A R I S Y L I N U X P T X A / S 4 0 0 O S /3 9 0
Data Warehouse Center Warehouse Server Warehouse Agent/Transformer Information Catalog Manager Connectors GUI Metadata Web SAP R/3
Y Y Y
Y Y Y Y Y Y
WebSphere MQ integration Classic Connect ODBC drivers QMF for Windows Additional tools Query Patroller OLE DB support Trillium integration
Y
Y Y Y Y
10
1.2.1 Warehousing tasks and objects

Building and populating a data warehouse (datamart) using DB2 Warehouse Management involves the following tasks and objects: Warehouse sources: Warehouse sources identify the source or operational data (relational database or view, local flat file, remote flat file) that has connectivity to your network and use sources specifications (as found out in system catalog for relational databases) to define access in metadata. Warehouse targets: Warehouse targets are database tables or files that contain data that has been transformed. Like a warehouse source, users can use warehouse targets to provide data to other warehouse targets or to build relational or multidimensional datamarts. Warehouse agents: Agents manage the flow of data between the warehouse sources and the target warehouses. They use Open Database Connectivity (ODBC) or DB2 Call Level Interface (CLI) to communicate with source and target databases but also may use programs and utilities. Several agents can handle the transfer of data, depending on the existing connectivity configuration and on the volume of data to move. Additional instances of an agent can be generated if multiple steps that require the same agent are running simultaneously. Processes and steps: A process contains a series of steps that perform a transformation and data movement for a specific warehouse use, in general into warehouse target. A process can produce a single table, a set of summary tables, or a flat file. However, a step is the definition of a single operation with DB2 Warehouse Manager, and can be any of the following: SQL steps: These use SQL SELECT statement to extract data from a source and generate an SQL INSERT statement to insert data in the warehouse target. More efficient database access are provided when SQL INSERT INTO... FROM SELECT statements are generated (see , Agent, source, and target are located on the same system on page 18). Transformer steps: These are stored procedures and user-defined functions to speed up warehouse development (see Figure 1-3) with statistical or specific warehouse transformers: To clean data To invert data To pivot data To generate primary keys To generate period tables To calculate various statistics (correlation, regression, chi-square, subtotals, moving average and so on)
11
Fast Deployment with Pre-built Transformations
Figure 1-3 Pre-built transformers
User Defined Program (UDP) steps: A UDP is an external program or script defined to the Data Warehouse Center that represents a business-specific transformation to be started by the Data Warehouse Center. A UDP can perform many different tasks, for example: Export data from a table Manipulate that data in regard to specific business requirements Write the data to a temporary output resource or a warehouse target
A step can be run on demand and can be scheduled to run at the same time. When scheduling a process, all the root steps in the process run at the scheduled time.
12
bd e m as n i eb d luo hs se lb at te g r a t/ ec ruos
seit reporp ro f kci l c t h g i r
s re mro f sn a r t esuoher aw
s re m ro fs n a r t l ac i tsi t a ts
1.2.2 The Data Warehouse Center

The Data Warehouse Center, integrated with the DB2 Control Center, provides the administrative graphical user interface for the data movement and transformation environment. The Data Warehouse Center is a warehouse management system that can build multi-tiered data warehouses and is totally driven by metadata. It can support sources of various types of relational databases, ODBC sources, replication tables, OLE-DB sources, application sources like WebSphere MQ queues, SAP R/3 and Web.
The Data Warehouse Center graphical interface

A graphical process modeler is used to design and model the warehouse processes in moving, transforming and loading data to the warehouse and the data marts. Data Warehouse Center also has extensive scheduling capabilities that allow you to automate all warehouse processes as well as a graphical process monitoring facility. Finally, Data Warehouse Center supports targets of DB2 UDB relational table, DB2 OLAP Server and Hyperion Essbase Server cubes and exports to flat files using the local agent. The Data Warehouse Center GUI Interface is shown in Figure 1-4.
13
Figure 1-4 Data Warehouse Center GUI Interface
The Data Warehouse Center eases the tasks necessary to do the following: Register and access data sources: DB2 UDB, Informix, Oracle, Microsoft SQL Server, Sybase, flat file sources, OLE-DB sources, application sources like WebSphere MQ queues, SAP R/3 and Web
14
Define extraction steps: Several extraction methods are available within Data Warehouse Center including SQL SELECT, DB2 UDB utility execution (load, import, export and so on). In addition, other extraction tools as ETI*Extract, cleaning tool as Vality INTEGRITY or user-written extraction program can be plugged into the process. Define transformation steps: There are over 100 built-in transformations leveraging SQL: Data Warehouse Center provides rich transformation features based on SQL. DB2 UDB provides a very rich SQL capability enabling a variety of transformation functions. DB2 Warehouse Manager provides specific transformers, including commonly used programs for database utility execution, file transfers, job submission, DB2 OLAP Server cube load and precalculation, and more. Users can extend transformation logic providing user defined functions or other executable. Define data movement and warehouse population: Data Warehouse Center supports several styles of data movement: Full table replacement using delete or load replace. Table histories using inserts or load append. Rolling periods (such as a rolling year) that automatically delete the oldest period and insert or append the most recent. Change replication to enable trickle feeding the warehouse for higher concurrency or managing network bandwidth constraints. Change replication leverages IBMs Data Replication technology that provides insert, update, and delete operations. Model, automate, and monitor processes: The following capabilities are supported for handling processes: Processes will be automatically launched at scheduled times. Processes can be triggered externally. If a problem occurs, retries will be executed automatically. Notifications may be sent out.
Publish metadata to Information Catalog Manager: The following capabilities are supported for handling metadata: Publish metadata from Data Warehouse Center to Information Catalog Manager. Manage and interchange metadata, importing metadata from partners tools as Erwin, ETI*Extract, Vality INTEGRITY and also importing and exporting metadata to and from eXtended Markup Language (XML) files that conform to the Common Warehouse Metamodel (CWM) standard defined by Object Management Group (OMG).
15
1.2.3 The warehouse server

The warehouse server interacts with Data Warehouse Center and warehouse agents. It manages the metadata definitions that describe the warehouse processes, automates the workflow for warehouse processes, monitors warehouse process activity, and gathers statistics about the completion of warehouse processes.The warehouse server only runs on Windows NT or Windows 2000.
1.2.4 The warehouse control database

The warehouse control database is a DB2 UDB database that hosts the metadata used to define and execute the warehouse processes. It must be stored in a DB2 UDB database on a UNIX or Intel server. It most often resides on the same platform as the warehouse server to provide local access performance. It can, however, be configured remotely.
1.2.5 The warehouse agent

DB2 Warehouse Manager is a scalable agent-based architecture and enables: Point-to-point fast data transfer when the agent is co-located with the target warehouse Multiple concurrent processes at a single agent site where the warehouse server instructs the agent to perform multiple steps concurrently. DB2 Warehouse Manager is then able to run multiple steps in parallel across servers Distributed processing across multiple agent sites when the transformation process specifies multiple agent sites for execution An agent is a process/job that acts as a medium for data flow between source and target databases. It resides on a machine named agent site, where agent code is installed that is registered as an agent site in warehouse server metadata. An agent is an instance of the agent code: Running on an agent site as a temporary process Performing data activities on behalf of a step (One step is executed by one single agent) Executing the command list received from the warehouse server Each step in a process being executed by a single agent
16
An agent is controlled by the warehouse server (see Figure 1-5). Agents are temporary processes invoked by the warehouse server via an agent daemon running on the appropriate agent site. They can issue a system command or run User Defined Programs, for example: Request data from the source or target data resources Initiate system commands on the agent site machine Run an user-defined program
Control Database
Source DB
Warehouse NT Server
Messages
Agent
Data Flow
Target DB Transformers
Figure 1-5 Agent-based architecture
The warehouse server also contains a warehouse agent called the local agent, that is defined as the default warehouse agent for all Data Warehouse Center activities. Note: The default warehouse agent is available only on Windows NT/2000. A warehouse agent started from the Data Warehouse Center on another agent site will be called the remote agent. The agent daemon is responsible for starting an agent process on the remote agent sites. It can also forcibly terminate an agent process if the agent ceases to respond to the warehouse server. These are the different agent configurations: Agent, source, and target are located on the same system. Agent and target are located on the same system. Agent and source are located on the same system. Agent, source, and target are located on three different platforms.
17
This last configuration is mentioned here to let you know that it exists and should be used exceptionally (for performances reasons and limitations on options) to source and target HP or LINUX data, for example.
Agent, source, and target are located on the same system

The warehouse agent may be installed on a system that contains both source and target tables, as described in Figure 1-6. Additionally, the source and target tables might live in the same or separate databases. If an SQL process is defined that the warehouse server believes the source and target to be in the same physical database. then it will generate an INSERT INTO.... FROM SELECT SQL statement and achieve the complete manipulation using only one SQL statement.
The agent, source, and target are located on the same machine
eServer: Agent site Agent Agent Daemon Source/Target
TCP/IP
Warehouse Manager Server
Figure 1-6 Agent configuration: all components on the same machine
In the configuration where the source and target are not in the same database, the agent: Passes SQL statements that extract data from the source tables on the local database Transforms the data if required Writes the data to the target table on the local database
18
Performance is no better here than when everything is on the same platform. However, performance is much better when source and target tables share the same database.
Agent and target are located on the same system

The warehouse agent may be installed on the system that contains only the DB2 UDB or DataJoiner target tables, as described in Figure 1-7.
The agent and target are located on the same machine

eServer:Agent site Agent
DRDA
Source
Target Agent Daemon
TCP/IP
Warehouse Manager Server
Figure 1-7 Agent configuration: agent located on the target server
In this configuration, the agent: Passes SQL statements that extract data from the remote source tables Transforms the data if required Writes the data to the target table on the local database This configuration offers the best performance in distributed environment by optimizing the DB2 data flow (DRDA or DB2 private protocols) and using block fetching for the extraction. This is the recommended configuration to use with DB2 Warehouse Manager when source and target are on physically separate machines.
19
Agent and source are located on the same system

The warehouse agent may be installed on the system that contains only source tables, as described in Figure 1-8.
The agent and source are located on the same machine
eServer:Agent site Agent DRDA DB2 private protocols Source Agent Daemon Target
TCP/IP Warehouse Manager Server
Figure 1-8 Agent configuration: agent located on the source server
In this configuration, the agent: Passes SQL statements that extract data from the local source tables. Transforms the data if required. Writes the data to the target table on the remote database. This configuration does not optimize the DB2data flow (DRDA or DB2 private protocols) and should be justified by specific architecture requirements.
1.2.6 Publishing metadata to Information Catalog Manager

Publishing metadata is the process of transferring metadata from the Data Warehouse Center to the Information Catalog Manager. Metadata published to Information Catalog Manager from the warehouse control database are stored in a specific database called Information Catalog. An information catalog is the set of tables managed by the Information Catalog Manager that contains business metadata that helps users identify and locate data and information available to them in the organization. When you publish metadata to the information catalog, you provide end users and system analysts with a fast path to seeing what is in the data warehouse.
20
1.2.7 Accessing data sources

These are the three main steps to set up data sources: Determine the data sources to use. Set up connectivity between sources and agent. Define the sources to the Data Warehouse Center as warehouse sources in order to register metadata on data sources. In this section, as the connectivity between sources and agent is one of the main causes of failure, we will focus on setting up connectivity between sources and agents. Data access is available as follows:
For relational databases:

DB2 UDB access in Data Warehouse Center/ DB2 Warehouse Manager can be done using DB2 Call Level Interface (CLI) or ODBC. However all non DB2 data are accessed via ODBC. They can also be accessed using DataJoiner. When sources (IBM DB2 UDB, IBM Informix, Oracle, Sybase, Microsoft SQL Server, and flat files) are mixed then everything is driven via ODBC. The use of ODBC requires that there is a lower level driver available to drive the specific database. So, for DB2 UDB this will be the DB2 Runtime Client which can access any DB2 family database. For other non-DB2 databases the appropriate client driver program is required as described in Table 1-2.
Table 1-2 Client driver program by system
RDBMS SYSTEMS Any platform For Windows NT/2000 For OS/2 For UNIX Version V7 V5,V6 and V7.x V7.x and V9.x Client enabler CAE V7 i-connect 7.2 i-connect 9.x Informix-Net for OS/2, Intersolv V3.6 driver manager and Informix driver Intersolv V3.6 driver manager and Informix driver V8.1.6 V8.1.6 ORACLE Net8 V8.1.6 Intersolv V3.6 Driver Manager and Oracle driver ORACLE SQL*Net V2.1.4 for OS/2, Intersolv V3.6 driver manager and Oracle drive
DB2 UDB Informix
Oracle
For Windows NT/2000 For UNIX For OS/2
21
RDBMS
SYSTEMS For Windows NT/2000 For UNIX
Version V6 V7
Client enabler DB-Library and Net-Library V6.0 DB-Library and Net-Library V7.0 Data Warehouse Center ODBC driver manager (Merant) Open Client Library v10.3 and the appropriate Sybase Net-Library Open Client Library v10 for OS/2, Intersolv V3.6 driver manager and Sybase driver Open Client Library v11.01 and the appropriate Sybase Net-Library Intersolv V3.6 Driver Manager and SYBASE driver
Microsoft SQL Server
Sybase
For Windows NT/2000 For OS/2 For Alpha systems For UNIX
For IMS and VSAM:

Use Classic Connect ODBC driver shipped with DB2 Warehouse Manager for both IMS and VSAM files on z-OS and OS/390 TCP/IP or APPC communications Alternative solutions may be used as described in Building the Operational Data Store on DB2 UDB, SG24-6513: Use IMS Data Propagator V3 and its MQSeries capability to load a data staging area in DB2 UDB Use WebSphere MQ (MQSeries) to read from VSAM files to MQSeries message queues and use DB2 Warehouse Manager MQ wizard facility to read these MQ queues.
For flat files on z-OS and OS/390:

Use File Transfer Program (FTP) or Network File System (NFS) to access data on the host Use TCP/IP for communications
For flat files on the LAN:

Use TCP/IP for communications with a pre-access or post-access command if required Windows NT/2000, AIX, SUN Solaris agents support the warehouse sources listed below. The Windows NT agent is provided by default with Data Warehouse Center.
22
AS/400 (iSeries) and OS/390 (zSeries) agents support access to DB2 UDB family databases. The OS/2 agent does not support Microsoft SQL Server, IBM Informix warehouse sources directly. Setting up warehouse sources with Data Warehouse Center is described, step-by-step, in Data Warehouse Center Administration Guide, SC26-9993.
1.2.8 DB2 Warehouse Manager Connectors

Priced features of DB2 Warehouse Manager V7.2, the major two additional connectors are: Connector for the Web operating on the same machine as the Windows NT/2000 or AIX or SUN Solaris agent Connector for SAP R/3 operating on the same machine as the Windows NT/2000 agent Connectors should always be installed, after the DB2 Warehouse Manager product has been installed on the appropriate platform.
Connector for the Web

This connector allows to integrate the Web channel data with enterprise data through the data warehouse. It brings the clickstream data into the data warehouse by tracking: Site health Traffic measures Visitor behavior It also can be used by users to integrate the Web traffic or Web channel data with Commerce data to evaluate marketing, merchandising and campaign effectiveness.
Prerequisites
The Web Connector supports Windows NT or 2000, AIX or SUN Solaris. It is installed on the same machine as the warehouse agent. Prerequisites before installing the Web Connector are: DB2 Warehouse Manager V7.2 product WebSphere Site Analyzer V4.0 package containing: Site Analyzer V4.0 DB2 UDB V7.2 WebSphere Application Server Advanced Edition V3.5.4
23
WebSphere Site Analyzer V4.0 (WSA) is a Web-based client-server system that: Captures raw data from different types of servers (raw data sources, also called Data Imports) as server logs: HTTP servers FTP servers WebSphere Edge Server WebSphere Portal Server WebSphere Application Server WebSphere Commerce Server WebSphere Personalization Server
Data is also captured as real time data or single pixel data capture feature through the browser (called Web Tracker). Does processing of Data Imports and populates an externalized Webmart star schema database stored in DB2 UDB or Oracle. Generates reports. WSA InfoCenter provides additional documentation on tuning and setup for WSA. WSA can be set up easily, the creation and launching of Web application servers being done automatically by the product install.
How does it work?

Integration between WebSphere Site Analyzer and DB2 Warehouse Manager is shown in Figure 1-9.
The Web Connector
Connector for the Web
Data Warehouse Center (DWC) DB2 Warehouse Manager
WebSphere Site Analyzer System
DB2 DB
Figure 1-9 The Web Connector integration
24
These are the main implementation steps, shown in Figure 1-10:
In WSA:
Define a project Define a Data Import
In Data Warehouse Center:

Define the new warehouse source Define the polling step to check that the Webmart has been populated Define a SQL step to populate the warehouse target Run steps using the Data Warehouse Center Work In Progress panel
clickstream assimilation Log files Single pixel data DB log tables
WSA WebMart
WebSphere Site Analyzer
WSA Polling step
Agent
Status = done
DWC SQL step
Warehouse Target
Warehouse WSA WebMart Source = WSA WebMart
Figure 1-10 WebSphere Site Analyzer and DB2 Warehouse Manager
Detailed information on integrating WSA and Data Warehouse Center and hints and tips to start implementing the Web Connector are described in Appendix A, Using DB2 Warehouse Manager Connector for the Web on page 313.
25
Connector for SAP R/3

SAP R/3 (for Systems, Applications and Products in Data Processing) is SAPs integrated business software solution for client/server and distributed open systems to handle all of the business management tasks of a company. This software is highly customizable using SAPs proprietary programing language, ABAP. DB2 Warehouse Manager Connector for SAP R/3 reads the SAP R/3 Business Object Repository (BOR) to retrieve all the Business Objects (BO) from SAP R/3 and use SAP Business Application Programming Interface (BAPI) specific methods to access and read data. BAPIs provide consistent and persistent data but do not provide a mechanism to capture just the deltas. The metadata of Business Objects is read from SAP R/3 Business Object Repository (BOR) and can be defined as data source on DB2 Warehouse Manager: The BOR represents the business data, transactions and events within the SAP R/3 system as an integrated set of SAP Business Objects composed of data, business rules and constraints The BOR manages these objects which are related to each other in objects models
Prerequisites
The SAP R/3 Connector supports only Windows NT and 2000 and can access any SAP R/3 server platforms. It is installed on the same machine as the warehouse agent. Prerequisites before installing the SAP R/3 Connector are: DB2 Warehouse Manager V7.2 product SAP R/3 Release 4.6C with an English language option installed RFC Runtime module (LIBRFC32.DLL) from the presentation CD-ROM, SAP Release 4.6D Compilation 4 included with the SAP clients Optionally install SAPGUI for troubleshooting
How does it work?

The DB2 Warehouse Manager Connector for SAP R/3 accesses SAP R/3 data via two BAPIs, GetList and GetDetail of any released Business Objects stored in the BOR:
GetList is used to search for object instances that fulfill a certain selection criteria and returns a list of object key fields GetDetail obtains more information about a specific instance identified in the object key field
26
Integration between SAP R/3 and DB2 Warehouse Manager is shown in Figure 1-9.
The SAP R/3 Connector

BO Namelist BO metadata BAPI:
GetList GetDetail
SAP R/3
Connector for SAP R/3
Data Warehouse Center (DWC) DB2 Warehouse Manager
BOR
DB2 DB
Figure 1-11 The SAP R/3 Connector integration
These are the main implementation steps, in Data Warehouse Center: Define SAP as a new warehouse source: Specify the right type of connection Select SAP R/3 Business Objects Define properties of Business Objects by selecting and mapping its metadata (parameters) Define the SAP R/3 process using Data Warehouse Center process model: Select Business Object input parameters Select output parameters to be mapped to columns Run the step using the Data Warehouse Center Work In Progress panel. Detailed information on integrating SAP R/3 Servers and Data Warehouse Center and hints and tips to start implementing the SAP Connector are described in Appendix B, Using DB2 Warehouse Manager Connector for SAP on page 331.
1.2.9 Additional tools and functions

DB2 Warehouse Manager provides also the following tools and functions:
27
OLE DB access: DB2 Warehouse Manager can access an OLE DB source from a view of such table function and allows the user to invoke the OLE DB Assist wizard to create the necessary table function and view. The view definition is then imported into DB2 Warehouse Manager to create a warehouse source view. The view can be used as any other DB2 UDB view. The OLE DB support is Windows only. WebSphere MQ integration: DB2 UDB V7.2 marries the cross platform integration strengths of WebSphere MQ with the rich function of SQL for building near real time data warehouses. A set of MQSeries functions are provided with DB2 UDB V7.2 to allow SQL statement to include messaging operations. This support is available to applications written in any supported language for example C, Java, SQL (stored procedures) using any of the database interfaces.
These functions are used within DB2 Warehouse Manager to read data from message queues. DB2 Warehouse Manager provides an MQ Assist Wizard which makes it relatively easy for the user to define the queues as sources. The user invokes the MQ-Assist wizard to: Define the format of the MQSeries messages as fixed or delimited strings Create a table function to expose the MQSeries queue as a table Create a view for this table function The view definition is imported into the Data Warehouse Center (see Figure 1-12) and the new warehouse view source then can be accessed and manipulated with Data Warehouse Center as any other warehouse source. This requires DB2 UDB 7.2 Server to be on the same system as the warehouse agent and WebSphere MQ V5.2.
Figure 1-12 MQ Wizard
28
Message queues can also be populated from XML documents. DB2 Warehouse Manager reads the XML data from MQSeries using UDFs and stored procedures provided and populates tables in a warehouse target using DB2 XML Extender functions. Examples of WebSphere MQ integration with DB2 Warehouse Manager are also provided in the redbook Building the Operational Data Store on DB2 UDB, SG24-6413.
Cleansing integration with Trillium Batch System (TBS): Trillium Software System (TSS) is a name and address cleansing product which reformats, standardizes, and verifies address data to produce usable information.
TSS V4 consists of 4 components: Converter Parser Geocoder Matcher
The components can be invoked by set of executables (the Trillium Batch System) and the user writes a command file which calls these executables to execute a name and address cleansing task. DB2 Warehouse Manager provides: The Import Metadata dialog to import the metadata of the TBS command file and create the Trillium User Defined Program (UDP) step The Trillium UDP to execute the TBS command file The import metadata dialog for TBS will create the following Data Warehouse Center objects: Warehouse source Warehouse target TBS User Defined Program (UDP) step in a subject area and process When invoked, the step will execute the TBS command file.The command file can either be local or remote.
QMF for Windows: QMF for Windows provides a tightly integrated, powerful, reliable, scalable query and reporting tool set for IBM's DB2 Family of database systems. QMF is positioned in the easy-to-use query and report writing space for advanced users with some knowledge of data structures, providing as output, charts, reports and applications that may be accessed in client/server mode or using the Web.
29
Query Patroller: Query Patroller is the tool to manage the query workload by tracking resource usage.
Query Patroller allows the database administrator to: Govern queries and trap all dynamic queries Analyze costs Manage resources that means limit workload by server or node and balance workload across node or server clusters Track and report usage Retry automatically the in-progress queries after the DB2 UDB warehouse target is restarted following a failure Query Patroller allows the end-users to: Schedule and prioritize queries for asynchronous execution Save results for later use Reuse result sets from previous queries
30
Chapter 2.
Possible failure scenarios and their consequences

While hardware and software is often extremely reliable, there is no guarantee against failure, and there is certainly no absolute guarantee against events such as a natural disaster, an act of war or sabotage, or any other unpredictable disaster. The objective of this chapter is to illustrate possible failures (however unlikely these may be) in hardware or software failures which can affect a DB2 Warehouse Manager environment. We will discuss some of the consequences. We also provide also some recommendations on configuration requirements before starting to use DB2 Warehouse Manager.
2.1 Hardware server failures

In a typical warehousing scenario, there are a number of machines or systems performing a server role; for example: The database server(s) that hold the target database The source server(s) from which the data is extracted A warehouse server that runs the warehouse schedule and maintains the warehouse management repository
31
We will take each of these in turn. Assume a scenario, which is reasonably typical: The source data is on a remote server (for example, on iSeries, zSeries) The target warehouse database is on a UNIX system. The warehouse server and its warehouse control database is on a Windows 2000 system. In this situation it is highly likely that the data access will be performed by distributed SQL access, and for best performance, we would install a warehouse agent on the UNIX box, accessing the AS/400 or iSeries by a DB2 DRDA mechanism. The warehouse server will issue requests to the agent sitting on the UNIX box which in turn will place an SQL request against the AS/400 (iSeries). So this is just like an application running on the UNIX machine accessing the AS/400 (iSeries) by distributed SQL.
2.1.1 Source server failure

If, for example, the iSeries source server or the communication link between the source server and agent fails for any reason, the agent code running on the UNIX machine will receive some kind of SQL error. The agent in turn will signal back to the warehouse server that there was a failure and will return error information that goes into the warehouse log. This will be true even if there is no explicit failure on the source server, for example, if the source server goes into some wait condition then the SQL request from the agent to the source server will time-out at some stage. Provided that the step which the agent was running has not performed any commits to the target database then no permanent updates will have been performed by that step and, once the reason for the source server failure has been solved, the step may be re-run without fear of duplication or overwriting. You should be aware that within an SQL step in DB2 Warehouse Manager, there is an option to perform incremental commits (that is, for the step to perform a commit every n rows). If restartability is required, then this option must be avoided. Should the source server failure be caused by some corruption of the source server, then a restoration or recovery of that source server will need to be performed. This is outside of the scope of the warehouse server and warehouse management.
32
2.1.2 Target server failure

Assuming we follow general guidelines and place the warehouse agent on the same machine as the target database, then a hardware failure of the server could affect the behavior of both the target database and the agent. If the failure were extremely severe, then it is reasonable to assume that: The agent code would not be able to complete. The warehouse server would not receive any completion signal of any kind from the agent. In this case, if the step the agent code was running at the time was an SQL step and there was only one unit of work involved then it is reasonable to assume that DB2 will take care of the restart situation and will ensure that no permanent updates took place against the target database. On the warehouse server side, this would almost certainly receive some time-out condition which would be interpreted by the warehouse server as a failure to run the step. So, with the provision that no updates had been performed against the target database for that step, it will be safe to re-run the step that failed and continue from there. It is more likely that a failure in the target server would be of a much less severe nature for example, there could be a disk error. In these cases, there are two possible outcomes. If the agent keeps running, then it should report back the failure to the warehouse server log. If there is some breakdown in communication or a wait, then the server-agent session will time-out. In either case, the server should report failure.
2.1.3 Warehouse server failure

This concerns any server crash that may occur. If a serious failure occurs within the machine that the warehouse server is running on, then the warehouse server will almost certainly need to be restarted. Depending on the severity (for example if there was a permanent disk failure) then it may need restarting on a different machine. Remedying this type of failure will be covered in more detail later.
2.2 Network failures

Following on from the previous situation, considering failure of the target server (and thus failure of the agent code), an out-of-line situation might occur if the agent performed the step's work satisfactorily and committed the work to the database and then failed to communicate satisfactory completion back to the warehouse server. This might happen if for some reason the communications link should fail at the precise moment the agent was sending the feedback information.
Chapter 2. Possible failure scenarios and their consequences
33
In this case the data updates would have been committed permanently but because a time-out was received from the agent connection, the warehouse server would believe the step did not complete. Since it would not appear to complete the assumption has to be made that the step did not run and therefore that the updates were not applied. This means that a network failure would appear to be potentially the most problematic when deciding whether to restart work or not. To be on the safe side, every time there has been a failure due to a communication error, the safest action should be to check what data has been committed to the database and take the appropriate action. Log messages give the details of what communications and commands transpired between the server and the agent. Log messages should be checked to see if the server had sent a commit/rollback before the failure occurred.
2.3 Disk failures

There are two potential types of failure: Those that are recognized and that generate some error situation. Those that occur but do not immediately generate an error. Since the vast majority of data access and update will be through the relational database manager then we can assume that DB2 UDB will be handling the error situations that present themselves during the unit of work. If there is a recognized error then the normal processing will be to rollback and any work will not be committed. For situations (if any) which take some time to unfold, then the disk may have gotten into an unrecoverable condition. In this case, one would have to rely on recovering the databases from earlier copies and the logs. This will be covered later.
2.4 Failures in DB2 Warehouse Manager server

It is reasonable to assume that a serious hardware or software failure in the warehouse server will cause a serious disruption of the warehousing service. If a server stops/crashes when it is processing steps, when the server is restarted, it will mark all jobs failed that were in a populating state.
34
What is important is to understand what to do to recover the situation, how to understand what the latest status of the target database(s) is and which processes and steps have to be re-run (if any). Let us consider several situations:
1. The processor or similar (non-disk) hardware becomes non-functional.

Since the DB2 Warehouse Manager server runs on a standard Windows NT or Windows 2000 base, it might be sufficient to merely swap the disk drives to an alternative Intel machine, reboot, and restart DB2 Warehouse Manager. You must assume that the Intel machine will take the exact same place in the network as the problematic one, otherwise you will have to reconfigure some services in addition to TCP/IP settings so that the client and agents are able to talk back to the server. All those processes/steps which were in flight would almost certainly be registered as still running or completed unsatisfactorily. To check to see if the job finished, agent logs can be used if you are running with agent logging on. The target database state can also be checked.
2. The disk hardware becomes corrupted or malfunctions in some way.

The conclusion must be that in this situation there is no alternative but to reinstate the warehouse control database. This might in exceptional circumstances involve replacing the disk subsystem hardware. Even if that is not required, there almost certainly will be a requirement to restore the warehouse control database from a backup. Chapter 5, Warehouse control database recovery scenarios on page 147 deals with recovery of the warehouse control database. When the DB2 Warehouse Manager system is set up, a policy decision needs to be made as to what form of logging is to be specified for the warehouse control database. This decision will dictate what the recovery process will be. Another way may be to install the warehouse control database on a separate DB2 server from the warehouse server. This way, the only thing to reinstall is the warehouse server.
3. Software or user corruption of the warehouse control database occurs.

The remedial action for this situation will be similar to situation (2). It is worth noting that various hardware and software solutions exist to help minimize the likelihood of such problems described above. Other solutions exist which, while they cannot help prevent the specific problem, help minimize the recovery time. For example:
35
Disks are available as RAID arrays. This can help keep the disk subsystem on-line and allow hot swapping of the faulty device(s) at the convenience of the support staff. Server systems can be configured in such a way as to be able to take over devices belonging to another server. Let us return to the issue of whether to use circular logging or logging which allows roll forward recovery. If you are reading this book, then you are most likely interested in providing a DB2 Warehouse Manager environment that provides a reasonably high service level and one where relatively High Availability of the latest data is important. In this case, our recommendation would be to set up the warehouse control database with roll forward recovery enabled. This means that special measures will need to be taken to archive the logs to a suitable place and to make sure that those logs will be available in a timely way should recovery be required. This is a standard DB2 planning and administrative activity. Once roll forward recovery is enabled, an operations or administration person can restore the database from a backup and also apply all the logs to a point in time thereby ensuring that warehouse control database is restored to its latest possible status. Without roll forward recovery (or in other words with circular logging) the warehouse control database can be restored to the status it had at the last backup. This means that information about the work in progress and related information updated since the last backup would be lost.
2.5 Agent failure

Some of the issues regarding agent failure have been covered earlier. To be more precise, agent failure (or failure while the agent is processing a step) might be caused by a number of things: As mentioned before, there could be an error in source or target systems. If the agent remains running, then the error should be reported back to the server unless at the same time there is some communications failure between warehouse agent and warehouse server. In the very unusual circumstances that the communication failed at the same time then the server would assume that the step did not complete which would in this case be true. Strangely enough, we have to watch for cases when there was no agent failure when the step has completed satisfactorily, but this fact could not be communicated back to the server.
36
The operating system on which the agent is running may perform some strange behavior, causing unpredictable actions. It is highly unlikely, but possible, that the operating system on which the agent is running may perform strange behavior, thus causing unpredictable actions. The most likely consequence of this will be some kind or error reported to the server or a time-out reported at the server for that agent. In all cases, the server will assume that the step did not complete, and it is fair to say it is unlikely that the step did actually complete on that failed system.
2.6 Recommendations
When starting to plan and design your warehouse system, these are the main recommendations Take time to brainstorm a list of possible failure scenarios. Agree upon their degree of criticality to your environment, and then decide which scenarios must be accommodated in your system. Document what action should be taken for each of the important scenarios, should they occur. These considerations will be important for the operations staff of the warehouse. Chapter 6, Problem determination methodology on page 229, which covers problem determination in a DB2 Warehouse Manager environment, would help you understand what failure scenario has occurred.
37
38
Chapter 3.
Database backup and recovery

Given the increasing competition in todays tough business climate, it is vital that organizations provide cost-effective and rapid access to business information for a wide range of business users. The access to business intelligence system became a day-to-day operation, just like any other operational transaction processing system, and it is key to any data warehouse to minimize the effect of any hardware or software failure and prevent any data loss. When using DB2 Warehouse Manager the vast majority of the metadata and the target data will be held in DB2 UDB, respectively in the warehouse control database and in the target warehouse(s). One of the main ways of ensuring the integrity of the data is to put in place the appropriate backup and recovery procedures. This chapter explains the issues and discusses the possible database backup and recovery solutions including disaster recovery solution, when building the warehouse control database and the target warehouses on DB2 UDB on Intel or UNIX platforms. When target warehouses are implemented on AS/400 (iSeries) or OS/390 and z/OS (zSeries) platforms, specific backup and recovery procedures on these platforms should also be considered.
39
3.1 Overview
A database can become unusable because of hardware or software failure, or both. You may, at one time or another, encounter storage problems, power interruptions, and application failures, and different failure scenarios require different recovery actions. You can protect your data against the possibility of loss by having a well rehearsed recovery strategy in place. DB2 UDB provides a range of facilities for backing up, restoring, and rolling data forward, which enable one to build a recovery procedure. Good warehousing practice covers reliability of the target databases together with the warehouse control databases. Just protecting the target databases is not enough if you wish to keep an operational service running satisfactorily. Similarly just maintaining backup of the warehouse metadata helps your IT staff but doesn't satisfy the users who need the data from the target database Some of the questions that you should answer when developing your recovery strategy are: Will the database be recoverable? How much time can be spent recovering the database? How much time will pass between backup operations? How much storage space can be allocated for backup copies and archived logs? Will tablespace level backups be sufficient, or will full database backups be necessary? Your database recovery strategy should ensure that all information is available when it is required for database recovery. You should include a regular schedule for taking database backups. You should also include in your overall strategy procedures for recovering command scripts, applications, user-defined functions (UDFs), stored procedure code in operating system libraries, and load copies. The concept of a database backup is the same as any other data backup: taking a copy of the data and then storing it on a different medium in case of failure or damage to the original. The simplest case of a backup involves shutting down the database to ensure that no further transactions occur, and then simply backing it up. You can then rebuild the database if it becomes damaged or corrupted in some way. The rebuilding of the database is called recovery. Different recovery methods are discussed to fit with your data warehouse business requirements.
40
3.2 Planning considerations

Planning is one of the most important areas for consideration before beginning to do database backups. We will cover the factors which should be weighed against one another in planning for recovery, for example, type of database, backup windows and relative speed of backup and recovery methods. We also introduce various backup methods.
3.2.1 Speed of recovery

If you ask users how quickly they would like you to be able to recover lost data, they usually answer immediately. In practice, however, recovery takes time. The actual time taken depends on a number of factors, some of which are outside your control (for example, hardware may need to be repaired or replaced). Nevertheless, there are certain things that you can control and that will help to ensure that recovery time is acceptable: Develop a strategy that strikes the right balance between the cost of backup and the speed of recovery. Document the procedures necessary to recover from the loss of different groups or types of data files. Estimate the time required to execute these procedures (and do not forget the time involved in identifying the problem and the solution). Set user expectations realistically, for example, by publishing service levels that you are confident you can achieve.
3.2.2 Backup window

DB2 UDB allows online backup and recovery as well as offline backup and recovery. Offline backup does not allow databases to be backed up while they are in use. In such cases, you need to force off all the applications connected to the database before the backup starts, and you cannot restart the database until after the backup has completed. This means that users cannot use applications.You need to ensure that the times at which databases are shut down and unavailable are acceptable to your users.
Chapter 3. Database backup and recovery
41
Even if you can perform backups while the database is operational, you need to ensure that any load on processors or networks caused by the backup process does not result in performance or response degradation that is unacceptable to your users.
3.2.3 Recovery points

You need to define the points in time to which you will restore data. For example, you may need to recover the data to the state it was in when the last transaction was completed. Alternatively, it may be acceptable to restore the data to a consistent state that is no more than 24 hours old. In addition to either of these, you may be required to restore individual tables to the state they were in at any particular date within the last 30 days. Whatever your situation, you need to consider recovery points and define a policy that is both achievable and acceptable to your user community.
3.2.4 Unit of recovery

In some circumstances, it may not be sufficient to restore individual tables (or even entire databases) to the state they were in at some point in the past. Sometimes, in order to maintain data consistency, you may need to restore data held in tables or files that have not been lost or damaged. This undamaged data needs to be restored to the same point-in-time as the damaged data. In developing your backup strategy, you need to understand the relationships between the data objects on which user applications rely. Many applications rely on relationships that extend beyond the data held in a single database. For example, an engineering database application holds references to documents that exist as independent file system files. If the engineering database is lost and restored to the point-in-time at which the last backup was taken, references to documents may be lost. Alternatively, the medium on which some of the documents are stored may be damaged. If the data files used to hold the documents are restored to the point-in-time at which the last backup was taken, the engineering database may contain references to documents that do not exist. There are many other situations where you need to ensure that data consistency is preserved.
42
The key point is that your backup and recovery strategy must take into account the needs of the applications that use the data.
3.2.5 Backup of other supporting files

There are a certain number of files associated with DB2 UDB which need to be backed up in addition to the database itself. These files can be database configuration parameter files, initialization parameter files, password file, files that define the environment or network configuration files. They are external files and are not part of the database, because they must be accessible for reading, or editing even when the database is down. For example, the password file provides authentication for database administration especially when starting up a database from a remote site. You must ensure that these files are also backed up.
3.2.6 Keeping related data together

As part of your database design, you will know the relationships that exist between tables. These relationships can be expressed at the application level, when transactions update more than one table, or at the database level, where referential integrity exists between tables, or where triggers on one table affect another table. You should consider these relationships when developing a recovery plan. You will want to back up related sets of data together. Such sets can be established at either the tablespace or the database level. By keeping related sets of data together, you can recover to a point where all of the data is consistent. This is especially important if you want to be able to perform point-in-time rollforward recovery on tablespaces.
3.2.7 Using different operating systems

When working in an environment that has more than one operating system, you must consider that in most cases, the backup and recovery plans cannot be integrated. That is, you cannot usually back up a database on one operating system, and then restore that database on another operating system. In such cases, you should keep the recovery plans for each operating system separate and independent.
43
If you must move tables from one operating system to another, you can use the db2move command, or the export utility followed by the import or the load utility. For more information about these utilities, see the Section 3.8, Useful data movement tools on page 88.
3.3 Recovery concepts and terminology

To help you better understand the task of recovering a damaged database, we describe some of the concepts and terminology as: Unit of work Recovery and backups methods Database logs Recovery history file Disaster recovery
3.3.1 Unit of work

In DB2 UDB terminology, a unit of work (UOW) and a transaction correspond to the same concept. A UOW is a recoverable sequence of operations within an application process. A UOW is initiated when an application process is started. A UOW is also initiated when the previous unit of work is ended by something other than the termination of the application process. A UOW is ended by a commit operation, a rollback operation, or the end of an application process. A commit or rollback operation affects only the database changes made within the UOW it ends. While these changes remain uncommitted, other application processes are unable to perceive them. Once committed, these database changes are accessible by other application processes and can no longer be backed out by a rollback.
3.3.2 Recovery methods

In this section, we discuss the basic recovery methods provided by DB2 UDB.
Crash recovery
Crash recovery is the automatic recovery of the database if a failure occurs before all of the changes that are part of one or more units of work (transactions) are completed and committed.
44
DB2 UDB will automatically roll back incomplete transactions and complete committed transactions that were in memory when the crash occurred. If a failure occurs before all of the changes that are part of the unit of work are completed and committed, the database is left in an inconsistent and unusable state. Crash recovery is the process by which the database is moved back to a consistent and usable state. This is managed by the DB2 UDB database manager itself. If you want the rollback of incomplete units of work to be done automatically by the database manager, you want to enable the automatic restart (AUTORESTART) database configuration parameter by setting it to ON. (This is the default value.) The first connection to the database after the crash will cause the database to be restarted or placed into a consistent status. If the AUTORESTART parameter is set to OFF, you must manually execute the restart database command. Figure 3-1 shows the effect of a power failure.
R E S TA R T D B U N IT S O F W O R K D C B A RO LLBACK RO LLBACK
CRASH T IM E
Figure 3-1 Crash recovery
At the time of the crash, units of work in application A and C were in progress but not yet committed. The units of work in application B and D had already been committed. The changes made by application D had been committed but the modified data pages had not been permanently written to the database on disk. When the database is restarted, application D's changes are written on disk and the units of work in application A and C are rolled back. After these events are completed, the database is placed into a consistent state.
45
When DB2 UDB is restarted, you can check the status in the database configuration file.
Attention: The database configuration file would only be useful to check when the database is shutdown. As soon as an application connects to the database, the database is marked as inconsistent and when the database shuts down properly, the database is marked consistent.
Figure 3-2 shows the status of the TBC_MD database.
Figure 3-2 Database status from the Control Center
You can display this screen by selecting Configure...>Status against the TBC_MD database in the Control Center.
46
Restore recovery
Sometimes this method of recovery is called a version recovery. Restore recovery is the restoration of a previous version of the database, using an image that was created during a backup operation at an earlier time. You use this recovery method with databases for which you do not have archived logs. You can also use this method with recoverable databases by using the WITHOUT ROLLING FORWARD option on the restore database command. A database restore operation rebuilds the entire database using a backup image created earlier. A database backup allows you to restore a database to a state identical to the one at the time that the backup was made. All the modifications made in the database after the time of the backup are lost. Figure 3-3 shows an example of restore recovery. A database backup was taken at time t1. Between t1 and t3, some database tables were loaded in the tablespace TBSP1. Just before time t3, the tablespace TBSP1 is dropped by mistake. To recover the tables in TBSP1 at the time of t1, the BCK1 backup could be restored.
DROP TABLESPACE TBSP1 UNITS OF WORK

t2 t3
t1
time
BCK1
RST1
RESTORE DATABASE
Figure 3-3 Restore recovery
Using the restore recovery method, you must schedule and perform full backups of the database on a regular basis.
47
The restore only method involves making an offline, full database backup copy at scheduled times. With this method, the restored database is only as current as the time that the last backup was made. For instance, if you make a backup copy at the end of each day and you lose the database midway through the next day, you will lose a half-day's worth of changes. The default values of the relevant database configuration parameters are set to use this method of recover: logretain=NO and userexit=NO. A full database backup involves making copies of: User tables used to hold user data System catalog tables used by DB2 UDB Any control files and parameter files that DB2 UDB uses DB2 UDB allows you to perform full database backup when the database is either online or offline. An offline backup is performed when the only process connected to the database is the backup process itself. Offline backup involves shutting the database down before you start the backup and restarting the database after backup is complete.Offline backups are relatively simple to administer. You need to schedule sufficient time to perform the backup to ensure that the periods when the database will be unavailable are acceptable to your users. The simplest approach to database backup is to perform only full, offline backups at regular intervals. This approach is relatively easy to administer, and recovery is relatively straightforward.
Rollforward recovery
Rollforward recovery is the reapplication of transactions recorded in the database log files after a database or a tablespace backup image has been restored. If you cannot afford to risk losing changes made after a database backup, you should plan to use rollforward recovery. To use the rollforward recovery method, you must have taken a backup of the database, and archived the logs by enabling either the logretain or the userexit database configuration parameters, or both.
In the rollforward recovery method, changes made to the database are retained in logs. With this method, you first restore the database or the tablespaces using a backup copy. Then you use the logs to reapply changes that were made to the database since the backup copy was created. There are two types of rollforward recovery to consider:
48
Database rollforward recovery: In this type of rollforward recovery, transactions recorded in database logs are applied following the database restore operation. The database logs record all changes made to the database. This method completes the recovery of the database to its state at a particular point-in-time, or to its state immediately before the failure (that is, to the end of the active logs.) Tablespace rollforward recovery: If the database is enabled for forward recovery, it is also possible to back up, restore, and roll tablespaces forward. To perform a tablespace restore and rollforward operation, you need a backup image of either the entire database (that is, all of the tablespaces), or one or more individual tablespaces. You also need the log records that affect the tablespaces that are to be recovered.
You can roll forward through the logs to one of two points: The end of the logs (called full recovery) or A particular point-in-time (called point-in-time recovery) You can make a tablespace rollforward recovery in the following two situations: After a tablespace restore operation, the tablespace is always in rollforward pending state, and it must be rolled forward. You need to invoke the rollforward database command to apply the logs against the tablespaces to either a point-in-time, or to the end of the logs. If one or more tablespaces are in rollforward pending state after crash recovery, you should correct the tablespace problem first. In some cases, correcting the tablespace problem does not involve a restore database operation. For example, a power loss could leave the tablespace in rollforward pending state. In this case you do not have to do a restore database operation. Once the problem with the tablespace is corrected, you can use the rollforward database command to apply the logs against the tablespaces to the end of the logs. If the problem is corrected before crash recovery, crash recovery may be sufficient to take the database to a consistent, usable state.
Note: If the tablespace in error contains the system catalog tables, you will not be able to start the database. You must restore the SYSCATSPACE tablespace, then perform rollforward recovery to the end of the logs.
Figure 3-4 shows an example of rollforward recovery. We can restore the tablespace TBSP1 and all the changes made to the database since the backup BCK1 by first restoring the BCK1 and then performing a rollforward to a point-in-time just before the drop tablespace command was issued.
49
D R O P TA B L E S P A C E T B S P 1 U N IT S O F W O R K
t2 t3
t1
t im e
BCK1
RST1
1.
RESTO RE D ATABASE
2.
R O LLFO RW A R D D A TA B A S E
Figure 3-4 Rollforward recovery
With rollforward recovery enabled by setting the logretain=RECOVERY and/or userexit=YES, you can take advantage of online backup and tablespace level backup. When performing an online backup, other applications can be connected to the database. For full database rollforward recovery, you can choose to recover to the end of the logs or to a specified point-in-time. Clearly, if a database is being backed up while users are updating it, it is likely that the data backed up will be inconsistent. DB2 UDB uses log files during the recovery process to recover the database to a fully consistent state. This approach requires that you retain DB2 UDB log files and indicate to DB2 UDB when you are about to start the backup and when you have completed the backup. To perform a point-in-time recovery, you must specify the time in Coordinated Universal Time (CUT). The format is yyyy-mm-dd-hh.mm.ss.nnnnnn. The time stamp used on the backup is based on the local time that the backup started.
Note: The special register, CURRENT TIMEZONE, holds the difference between CUT and the local time. Local time is the CUT plus the value of CURRENT TIMEZONE.
50
DB2 UDB allows you to quiesce activity on portions of the database (for example, a particular tablespace) so that a set of complete tables is temporarily frozen in a consistent state. You then can back up the set of tables that has been frozen. Once the backup has completed, you can reactivate the tablespace. Make sure that the subset you back up (as part of a partial backup) represents a complete logical unit of recovery from the point of view of the application. You may also need to back up data files that DB2 UDB does not manage. You must also ensure that the unit of recovery is consistent from the point of view of DB2 UDB. If you have added a new data file to a tablespace, you must ensure that any control file that DB2 UDB uses to define the relationship between data files and tablespaces is also backed up
Incremental backup
DB2 UDB provides backup facilities for data that has changed since the last offline or online database backup. This will save tape or disk space but may or may not reduce the time to do a backup, because the DB2 UDB still needs to read the data blocks to determine if it they have changed since the last backup. When recovery is needed, the database backup and incremental backups are both required to fully recover the database. This can take more time to recover a database. Figure 3-5 illustrates two examples of incremental backup.
Sunday
Mon.
Tue.
Wed
Th.
Fri.
Sat
Sunday
Full
Full
Cumulative Backups
Full
Delta Backups
Full
Figure 3-5 Incremental backup
51
In the first example, only cumulative backups (or incremental backups) are taken. Each consecutive cumulative backup will grow as it contains all of the changes since the last full backup. However in the case of a recovery, it will be faster because only the last full backup and most recent cumulative backup image would be necessary. In the second example, only delta backups are taken. Each consecutive delta backup will only contain the changes since the last delta backup. In this way, you can reduce the size of each delta backup in comparison to the previous example. However it will be slower for you to recover because, in addition to the last full backup, you would need all delta backups up to the failing point. Incremental backups are useful when saving space or when saving bandwidth when backing up over the network.
3.3.3 Database logs

In DB2 UDB databases, log files are used to keep records of all data changes. They are specific to DB2 UDB activity. There are two basic types of DB2 UDB logging: Circular logging Archive logging In addition, capture logging is available for replication purposes. Each type of logging corresponds to the method of recovery you want to perform. Circular logging is used if the maximum recovery you want to perform is crash or restore recovery. Archive logging is used if you want to be able to perform rollforward recovery.
Circular logging
Circular logging is the default behavior when a new database is created. (The logretain database configuration parameter setting is NO.) With this type of logging, only full, offline backups of the database are valid.
As the name suggests, circular logging uses a ring of online logs to provide recovery from transaction failures and system crashes. The logs are used in a round-robin fashion and retained only to the point of ensuring the integrity of current transactions. Circular logging does not allow you to roll a database forward through transactions performed after the last full backup operation. All changes occurring since the last backup operation are lost. Only the crash recovery and restore recovery can be performed when this type of logging.
52
Active logs are used during crash recovery to prevent a failure (system power or application error) from leaving a database in an inconsistent state.
The data changes are recorded in the log files and when all the units of work are committed or rolled back in a particular log file, the file can be reused. The number of log files used by circular logging is defined by the logprimary and logsecondary database configuration parameters. If there are UOWs running in a database using all the primary log files and still not reaching a point of consistency, then a secondary log files are allocated one at a time. Figure 3-6 shows how log files are used in circular logging.
DB2 server
Transaction
Circular Logs
Database Log Path
Active Log Files
Active Log Files
Figure 3-6 Circular logging
Archive logging
Archive logging is used specifically for rollforward recovery. You can configure this logging mode by setting the logretain database configuration parameter to RECOVERY.
Rollforward recovery can use both archived logs and active logs to rebuild a database or a tablespace either to the end of the logs, or to a specific point-in-time. The rollforward utility achieves this by reapplying committed changes found in the following three types of log files:
53
Active logs: Crash recovery also manipulates active logs, which uses them to place the database into a consistent state. They contain transaction records that have not been committed and also the committed transaction information that has not been written to the database on disk. You can locate active log files in the LOGPATH directory. Online archived logs: When changes in the active log are no longer needed for normal processing, the log is closed, and becomes an archived log. An archived log is said to be online when it is stored in the database log path directory (see Figure 3-7).
DB2 server
Transaction
Online Archived
Database Log Path
Active Log Files
Log Files
Log Retain
Figure 3-7 Online archive logging
Offline archived logs: An archived log is said to be offline when it is no longer found in the database log path directory. When you want to use archive logging (see Figure 3-8) you must make provision for the logs to be stored away from the database. This is done in DB2 UDB by specifying a userexit parameter and interfacing to a suitable archive manager. Full documentation of this is supplied in the DB2 manuals.
54
DB2 server
Transaction
Database Log Path
Active Log Files
Offline Archived
User Exit
Log Files
Figure 3-8 Offline archive logging
Capture logging
Capture logging is used for replication processing. Log files are retained until replication processing has completed, at which time they are deleted automatically.Capture logging can be configured by setting the logretain database configuration parameter to CAPTURE. All DB2 UDB utilities handle this logging mode in the same way as they handle circular logging. Neither online backup operations, nor tablespace backup and restore operations, nor rollforward operations are allowed.
This is unlikely to be used in a warehousing environment that uses DB2 Warehouse Manager.
55
Database configuration parameters for logging

In this section, we will present the database configuration parameters that are related to database logging. The default values of these parameters do not support rollforward recovery, so if you plan to use it, you must change some of the default values.
Figure 3-9 Database configuration parameters related to database logging
As shown in Figure 3-9, you can use the Control Center. Click Configure... database option to display the Logs page.
56
logprimary
The number of primary logs that will be created. A primary log, whether empty or full, requires the same amount of disk space. Thus, if you configure more logs than you need, you use disk space unnecessarily. If you configure too few logs, you can encounter a log-full condition. As you select the number of logs to configure, you must consider the size you make each log and whether your application can handle a log-full condition. If you are enabling an existing database for rollforward recovery, change the number of primary logs to the sum of the number of primary and secondary logs, plus 1. Additional information is logged for LONG VARCHAR and LOB fields in a database enabled for rollforward recovery. The total log file size limit is 32 GB. That is, the number of log files (logprimary + logsecond) multiplied by the size of each log file, in bytes (logfilsiz * 4096) must be less than 32 GB.
logsecond
The number of secondary log files that are created and used for recovery, if needed. If the primary log files become full, secondary log files (of size logfilsiz) are allocated, one at a time as needed, up to the maximum number specified by this parameter. An error is returned, and activity against the database is stopped, if more than the configured number of secondary log files is required.
logfilsiz
The size of each configured log, in number of 4-KB pages. The size of the log file has a direct bearing on performance. If the database is configured to retain logs, each time a log is filled, a request is issued for allocation and initialization of a new log. Increasing the size of the log reduces the number of requests required to allocate and initialize new logs. (Keep in mind, however, that with a larger log size it takes more time to format each new log). logbufsz The amount of database shared memory to use as a buffer for log records before writing these records to disk. The log records are written to disk when any one of the following events occurs: A transaction commits The log buffer becomes full Some other internal database manager event occurs.
57
Increasing the log buffer size results in more efficient input/output (I/O) activity associated with logging, because the log records are written to disk less frequently, and more records are written each time.
newlogpath
The database logs are initially created in SQLOGDIR, which is a subdirectory of the database directory. You can change the location in which active logs and future archive logs are placed by changing the value of this configuration parameter to point to a different directory or to a device. Archive logs that are currently stored in the database log path directory are not moved to the new location if the database is configured for rollforward recovery.
logretain
If logretain is set to RECOVERY, archived logs are kept in the database log path directory, and the database is considered to be recoverable, meaning that rollforward recovery is enabled. If logretain is set to CAPTURE, the replication capture program calls the prune logfile command to delete log files when the capture program completes. You should not set logretain to CAPTURE if you want to perform rollforward recovery on the database.
userexit
With this parameter set to YES, rollforward recovery is enabled causing the database manager to call a user exit program for archiving and retrieving logs. The log files are archived in a location that is different from the active log path. The parameters described above can be updated using the Control Center. The same task can be performed from a Command Window or Command Center by using the DB2 UDB command: UPDATE DATABASE CONFIGURATION FOR db USING parameter Table 3-1 summarizes the relationship between the values of the logretain and userexit parameters, the mode of database logging, the type of backup that you can choose, and the mode of rollforward.
58
Table 3-1 The logretain and userexit parameters

logretain userexit Logging Backup Rollforward Description
No Recovery
No No
Circular Archive
Database offline Database or Tablespace online or offline Database or Tablespace online or offline Database or Tablespace online or offline
No Point-in-time or to end of logs Point-in-time or to end of logs Point-in-time or to end of logs
Default Log files stored in SQLOGDIR Log files in SQLOGDIR erased by userexit Log files in SQLOGDIR erased by userexit
No
Yes
Archive
Recovery
Yes
Archive
Recovery history file

The recovery history file (as shown in Figure 3-10) is updated when any of the following operations is performed: A database or tablespaces are backed up. A database or tablespaces are restored. A database or tablespaces are rolled forward. A tablespace is created. A tablespace is altered. A tablespace is quiesced. A tablespace is renamed. A tablespace is dropped. A table is loaded. A table is dropped. A table is reorganized. Table statistics are updated.
59
Figure 3-10 Creating and updating the Recovery History File
An option on the restore command allows only the recovery history file to be restored. This is useful in recovery situations where the current database is unusable or not available, and the associated recovery history file is damaged or deleted. You can then review the recovery history file to determine which backup to use to restore the database. The recovery history file contains the summarized backup information that you can use to recover all or part of the database to a given point-in-time. The information in the file includes: An identification (ID) field to uniquely identify each entry The part of the database that was copied, and how it was copied The time the copy was made The location of the copy (stating both the device information and the logical way to access the copy) The last time a restore operation was done The time at which a tablespace was renamed, showing the previous and the current name of the tablespace The status of a backup operation: active, inactive, expired, or deleted The last log sequence number saved by the database backup or processed during a rollforward recovery operation. To see the entries in the recovery history file, use the db2 list history command.
60
Here are some examples of the db2 list history command:
db2 db2 db2 db2 db2 db2
list list list list list list
history containing schema1.table1 for db1 backup containing schema1.table1 for db1 history since 20011122 for db1 backup since 20011122 for db1 history all for db1 backup all for db1
To access the recovery history file from the Control Center, select Journal and then the History tab.
Figure 3-11 Recovery History File from DB2 UDB journal
The default retention period for the history file is 366 days, but this may be overridden by using the database configuration parameter rec_his_retentn. The maximum value is 30000.
61
You can also manually remove old entries from the file by using the prune history command. A minimum yyyy may be specified as the timestamp. All entries with timestamps equal to or less than the timestamp provided are deleted from the recovery history file. The WITH FORCE OPTION specifies that the entries will be pruned according to the timestamp provided, even if some entries from the most recent restore set are deleted from the file. Some examples are: db2 prune history 20011122100922 db2 prune history 20011122 with force option
Tablespace state
In a typical recovery situation, it is very important for you to understand the current status of a tablespace indicated by its state. The tablespace states most commonly associated with recovery are:
Rollforward pending: A tablespace is put in this state after it is restored, or following an input/output (I/O) error. After it is restored, the tablespace can be rolled forward to the end of the logs or to a point-in-time. Following an I/O error, the tablespace must be rolled forward to the end of the logs or to a point in time. Rollforward-in-progress: A tablespace is put in this state when a rollforward operation on that tablespace is in progress. Once the rollforward operation completes successfully, the tablespace is no longer in rollforward-inprogress state. The tablespace can also be taken out of this state if the rollforward operation is cancelled. Restore pending: A tablespace is put in this state if a rollforward operation on that tablespace is cancelled, or if a rollforward operation on that tablespace encounters an unrecoverable error, in which case the tablespace must be restored and rolled forward again. Backup pending: A tablespace is put in this state after a point-in-time rollforward operation, or after a load operation with the no copy option. The tablespace must be backed up before it can be used. If it is not backed up, the tablespace cannot be updated, but read only operations are allowed.
3.3.4 Disaster recovery

The term disaster recovery is used to describe the activities that need to be done to restore the database in the event of a fire, earthquake, vandalism, or other catastrophic events. You might want to consider including one or more of the following in your disaster recovery plan:
62
A site to be used in the event of an emergency A different machine on which to recover the database Off-site storage of database backups and archived logs. If your plan for disaster recovery is to recover the entire database on another machine, you need to have at least one full database backup and all the archived logs for the database. This means that you must make provision for ensuring the database backups and the archived logs are moved (either electronically or using removable media) away from the site. This is covered a little further in 3.9, Disaster recovery on page 92.
3.4 Database backup

In this section, we focus our discussion on database backup. The concept of a database backup is the same as any other data backup: taking a copy of the data and storing it on a different medium in case of failure or damage to the original. The simplest case of a backup involves shutting down the database to ensure that no further transactions occur, and then simply backing it up. If your database is damaged or corrupted, you can rebuild it from the backup image. The purpose of this section is help you understand the techniques well enough to design an appropriate backup strategy. Often, a combination of techniques is used to back up target warehouses, warehouse control database and Information Catalog Manager database in DB2 Warehouse Manager.
3.4.1 Using the db2 backup db command

You only need to specify the alias name of the database that you want to backup to execute the simplest form of the db2 backup db command requires. For example: db2 backup db TBC_MD or db2 backup database TBC_MD At the end of successful completion of the command, you will have acquired a new backup image that is located in the path or the directory from which the command was issued. It is located in this directory because the command in this example does not explicitly specify a target location for the backup image. On Windows NT/2000, for example, this command (when issued from the root directory) creates an image that appears in a directory listing, as shown in Example 3-1.
63
Example 3-1 Directory listing

Directory of D:\TBC_MD.0 \DB2 \NODE0000 \CATN0000 \20010320 03/20/2001 12:26p <DIR> 03/20/2001 12:26p <DIR> 03/20/2001 12:27p . .. 122644.001
12,615,680
Backup images are created at the target location that you have the option to specify when you invoke the backup utility. This location can be: A directory (for backups to disk or diskette) A device (for backups to tape) A Tivoli Storage Manager (TSM) server (see 4.3.5, Tivoli Storage Manager on page 138) Another vendors server Specified through a user exit program (OS/2 only). On UNIX based systems, file names for backup images created on disk consist of a concatenation of several elements, separated by periods: DB_alias.Type.Inst_name.NODEnnnn.CATNnnnn.timestamp.Seq_num For example: STAFF.0.DB201.NODE0000.CATN0000.19950922120112.001 On other platforms, a four-level subdirectory tree is used: DB_alias.Type\Inst_name.NODEnnnn\CATNnnnn\yyyymmdd\hhmmss.Seq_num For example, on Windows NT/2000: TBC_MD.0\DB2\NODE0000\CATN0000\20010320\122644.001
DB_alias Type
A 1- to 8-character database alias name that was specified when the backup utility was invoked. Type of backup operation, where: 0 represents a full database-level backup, 3 represents a tablespace-level backup, and 4 represents a backup image generated by the load...copy to command. A 1- to 8-character name of the current instance that is taken from the DB2INSTANCE environment variable. The node number. In non-partitioned database systems, this is always NODE0000.In partitioned database systems, it is NODExxxx, where xxxx is the number assigned to the node in the db2nodes.cfg file.
Inst_name NODEnnnn
64
CATNnnnn
The node number of the catalog node for the database. In non-partitioned database systems, this is always CATN0000. In partitioned database systems, it is CATNxxxx, where xxxx is the number assigned to the node in the db2nodes.cfg file. A 14-character representation of the date and time (timestamp) at which the backup operation was performed. The time stamp is in the form yyyymmddhhnnss, where:
Time stamp
yyyy represents the year (1995 to 9999) mm represents the month (01 to 12) dd represents the day of the month (01 to 31) hh represents the hour (00 to 23) nn represents the minutes (00 to 59) ss represents the seconds (00 to 59)
Seq_num
A 3-digit number for sequence number used as a file extension.
Before using backup, you should not be connected to the database that is to be backed up: the backup utility automatically establishes a connection to the specified database, and this connection is terminated at the completion of the backup operation. The database can be local or remote. The backup image remains on the database server, unless you are using a storage management product such as Tivoli Storage Manager (TSM).
3.4.2 Invoking the db2 backup db command

The db2 backup db command can be invoked through: The DB2 Command Line Processor (CLP). Following is an example of the db2 backup db command issued through the CLP: db2 backup db TBC_MD to c:\DB2Backups The Backup Database notebook or wizard in the Control Center, as shown in Figure 3-12.
65
Figure 3-12 Backup Database notebook in the Control Center
To open the Backup Database notebook or wizard: 1. From the Control Center, expand the object tree until you find the Databases folder. 2. Click the Databases folder. Any existing databases are displayed in the pane on the right side of the window (the contents pane). 3. Right-click the database you want in the contents pane, and select Backup >Database... or Backup> Database Using Wizard from the pop-up menu. The Backup Database notebook or the Backup Database wizard opens.
3.4.3 Displaying backup information

You can use db2ckbkp to display information about existing backup images. This utility allows you to: Test the integrity of a backup image and determine whether or not it can be restored. Display information that is stored in the backup header.
3.4.4 LAN-free backup

Normally, a database backup has to go over the Local Area Network (LAN) to the storage destination and may impact the users or applications using the same LAN.
66
One way to overcome this problem is to use a dedicated LAN for backup so that the data transfer will no longer interfere with the work of other users and applications. However, even when using a dedicated LAN, all the data and metadata (file permissions, owner, and so on) must still be handled by the application that will receive the backup data. This application will need resources such as CPU, memory, or disk space to buffer and manage data and metadata. To free the application from the work needed for handling the backup data itself, a LAN-free solution can be used. LAN-free means that a group of machines are able to share the same storage devices over a high performance connection. A LAN-free solution also provides an easy way of defining storage devices to machines without much cabling effort, as the devices all communicate over the same high performance connection.
3.4.5 Backing up to named pipes

Support is now available for database backup to (and database restore from) local named pipes on UNIX based systems. Both the writer and the reader of the named pipe must be on the same machine. The pipe must exist and be located on a local file system. Because the named pipe is treated as a local device, there is no need to specify that the target is a named pipe. Following is an AIX example: 1. Create a named pipe:
mkfifo /u/dmcinnis/mypipe
2. Use this pipe as the target for a database backup operation:

db2 backup db TBC_MD to /u/dmcinnis/mypipe
3. If this backup image is going to be used by the restore utility, the restore operation must be invoked before the backup operation, so that it will not miss any data:
db2 restore db TBC_MD into mynewdb from /u/dmcinnis/mypipe
67
3.4.6 Optimizing backup performance

To reduce the amount of time required to complete a backup operation: Specify tablespace backup and use incremental backup when possible. You can back up (and subsequently recover) part of a database by using the TABLESPACE option on the BACKUP DATABASE command. This facilitates the management of table data, indexes, and long field or large object (LOB) data in separate tablespaces. Backup performance will be much more optimized by using incremental backups (See, for example, 5.3.4, Incremental backup on page 168). Increase the value of the PARALLELISM parameter. (PARALLELISM n). This parameter is on the backup database command so that it reflects the number of tablespaces being backed up. Using this parameter can dramatically reduce the time required to complete the backup (especially if the backup is going to a disk). This parameter defines the number (n) of processes that are started to read data from the database. Each process is assigned to back up a specific tablespace. When it completes backing up the tablespace, it requests another. Each process will be assigned a tablespace to complete, therefore, to get better performance, let this value be less than the number of tablespaces, since setting up the value higher than the number of tablespaces does not show any significant difference in performance. Note, however, that each process or thread requires both memory and CPU overhead: on a heavily loaded system, keep the PARALLELISM parameter at its default value of 1. If backing up to different targets (for example, using multiple sessions to send the data to Tivoli Storage Manager), parallelism should not be greater than the number of targets (sessions) Increase the backup buffer size (BUFFER buff-size). This value is used as the buffer allocation size in pages (4 KB) when building the backup image. For a buffer size of zero, the value of the database manager configuration parameter BACKBUFSZ will be used as the buffer allocation size. When backing up a database, the data is first copied to an internal buffer. Data is then written from this buffer to the backup media when the buffer is full. Tuning BACKBUFSZ can help improve the performance of the backup utility as well as minimize the impact on the performance of other concurrent database operations.
68
The ideal backup buffer size is a multiple of the tablespace extent size. If you have multiple tablespaces with different extent sizes, specify a value that is a multiple of the largest extent size. Increase the number of buffers (WITH num-buff BUFFERS) If you use multiple buffers and I/O channels, you should use at least twice as many buffers as channels to ensure that the channels do not have to wait for data. The numbers of buffers should be: #sessions + #parallelism +2. Also, the following calculation must fit: (num-buffers * buffer-size) < UTIL_HEAP_SZ (UTIL_HEAP_SZ is the database utility heap size).
3.4.7 Backup restrictions

The following restrictions apply to the backup utility: A tablespace backup operation and a tablespace restore operation cannot be run at the same time, even if different tablespaces are involved. If you want to be able to do rollforward recovery in a partitioned database environment, you must regularly back up the database on the list of nodes, and you must have at least one backup image for of the rest of the nodes in the system (even those that do not contain user data for that database). Three situations require that the backed-up image of a database partition at a database partition server does not contain user data for the database: You added a database partition server to the database system after taking the last backup, and you need to do forward recovery on this database partition server. Point-in-time recovery is used, which requires that all database partitions in the system are in rollforward pending state. For disaster recovery, you must have done a full database backup and kept all the logs even if you are used to run tablespaces backups. Backup strategy should also include periodic full database backups with partial backups for critical tablespaces.
69
3.5 Database restore

This section describes the DB2 restore utility, which is used to rebuild damaged or corrupted DB2 Warehouse Manager databases or tablespaces that were previously backed up.
3.5.1 Using the db2 restore db command

You can run the simplest form of the db2 restore db command by specifying only the alias name of the database that you want to restore. For example: db2 restore db TBC_MD In this example, because the TBC_MD database exists, the message in Example 3-2 is returned.
Example 3-2 Restore error message
SQL2539W Warning! Restoring to an existing database that is the same as the backup image database. The database files will be deleted. Do you want to continue ?(y/n)
If you specify y, and a backup image for the TBC_MD database exists, the restore operation should complete successfully. No applications can be running against the database when the operation starts, and the restore utility prevents other applications from accessing the database until the restore operation completes successfully. A tablespace restore operation, however, can be done online. A tablespace is not usable until the restore operation (followed by rollforward recovery) completes successfully. If you have tables that span more than one tablespace, you should back up and restore the set of tablespaces together. When doing a partial or subset restore operation, you can use either a tablespace-level backup image, or a full database-level backup image and choose one or more tablespaces from that image. You need to make available all the log files associated with these tablespaces from the time that the backup image. You should not be connected to the database that is to be restored when restoring to an existing database. The restore utility automatically establishes a connection to the specified database, and this connection is terminated at the completion of the restore operation.
70
When restoring to a new database, an instance attachment is required to create the database. When restoring to a new remote database, you must first attach to the instance where the new database will reside. Then, create the new database, specifying the code page and the territory of the server. The database can be local or remote.
3.5.2 Invoking the db2 restore db command

The db2 restore db command can be invoked through: The DB2 Command Line Processor (CLP). Following is an example of the db2 restore db command issued through the CLP: db2 restore db TBC_MD from D:\DB2Backups taken at 20010320122644 The Restore Database notebook or wizard in the Control Center. To open the Restore Database notebook or wizard: a. From the Control Center, expand the object tree until you find the Databases folder. b. Click on the Databases folder. Any existing databases are displayed in the pane on the right side of the window (the contents pane). c. Right-click the database you want in the contents pane, and select Restore >Database... or Restore> Database Using Wizard from the pop-up menu. The Restore Database notebook or the Restore Database wizard opens (Figure 3-13).
71
Figure 3-13 Restore Database notebook in the Control Center
An Application Programming Interface (API), sqlrestore.
3.5.3 Tablespace redirected restore

Tablespace redirected restore is used when tablespace containers cannot be restored to their original location. During a database backup operation, a record is kept of all the tablespace containers associated with the tablespaces that are being backed up. During a restore operation, all containers listed in the backup image are checked to determine if they exist and if they are accessible. If one or more of these containers is inaccessible because of media failure (or for any other reason), the restore operation will fail.
72
For a successful restore operation in this case, you need to redirect the data restoration to different containers. DB2 UDB supports adding, changing, or removing tablespace containers. You can redefine tablespace containers by invoking the restore database command and specifying the REDIRECT parameter, or by using the Containers page of the Restore Database notebook in the Control Center (Figure 3-14).
Figure 3-14 Restore Database notebook with Redirect option
Following is a typical redirected restore scenario for a database whose alias is MYDB: 1. Issue a RESTORE DATABASE command with the REDIRECT option. db2 restore db mydb replace existing redirect After successful completion of step 1, and before completing step 3, the restore operation can be aborted by issuing: db2 restore db mydb abort
73
2. Issue a SET TABLESPACE CONTAINERS command for each tablespace whose containers must be redefined. For example: db2 set tablespace containers for 5 using (file 'f:\ts3con1'20000,file 'f:\ts3con2'20000) To verify that the containers of the restored database are the ones specified in this step, issue the LIST TABLESPACE CONTAINERS command. 3. After successful completion of steps 1 and 2, issue: db2 restore db mydb continue This is the final step of the redirected restore operation. 4. If step 3 fails, or if the restore operation has been aborted, the redirected restore can be restarted, beginning at step 1. During a redirected restore operation, directory and file containers are automatically created if they do not already exist. The database manager does not automatically create device containers. Container redirection provides considerable flexibility for managing tablespace containers. For example, even though adding containers to SMS tablespaces is not supported, you could accomplish this by specifying an additional container when invoking a redirected restore operation. Similarly, you could move a DMS tablespace from file containers to device containers.
3.5.4 Restoring to an existing database

You can restore a full database backup image to an existing database. The backup image may differ from the existing database in its alias name, its database name, or its database seed. A database seed is a unique identifier for a database that does not change during the life of the database. The seed is assigned by the database manager when the database is created. It remains unchanged following a restore operation, even if the backup image has a different database seed. DB2 UDB always uses the seed from the backup image. When restoring to an existing database, the restore utility: Deletes table, index, and long field data from the existing database, and replaces it with data from the backup image. Replaces table entries for each tablespace being restored. Retains the recovery history file, unless it is damaged. If the recovery history file is damaged, the database manager copies the file from the backup image. Retains the authentication type for the existing database.
74
Retains the database directories for the existing database. The directories define where the database resides, and how it is cataloged. Compares the database seeds. If the seeds are different, the restore utility: Deletes the logs associated with the existing database. Copies the database configuration file from the backup image. Sets NEWLOGPATH to the value of the logpath database configuration parameter if NEWLOGPATH was specified on the restore database command. If the database seeds are the same, the restore utility: Deletes the logs if the image is for a non-recoverable database. Retains the current database configuration file, unless the file has been corrupted, in which case the file is copied from the backup image. Sets NEWLOGPATH to the value of the logpath database configuration parameter if NEWLOGPATH was specified on the RESTORE DATABASE command; otherwise, copies the current log path to the database configuration file. Validates the log path: If the path cannot be used by the database, changes the database configuration to use the default log path.
3.5.5 Restoring to a new database

You can create a new database and then restore a full database backup image to it. The code pages of the backup image and the target database must match. When restoring to a new database, the restore utility: Creates a new database, using the database alias name that was specified through the target database alias parameter. (If a target database alias was not specified, the restore utility creates the database with an alias that is the same as that specified through the source database alias parameter.) Restores the database configuration file from the backup image. Sets NEWLOGPATH to the value of the logpath database configuration parameter if NEWLOGPATH was specified on the restore database command. Validates the log path: If the path cannot be used by the database, changes the database configuration to use the default log path. Restores the authentication type from the backup image. Restores the comments from the database directories in the backup image. Restores the recovery history file for the database.
75
3.5.6 Optimizing restore performance

To reduce the amount of time required to complete a restore operation: Increase the number of TSM I/O sessions. (OPEN n SESSIONS) This parameter is only valid if the use tsm option is used on the DB2 restore command. If specified, the restore process will open n I/O sessions to the Tivoli Storage Manager server to receive the data. This number should also not be higher than the number of sessions used for backup. If there are not enough tape drives available at the Tivoli Storage Manager server at the time of restore, some of the sessions have to wait until there are free tape mounts available again. Increase the number of restore buffers. (WITH num-buff BUFFERS) The numbers of buffers should be: #sessions + #parallelism +2. Also, the following calculation must fit: (num-buffers * buffer-size) < UTIL_HEAP_SZ (UTIL_HEAP_SZ is the database utility heap size). Increase the size of restore buffers. (BUFFER buff-size) We recommend setting the restore buffer size to a multiple of the extent size. The value given must be equal to, or a multiple of, the backup buffer size that was specified when the backup image was taken. Otherwise, the smallest acceptable size of the buffer (4KB) is used for the restore buffer. If you do not specify the number of pages, each buffer will be allocated based on the database manager configuration parameter RESTBUFSZ. Increase the number of parallelism. (PARALLELISM) This specifies the number of buffer manipulators to be spawned during the restore process. The default value is 1. For restore, the number can be increased, but not higher than the number of Tivoli Storage Manager client sessions that have been started for the restore. The number of buffers should be at least the value selected for parallelism. In the next example, we assume that the backup was made using two Tivoli Storage Manager sessions. For restore, it is also reasonable to use two Tivoli Storage Manager sessions and a parallelism of 2 (no matter how many tablespaces we have): db2 restore db TBC_MD use tsm open 2 sessions with 6 buffers parallelism 2
76
3.5.7 Restore restrictions

The following restrictions apply to the restore utility: You can only use the restore utility if the database has been previously backed up using the DB2 backup utility. If you are using the DB2 UDB Control Center, you cannot restore backup images that were created with previous versions of DB2 UDB. A database restore operation cannot be started while the rollforward process is running. You can restore a tablespace only if the tablespace currently exists, and if it is the same tablespace; same means that the tablespace was not dropped and then recreated between the backup and the restore operation.) You cannot restore a tablespace-level backup to a new database. You cannot perform an online tablespace-level restore operation involving the system catalog tables. On OS/2, a partial or subset restore operation is not possible when calling a user exit program. If the code page of the database being restored does not match a code page that is available to the application, or if the database manager does not support code page conversion from the database code page to a code page that is available to the application, the restored database will not be usable.
3.6 Rollforward recovery

This section describes the DB2 rollforward utility, which is used to recover a database by applying transactions that were recorded in the database recovery log files.
3.6.1 Using the db2 rollforward db command

The simplest form of the db2 rollforward db command requires only that you specify the alias name of the database that you want to rollforward recover. For example: db2 rollforward db TBC_MD stop The command returns the results shown in Example 3-3.
77
Example 3-3 Rollforward command results

Rollforward Status Input database alias Number of nodes have returned status Node number Rollforward status Next log file to be read Log files processed Last committed transaction = = = = = = = TBC_MD 1 0 not pending 2001-03-11-02.39.48.000000
DB20000I The ROLLFORWARD command completed successfully.
The general approach to rollforward recovery involves: 1. Invoking the rollforward utility without the STOP or COMPLETE option 2. Invoking the rollforward utility with the QUERY STATUS option If you specify recovery to the end of the logs, the QUERY STATUS option can indicate that one or more log files is missing, if the returned point in time is earlier than you expect. If you specify point-in-time recovery, the QUERY STATUS option will help you to ensure that the rollforward operation has completed at the correct point. 3. Invoking the rollforward utility with the STOP option. After the operation stops, it is not possible to roll additional changes forward. A database must be restored successfully (using the restore utility) before it can be rolled forward, but a tablespace does not. A tablespace may be temporarily put in rollforward pending state, but not require a restore operation to undo it (following a power interruption, for example). When the rollforward utility is invoked: If the database is in rollforward pending state, the database is rolled forward. If tablespaces are also in rollforward pending state, you must invoke the rollforward utility again after the database rollforward operation completes to roll the tablespaces forward. If the database is not in rollforward pending state, but tablespaces in the database are in rollforward pending state: If you specify a list of tablespaces, only those tablespaces are rolled forward. If you do not specify a list of tablespaces, all tablespaces that are in rollforward pending state are rolled forward.
78
A database rollforward operation runs offline. The database is not available for use until the rollforward operation completes successfully, and the operation cannot complete unless the STOP option was specified when the utility was invoked. A tablespace rollforward operation can run offline. The database is not available for use until the rollforward operation completes successfully. This occurs if the end of the logs is reached, or if the STOP option was specified when the utility was invoked. You can perform an online rollforward operation on tablespaces, as long as SYSCATSPACE is not included. When you perform an online rollforward operation on a tablespace, the tablespace is not available for use, but the other tablespaces in the database are available. When you first create a database, it is enabled for circular logging only. This means that logs are reused, rather than being saved or archived. With circular logging, rollforward recovery is not possible: only crash recovery or version recovery can be done. Archived logs document changes to a database that occur after a backup was taken. You enable log archiving (and rollforward recovery) by setting the logretain database configuration parameter to RECOVERY, or setting the userexit database configuration parameter to YES, or both. The default value for both of these parameters is NO, because initially, there is no backup image that you can use to recover the database. When you change the value of one or both of these parameters, the database is put into backup pending state, and you must take an offline backup of the database before it can be used again. The rollforward utility automatically establishes a connection to the specified database, and this connection is terminated at the completion of the rollforward operation. The database can be local or remote.
3.6.2 Invoking the db2 rollforward db command

The db2 rollforward db command can be invoked through: The DB2 Command Line Processor (CLP). Following is an example of the db2 rollforward db command issued through the CLP: db2 rollforward db TBC_MD to end of logs and stop
79
The Rollforward Database notebook in the Control Center. To open the Rollforward Database notebook: a. From the Control Center, expand the object tree until you find the Databases folder. b. Click on the Databases folder. Any existing databases are displayed in the pane on the right side of the window (the contents pane). c. Right-click the database you want in the contents pane, and select Roll-forward from the pop-up menu. The Rollforward Database notebook opens.
3.6.3 Rollforward cases

We provide three different rollforward cases.
Case 1:
The rollforward database command permits specification of multiple operations at once, each being separated with the keyword AND. For example, to roll forward to the end of logs, and complete, the separate commands: db2 rollforward db TBC_MD to end of logs db2 rollforward db TBC_MD complete can be combined as follows: db2 rollforward db TBC_MD to end of logs and complete Although the two are equivalent, it is recommended that such operations be done in two steps. It is important to verify that the rollforward operation has progressed as expected, before stopping it and possibly missing logs. This is especially important if a bad log is found during rollforward recovery, and the bad log is interpreted to mean the end of logs. In such cases, an undamaged backup copy of that log could be used to continue the rollforward operation through more logs.
Case 2:
Roll forward to the end of the logs (two tablespaces have been restored): db2 rollforward db TBC_MD to end of logs db2 rollforward db TBC_MD to end of logs and stop These two statements are equivalent. Neither AND STOP or AND COMPLETE is needed for tablespace rollforward recovery to the end of the logs. tablespace names are not required. If not specified, all tablespaces requiring rollforward recovery will be included. If only a subset of these tablespaces is to be rolled forward, their names must be specified.
80
Case 3:
After three tablespaces have been restored, roll one forward to the end of the logs, and the other two to a point-in-time, both to be done online: db2 rollforward db TBC_MD to end of logs tablespace(TBS1)online db2 rollforward db TBC_MD to 1998-04-03-14.21.56.245378 and stop tablespace(TBS2,TBS3) online Note that two rollforward operations cannot be run concurrently. The second command can only be invoked after the first rollforward operation completes successfully.
3.6.4 Tablespace rollforward recovery

In order to use tablespace-level backup and restore, archive logging must be turned on. Once the database is enabled for forward recovery, you have the option of backing up, restoring, and rolling forward tablespaces instead of the entire database. You may want to implement a recovery strategy for individual tablespaces because this can save time: it takes less time to recover a portion of the database than it does to recover the entire database. For example, if a disk is bad, and it contains only one tablespace, that tablespace can be restored and rolled forward without having to recover the entire database, and without impacting user access to the rest of the database, unless the damaged tablespace contains the system catalog tables. If the catalog tables are involved in a recovery situation, access to the entire database is impacted. Therefore, it is a good practice to maintain tablespace-level backups of your catalog tables. The duration of the recovery process will be reduced significantly. Tablespace-level backups also allow you to back up critical parts of the database more frequently than other parts, and requires less time than backing up the entire database. After a tablespace is restored, it is always in rollforward pending state. To make the tablespace usable, you must perform rollforward recovery on it. In most cases, you have the option of rolling forward to the end of the logs, or rolling forward to a point-in-time.
81
You cannot, however, roll tablespaces containing system catalog tables forward to a point-in-time. These tablespaces must be rolled forward to the end of the logs to ensure that all tablespaces in the database remain consistent. Before rolling a tablespace forward, invoke the list tablespaces show detail command. This command returns the minimum recovery time, which is the earliest point-in-time to which the tablespace can be rolled forward. The minimum recovery time is updated when data definition language (DDL) statements are run against the tablespace, or against tables in the tablespace. The tablespace must be rolled forward to at least the minimum recovery time, so that it becomes synchronized with the information in the system catalog tables. If recovering more than one tablespace, the tablespaces must be rolled forward to at least the highest minimum recovery time of all the tablespaces being recovered. If the data and the long objects in a table are in separate tablespaces, and the table has been reorganized, the tablespaces for both the data and the long objects must be restored and rolled forward together. You should take a backup of the affected tablespaces after the table is reorganized.
3.6.5 Point-in-time recovery

Point-in-time recovery allows the user to return to a prior state in the database. Tablespace point-in-time recovery adds granularity by further allowing specific tablespaces to be named. A tablespace point-in-time recovery requires the following steps: 1. Restore from a previous full database backup. 2. Roll forward the required tablespaces to a given point-in-time known as Coordinated Universal Time (CUT) 3. Perform a backup specifying only the tablespaces that were involved in the rollforward. This step is mandatory. The ability to perform backup, restore and rollforward on tablespaces gives you additional flexibility. If you are rolling tablespaces forward to a point in time, and a table is contained in multiple tablespaces, all of these tablespaces must be rolled forward simultaneously. If, for example, the table data is contained in one tablespace, and the index for the table is contained in another tablespace, you must roll both tablespaces forward simultaneously to the same point in time.
82
If you want to roll a tablespace forward to a point in time, and a table in the tablespace participates in a referential integrity relationship with another table that is contained in another tablespace, you should roll both tablespaces forward simultaneously to the same point in time. If you do not, both tablespaces will be in check pending state at the end of the point-in-time rollforward operation. If you roll both tablespaces forward simultaneously, the constraint will remain active at the end of the point-in-time rollforward operation.
3.6.6 Quiesced state

You must ensure that a point-in-time tablespace rollforward operation does not cause a transaction to be rolled back in some tablespaces, and committed in others. This can happen if: A point-in-time rollforward operation is performed on a subset of the tablespaces that were updated by a transaction, and that point in time precedes the time at which the transaction was committed. Any table contained in the tablespace being rolled forward to a point in time has an associated trigger, or is updated by a trigger that affects tablespaces other than the one that is being rolled forward. The solution is to find a suitable point in time that will prevent this from happening. You can issue the quiesce tablespaces for table command to create a transaction-consistent point in time for rolling tablespaces forward. Quiesce means to end a process by allowing operations to complete normally, but rejecting any new requests for work. The quiesce request (in share, intent to update, or exclusive mode) waits (through locking) for all running transactions against those tablespaces to complete, and blocks new requests. Thus, when the quiesce request is granted, the tablespaces are in a consistent state. To determine a suitable time to stop the rollforward operation, you can look in the recovery history file to find quiesce points, and check whether they occur after the minimum recovery time. Once the tablespace quiesce is completed, the tablespace remains in quiesced state. The quiesced state for a table must be reset explicitly by issuing QUIESCE mode RESET. The quiesced state cannot be reset if the tablespace is in load pending.
83
Figure 3-15 shows how to set a quiesce point in the Control Center. To set a quiesce point for a table, right-click the table you want in the contents pane, and select Quiesce from the pop-up menu. The Quiesce window opens.
Figure 3-15 How to set a quiesce point for a table
84
In Figure 3-16, you can see that the tablespace USERSPACE1 is in quiesced state.
Figure 3-16 Information about Quiesced state
After a tablespace point-in-time rollforward operation completes, the tablespace is put in backup pending state. You must take a backup of the tablespace, because all updates made to it between the point in time to which you rolled forward and the current time have been removed. You can no longer roll the tablespace forward to the current time from a previous database-level or tablespace-level backup image.
3.6.7 Recovering a dropped table

You may occasionally drop a table whose data you still need. If this is the case, you should consider making your critical tables recoverable following a drop table operation. You could recover the table data by invoking a database restore operation, followed by a database rollforward operation to a point in time before the table was dropped. This may be time consuming if the database is large, and your data will be unavailable during recovery.
85
DB2s dropped table recovery feature lets you recover your dropped table data using tablespace-level restore and rollforward operations. This will be faster than database-level recovery, and your database will remain available to users. For a dropped table to be recoverable, the tablespace in which the table resides must have the DROPPED TABLE RECOVERY option turned on. This can be done during tablespace creation, or by invoking the ALTER TABLESPACE statement (see the SQL Reference). The DROPPED TABLE RECOVERY option is tablespace-specific and limited to regular tablespaces. To determine if a tablespace is enabled for dropped table recovery, you can query the DROP_RECOVERY column in the SYSCAT.TABLESPACES catalog table. When a DROP TABLE statement is run against a table whose tablespace is enabled for dropped table recovery, an additional entry (identifying the dropped table) is made in the log files. An entry is also made in the recovery history file, containing information that can be used to recreate the table. Only one dropped table can be recovered at a time.
3.6.8 Rollforward restrictions

The following restrictions apply to the rollforward utility: You can only invoke one rollforward operation at a time. If there are many tablespaces to recover, you can specify all of them in the same operation. If you have renamed a tablespace following the most recent backup operation, ensure that you use the new name when rolling the tablespace forward. The previous tablespace name will not be recognized. You cannot cancel a rollforward operation that is running. You can only cancel a rollforward operation that has completed, but for which the STOP option has not been specified, or a rollforward operation that has failed before completing. You cannot continue a tablespace rollforward operation to a point in time, specifying a time stamp that is less than the previous one. If a point in time is not specified, the previous one is used. You can initiate a rollforward operation to a point in time by just specifying STOP, but this is only allowed if the tablespaces involved were all restored from the same offline backup image. In this case, no log processing is required.
86
If you start another rollforward operation with a different tablespace list before the in-progress rollforward operation is either completed or cancelled, an error message (SQL4908) is returned. Invoke the LIST TABLESPACES command on all nodes to determine which tablespaces are currently being rolled forward (rollforward in progress state), and which tablespaces are ready to be rolled forward (rollforward pending state). You have three options: Finish the in-progress rollforward operation on all tablespaces. Finish the in-progress rollforward operation on a subset of tablespaces. (This may not be possible if the rollforward operation is to continue to a specific point in time, which requires the participation of all nodes.) Cancel the in-progress rollforward operation.
3.7 Backup and restore hints for partitioned databases

Partitioned database backups are performed partition by partition. The db2 backup database command already discussed will only backup the partition to which you are attached when issued. The catalog partition must be backed up separately but once this is completed, all remaining partitions may be backed up independently and in parallel. This is most easily achieved using the db2_all command in the following steps (example assumes the catalog partition is on partition 1): Issue the backup command to the catalog partition only: db2_all "<<+1< db2 backup db test to ... When complete, issue the backup command to all partitions except the catalog in parallel (see || in the command below): db2_all "||<<-1< db2 backup db test to ... A partition backup is valid only if it is synchronized with the catalog partition. In the case of an offline backup this means that no database activity (connections) should be allowed from the start of the catalog backup to the end of the last partition backup. If there is a connection then restore of that partitions will fail.
Note: There in no error reported during the backup, and no tool to check the validity of each partition backup, but the restore fails with an SQL1471N error.
87
Restore is similarly performed on a partition by partition basis, with the catalog partition requiring to be restored first (then, as with the backup, the remaining partitions can be restored in parallel).
The db2 rollforward database command must be issued at the catalog partition. If a partition is damaged and only that partition needs to be restored, the process is as follows: (this is an example for partition 3 in the UNIX environment): 1. Access damaged partition (export DB2NODE=3) 2. Capture database configuration 3. Issue the command db2 drop database test at node 3 4. Issue the command: db2 create database test at node 3 5. Update database configuration with correct values 6. The database partition is placed in restore pending state 7. Issue the command: db2 restore database test
3.8 Useful data movement tools

This section focuses on some useful DB2 UDB tools for moving and loading data. Although these utilities can be used for backup and restore operations, they are really designed to move data, for example, for workload balancing or migration. You may find these DB2 UDB commands quite useful especially when you have to move tables from one operating system to another.
88
3.8.1 The db2move tool

The db2move tool is helpful to move large numbers of tables between DB2 UDB databases located on workstations. The tool queries the system catalog tables for a particular database and compiles a list of all user tables. It then exports these tables in PC/IXF format. The PC/IXF files can be imported or loaded to another local DB2 UDB database on the same system, or can be transferred to another workstation platform and imported or loaded to a DB2 UDB database on that platform. The tool calls the DB2 UDB export, import, and load APIs, depending on the action requested by the user. Therefore, the requesting user ID must have the correct authorization required by those APIs, or the request will fail. Also db2move inherits the limitations and restrictions of the APIs. For example: db2move TBC_MD export This will export all tables in the TBC_MD database; default values are used for all options. db2move TBC_MD export -tc userid1,us*rid2 -tn tbname1,*tbname2 This will export all tables created by userid1 or user IDs LIKE us%rid2, and with the name tbname1 or table names LIKE %tbname2. db2move TBC_MD import -l D:\LOBPATH1,C:\LOBPATH2 This example is applicable to OS/2 or the Windows operating system only. The command will import all tables in the TBC_MD database; LOB paths D:\LOBPATH1 and C:\LOBPATH2 are to be searched for LOB files. db2move TBC_MD load -l /home/userid/lobpath,/tmp This example is applicable to UNIX based systems only. The command will load all tables in the TBC_MD database; both the /home/userid/lobpath subdirectory and the tmp subdirectory are to be searched for LOB files. db2move TBC_MD import -io replace -u userid -p password This will import all tables in the TBC_MD database in REPLACE mode; the specified user ID and password will be used.
89
Usage Notes: The db2move tool:

Exports, imports, or loads user-created tables. If a database is to be duplicated from one operating system to another operating system, db2move only helps you to move the tables. You also need to move all other objects associated with the tables, such as: aliases, views, triggers, user-defined functions, and so on. db2look is another DB2 UDB tool to help you easily move some of these objects by extracting the Data Definition Language (DDL) statements from the database. When export, import, or load APIs are called by db2move, the FileTypeMod parameter is set to lobsinfile. That is, LOB data is kept in separate files from PC/IXF files. There are 26 000 file names available for LOB files. The lLOAD action must be run locally on the machine where the database and the data file reside. When the load API is called by db2move, the CopyTargetList parameter is set to NULL; that is, no copying is done. If logretain is on, the load operation cannot be rolled forward later. The tablespace where the loaded tables reside is placed in backup pending state, and is not accessible. A full database backup, or a tablespace backup, is required to take the tablespace out of backup pending state.
3.8.2 The db2look tool

The db2look tool is a statistics and DDL extraction tool provided by DB2 UDB to help you extracts the required DDL statements to reproduce the database objects of a production database on a test database. This tool can also generate the required UPDATE statements used to replicate the statistics on the objects in a test database, as well as the update database configuration and update database manager configuration parameters and the db2set statements so that the registry variables and configuration parameter settings on the test database match those of the production database. It is often advantageous to have a test system contain a subset of the production systems data. However, access plans selected for such a test system are not necessarily the same as those that would be selected for the production system. Both the catalog statistics and the configuration parameters for the test system must be updated to match those of the production system. Using this tool, you can easily create a test database where access plans are similar to those that would be used on the production system. You need a SELECT privilege on the system catalogs to run this tool.
90
For example: Generate the DDL statements for objects created by user walid in database DEPARTMENT. The db2look output is sent to file db2look.sql: db2look -d department -u walid -e -o db2look.sql Generate the UPDATE statements to replicate the statistics for the tables and indexes created by user walid in database DEPARTMENT. The output is sent to file db2look.sql: db2look -d department -u walid -m -o db2look.sql Generate both the DDL statements for the objects created by user walid and the UPDATE statements to replicate the statistics on the tables and indexes created by the same user. The db2look output is sent to file db2look.sql: db2look -d department -u walid -e -m -o db2look.sql Generate the DDL statements for objects created by all users in the database DEPARTMENT. The db2look output is sent to file db2look.sql: db2look -d department -a -e -o db2look.sql Generate the DDL statements for all user defined nodegroups, buffer pools and tablespaces. The db2look output is sent to file db2look.sql: db2look -d department -l -o db2look.sql Generate the UPDATE statements for the database and database manager configuration parameters, as well as the db2set statements for the registry variables in database DEPARTMENT. The db2look output is sent to file db2look.sql: db2look -d department -f -o db2look.sql Generate the DDL for all objects in database DEPARTMENT, the UPDATE statements to replicate the statistics on all tables and indexes in database DEPARTMENT, the GRANT authorization statements, the UPDATE statements for the database and database manager configuration parameters, the db2set statements for the registry variables, and the DDL for all user defined nodegroups, buffer pools and tablespaces in database DEPARTMENT. The output is sent to file db2look.sql: db2look -d department -a -e -m -l -x -f -o db2look.sql Generate the DDL statements for objects created by all users in the database DEPARTMENT. The db2look output is sent to file db2look.sql: db2look -d department -a -e -td %-o db2look.sql db2 -td%-f db2look.sql
91
3.8.3 Export and import utilities

DB2 UDB provides export and import utilities. These utilities operate on logical objects as opposed to physical objects. For example, you can use an export command to copy an individual table to a file system file. At some later time, you might want to restore the table, in which case you would use the import command. Although export and import can be used for backup and restore operations, they are really designed for moving data, for example, for workload balancing or migration.
3.9 Disaster recovery

A site can fail for a number of reasons listed below and more: Natural disasters Floods Hurricanes Earthquakes Infrastructure disasters Power failures Fires Water main breaks Operational disasters Viruses Human errors Sabotages When a site fails, it can remain non-operational for any length of time. With proper planning, you can resume the operation elsewhere. While it is possible to plan for and to minimize the potential damage resulting from any type of failure, it is not possible to cover all eventualities. In some warehouses which obtain all their input by batch processing they can choose to keep all their input data instead. Then when a recovery needs to be made they recover the backup and reprocess all the input files in order to get back to the latest state. However, while this saves having to handle logs, they would still have to transport all their input files off-site if they wanted to recover properly from a disaster situation.
92
In other simpler situations, if the source operational system is guaranteed to keep the old source data for longer than the interval between warehouse backups then a warehouse backup will be sufficient because the disaster recovery could recover the warehouse backup and then retrieve the originally processed source data. However, this might not be a trivial exercise. Although a site cannot be made 100% failure proof, the data in the database can be saved and operations using the same data can be resumed at another site, using specific disaster recovery scenario as: Transport of database backup Transport of database backup and archived logs Standby database via log shipping Standby database via replication Synchronous remote mirroring of all data and log files
3.9.1 Transport of database backup

Taking regular backups and transporting them off-site is the minimum level of protection from disaster. This might be the least expensive solution to implement, but suffers from the fact that transactions since the last backup may be lost. This is typically not acceptable unless transactions can be recovered in some reasonable way later without impacting the business. For example, the transactions since the last backup could be recorded on paper and keyed again after the last backup image is restored at another site.
3.9.2 Transport of database backup and archived logs

You can enable the DB2 UDB archive logging. It allows DB2 UDB logs to be transported to a remote site as soon as the log file is closed. A userexit may copy the file to a remote disaster recovery site over a network or to a physical medium such as a tape which can be shipped later. This mechanism is fairly inexpensive and will restore the database to a later point-in-time than just the last backup image.
3.9.3 Standby database via log shipping

The log shipping techniques is also known as log forwarding. The standby database is continuously rolling forward through the log files produced and copied by the production machine. When the production machine fails, a failover occurs: The remaining logs are transferred over to the standby machine. The standby database rolls forward TO END OF LOGS AND STOP. The clients reconnect to the standby database and resume operations.
93
The standby machine has its own resources such as disks, but must have the same physical and logical definitions as the production database. This technique is probably the most widely used disaster recovery method among DB2 UDB customers. The operations that are not logged such as NOT LOGGED INITIALLY transactions and CREATE INDEX, will not be replayed on the standby database. As a result, you want to re-synchronize the standby database after such operations. This could be best accomplished through the new Suspended I/O combined with split mirror. Figure 3-17 illustrates a standby database scenario using the Log Shipping technique.
Production Database
Standby Database
db2 ROLLFORWARD DATABASE <dbname>
split
LOGS
Figure 3-17 Standby database via log shipping
3.9.4 Standby database via replication

You can use replication to implement a standby database. This option has the following advantages: The standby system need not have the same physical definitions. This solution supports heterogeneous environments. The whole database does not need to be replicated. You can choose to replicate only the tables critical to your business. You can access the standby database in read only mode. The standby database can be concurrently accessed.
94
Figure 3-18 shows how the data propagator is used to capture logs.
Production Database
Standby Database
split
Log Capture SQL Apply
Figure 3-18 Standby database via replication
This method requires additional system resources such as disk and CPU. When replication is present, the system must be sized to accommodate it. This mechanism has the disadvange that the transactions may be lost when the production system crashes. As a result, this configuration is not as popular for disaster recovery protection as the log shipping technique.
3.9.5 Synchronous remote mirroring of all data and log files

This configuration is mirroring all the data and logs to a remote recovery site via storage subsystem over the network. A high speed network and a storage subsystem feature such as IBM ESS PPRC or IBM Highly Available Geographic Cluster for AIX (HAGEO), capable of efficient mirroring, are required to minimize the performance impact. See Figure 3-19.
PRODUCTION
STANDBY
DATA and LOGs
Synchronous Mirroring
DATA and LOGs MIRROR
Figure 3-19 Remote mirroring for disaster recovery
95
For example, the PPRC, one of the advanced copy services provided by IBM ESS, can be used to mirror the data and log files over the high speed network to the remote disaster recovery site. For better performance, you can avoid the extra I/O delay by turning on group commit using mincommit database configuration parameter, This can result in response time tradeoff, but generally improves overall throughput by utilizing log bandwidth more efficiently.Also you want to make page cleaners more aggressive to account for extra data I/O lag. Synchronous disk mirroring provides the highest degree of protection, but also at the highest cost. Table 3-2 summarizes the disaster recovery mechanisms with a variety of tradeoffs.
Table 3-2 Disaster recovery methods
Method
Transport of database backups
Pros
Low cost Recovery to time of last backup Recovery to time of last archived log Low cost
Cons
Lost all transactions since last backup
Transport of database backups and archived logs
Loses all transactions in the active logs Longer recovery time (database restore followed by rollforward through all the logs) Standby not available for use while in rollfoward mode Standby needs to be physically and logically identical Transaction loss a possibility Extra cycles on production database for transaction capture Some administrative actions not reflected on standby (e.g. NOT LOGGED operations)
Standby Database via Log Shipping
Support for no transaction loss Minimal impact to productions system Can read (and write) standby Standby need not be physically and logically identical Can choose to replicate only critical tables No transaction loss All changes to the database (including administrative) are replicated Shortest restart time
Standby Database via Replication
Synchronous mirroring of all data and log disks
Performance impact of synchronous mirroring to a geographically remote site High price (software/hardware/network)
96
Chapter 4.
High Availability techniques

In the previous chapter we showed how you could use various degrees of protection to safeguard against loss. Each situation has to decide what level of backup is appropriate to their needs and Chapter 3, Database backup and recovery on page 39 described how provision for recovery is essential. This chapter provides an overview of the techniques which can achieve high availability of your data by providing additional hardware and software which will improve the length of time it takes to recover from an error situation. This we refer to as High Availability. High Availability is of vital importance when running business intelligence applications. Single points of failure within a system can seriously impact both performance and availability. In this chapter we introduce High Availability techniques that you can apply for near-continuous availability of crucial processing facilities for the successful business intelligence applications. We also examine some of the identified single points of failure, and provide solutions for correcting these problems. Examples of recommended solutions that you can implement on your system are provided, as well as TBC_MD configuration scenarios. The following topics are discussed: High Availability overview High Availability and failover support based on AIX and Windows environments High Availability of media and split mirror feature
97
4.1 Overview
High Availability is the term used to describe systems that run and are available to customers more or less all the time. High Availability means different things to different people.
In some cases, High Availability means provision of more or less instantaneous cutover facilities and continuing on standby hardware/software as if the original failure had never occurred. In other cases High Availability means possibly up to 30 minutes outage and continuing as before, which can be handled by provision of standby machines that can take over the work of the failing machine after some handover process of both hardware and software. This has the important attribute of experiencing no degradation in performance. In many cases, High Availability in a warehousing environment means that users could put up with a small loss of availability and also accept a degraded service until things are corrected in full. This can be achieved by making provision for what is called mutual takeover, whereby a machine with existing work on it can take on the extra work of all or part of the failing machine until such time as things are repaired in full. For High Availability to occur: Transactions must be processed efficiently, without appreciable performance degradations (or even loss of availability) during peak operating periods. Systems must be able to recover quickly when hardware or software failures occur, or when disaster strikes. For example, you may want to use of a journaled file system, which is a file system that automatically logs changes to its structure. This ensures that any changes to the file system structure are safe from system crashes, assuming that the storage medium is not lost or corrupted. After a system crash, the changes in these logs are applied to the file system, in the order that they were initially logged. You can reduce recovery times by using a journaled file system, especially when your environment has large file systems, or when data integrity is particularly important. The ability to recover quickly depends critically on having a proven backup and recovery strategy in place. Software that powers the enterprise databases must be continuously running and available for transaction processing. To keep the database manager running, you must ensure that another database manager can take over if it fails. This is called failover capability and allows for the automatic transfer of workload from one system to another when there is hardware failure.
98
4.2 Clustering and failover support

You can achieve failover protection by keeping a copy of your database on another machine that is perpetually rolling the log files forward. Log shipping is the process of copying whole log files to a standby machine, either from an archive device, or through a user exit program running against the primary database. With this approach, the primary database is restored to the standby machine, using either the DB2 UDB restore utility or the split mirror function. You can use the new db2inidb function to quickly initialize the new database. The secondary database on the standby machine continuously rolls the log files forward. If the primary database fails, any remaining log files are copied over to the standby machine. After a rollforward to the end of the logs and stop operation, all clients are reconnected to the secondary database on the standby machine. You can get failover support by adding platform-specific software to your system; for example: High Availability Cluster Multi-Processing, Enhanced Scalability (HACMP), for AIX. Microsoft Cluster Server (MSCS), for Windows NT or Windows 2000. Sun Cluster, or VERITAS Cluster Server, for the Solaris Operating Environment. Multi-Computer/ServiceGuard, for Hewlett-Packard. Steeleye or Mission Critical Linux for Linux
Chapter 4. High Availability techniques
99
Failover strategies are usually based on clusters of systems. A cluster is a group of connected systems that work together as a single system. Each processor is known as a node within the cluster. Clustering allows servers to back each other up when failures occur, by picking up the workload of the failed server (Figure 4-1).
Clients
Public network Private Interconnect

(heartbeat monitor)
SCSI bus
Local Disk Local Disk
Node A
Disk 'A' Disk 'B'
Node B
Figure 4-1 Clusters and DB2
IP address takeover (or IP takeover) is the ability to transfer a server IP address from one machine to another when a server goes down; to a client application, the two machines appear at different times to be the same server. Failover software may use heartbeat monitoring or keepalive packets between systems to confirm availability. Heartbeat monitoring involves system services that maintain constant communication between all the nodes in a cluster. If a heartbeat is not detected, failover to a backup system starts. End users are usually not aware that a system has failed.
100
4.2.1 Cluster configurations

In this section, we introduce the two most common failover strategies on the market known as idle standby and mutual takeover, although the configurations associated with these terms may also be associated with different terms that depend on the vendor (Figure 4-2).
Idle Fail-over to Node 1
Active Active Active
Active Active Active
Active
Idle Standby
Active Fail-over to Node 1 Active Active Active
Mutual Takeover
Active
Active
Active
Active
Active Standby
Figure 4-2 High Availability clustering configurations
Others
101
Idle standby
In this configuration, one system is used to run a DB2 UDB instance, and the second system is idle, or standby mode, ready to take over the instance if there is an operating system or hardware failure involving the first system. Overall system performance is not impacted, because the standby system is idle until needed. Figure 4-3 is an idle standby example. The cluster includes all four nodes and one node is kept idle, standing by to take over the workload of any node that fails. Although this configuration is more costly, it does have the advantage that there is no performance degradation after failover of one machine at a time.
Parallel Database Network
Physical Machines
Logical Partitions Storage Interconnect Disks
Figure 4-3 Idle standby
102
Mutual takeover
In this configuration, each system is the designated backup for another system. Overall system performance may be impacted, because the backup system must do extra work following a failover: it must do its own work plus the work that was being done by the failed system. Figure 4-4 shows an example of a partitioned database consisting of four database partitions, each running on a separate server. The server hardware consists of four nodes connected with a high-speed network. Additional cluster software such as HACMP on AIX or MSCS, also known as Wolfpack, on Windows is installed. The four nodes are separated into two pairs. Each pair is defined as cluster to the cluster software. Each node acts as the failover server for the other node in the same cluster.
Parallel Database Network
Physical Machines
Cluster 1
Cluster 2
Logical Partitions Storage Interconnect Disks
3 3 4
4 4
Figure 4-4 Mutual takeover
The mutual takeover configuration means that a failure would result in that nodes partition being recovered and its workload being taken over from the other node in a cluster which is already running a workload. The cluster software typically allows the invocation of failover scripts when failover occurs. You use these scripts to adjust various operating parameters when a failover or failback occurs. For example, the failover scripts may be used to shrink the bufferpools of both affected logical partitions when a failover occurs, and restore them to normal on failback. DB2 UDB provides several sample takeover scripts, tailored for the various cluster software environments.
103
4.2.2 High Availability on AIX

Enhanced Scalability (ES) is a feature of High Availability Cluster Multi-Processing (HACMP) for AIX. This feature provides the same failover recovery and has the same event structure as HACMP. Enhanced scalability also provides: Larger HACMP clusters. Additional error coverage through user-defined events. Monitored areas can trigger user-defined events, which can be as diverse as the death of a process, or the fact that paging space is nearing capacity. Such events include pre- and post-events that can be added to the failover recovery process, if needed. Extra functions that are specific to the different implementations can be placed within the HACMP pre- and post-event streams. A rules file (usr/sbin/cluster/events/rules.hacmprd) contains the HACMP events. User-defined events are added to this file. The script files that are to be run when events occur are part of this definition. HACMP client utilities for monitoring and detecting status changes (in one or more clusters) from AIX physical nodes outside of the HACMP cluster. The nodes in HACMP ES clusters exchange messages called heartbeats, or keepalive packets, by which each node informs the other nodes about its availability. A node that has stopped responding causes the remaining nodes in the cluster to invoke recovery. The recovery process is called a node_down event and may also be referred to as failover. The completion of the recovery process is followed by the re-integration of the node into the cluster. This is called a node_up event. There are two types of events: standard events that are anticipated within the operations of HACMP ES, and user-defined events that are associated with the monitoring of parameters in hardware and software components. One of the standard events is the node_down event. When planning what should be done as part of the recovery process, HACMP allows two failover options: hot standby (or idle standby) and mutual takeover. In a hot standby configuration, the AIX processor node that is the takeover node
is not running any other workload. In a mutual takeover configuration, the AIX
processor node that is the takeover node is running other workloads.
104
Generally, DB2 UDB Enterprise - Extended Edition (EEE) runs in mutual takeover mode with partitions on each node. One exception is a scenario in which the catalog node is part of a hot standby configuration. HACMP ES failover recovery allows pre-defined (also known as cascading) assignment of a resource group to a physical node. The failover recovery procedure also allows floating (or rotating) assignment of a resource group to a physical node. IP addresses, and external disk volume groups, or file systems, or NFS file systems, and application servers within each resource group specify either an application or an application component, which can be manipulated by HACMP ES between physical nodes by failover and reintegration. Failover and reintegration behavior is specified by the type of resource group created, and by the number of nodes placed in the resource group. For example, there is a DB2 UDB database partition (logical node). If its log and tablespace containers were placed on external disks, and other nodes were linked to those disks, it would be possible for those other nodes to access these disks and to restart the database partition (on a takeover node). It is this type of operation that is automated by HACMP. HACMP ES can also be used to recover NFS file systems used by DB2 UDB instance main user directories.
HACMP configuration examples

The following examples illustrate different failover support configurations and show what happens when failure occurs.
105
Hot standby failover

The following hot standby failover scenario consists of a two-processor HACMP cluster running a single-partition database instance (see Figure 4-5).
Client Workstation
Network: LAN
Processor 1 LAN Connection
Processor 1 Standby LAN Connection
db2inst
Processor 2 LAN Connection
Processor 1
Network RS232 LINK
Processor 2
Figure 4-5 Example of a AIX hot standby failover configuration
Both processors have access to the installation directory, the instance directory, and the database directory. The database instance db2inst is being actively executed on processor 1. Processor 2 is not active, and is being used as a hot standby. A failure occurs on processor 1, and the instance is taken over by processor 2. Once the failover is complete, both remote and local applications can access the database within instance db2inst. The database will have to be manually restarted or, if AUTORESTART is on, the first connection to the database will initiate a restart operation.
106
Mutual DB2 instance failover

The following mutual instance failover scenario consists of an HACMP system with two processors known as node10 and node20 (see Figure 4-6).
Client Workstation
Node 10 Standby
LAN Connection
Network: LAN
Node 10 Standby
LAN Connection
Node 10 LAN Connection
db2inst 1
db2inst 2
Node 20 LAN Connection
Node 10 Network RS232 LINK
Node 20
Figure 4-6 Example of a AIX mutual instance failover configuration
Two instances, db2inst1 and db2inst2, are created from a single installation path on a shared file system. Instance db2inst1 is created on /u/db2inst1, and instance db2inst2 is created on /u/db2inst2. Both of these paths are on a shared file system that is accessible to both processors. Each instance has a single database, with a unique path, that is also on a shared resource accessible to both processors. Both instances are accessed via remote clients over the TCP/IP protocol: db2inst1 uses the service name db2inst1_port (port number 5500) db2inst2 uses the service name db2inst2_port (port number 5550)
107
Remote clients accessing the db2inst1 instance have this instance cataloged in their node directory using node10 as the host name. Remote clients accessing the db2inst2 instance have this instance cataloged in their node directory using node20 as the host name. Under normal operating conditions, db2inst1 is running on node10, and db2inst2 is running on node20. If node10 were to fail, the failover script will start db2inst1 on node20, and the external IP address associated with node10 will be switched over to node20. Once the instance has been started by the failover script, and the database has been restarted, the remote clients can connect to the database within this instance as if it were running on node10.
4.2.3 High Availability on the Windows operating system

You can set up your database system so that if a machine fails, the database server on the failed machine can run on another machine. On Windows NT, failover support can be implemented with Microsoft Cluster Server (MSCS). To use MSCS, you require Windows NT Version 4.0 Enterprise Edition with the MSCS feature installed. MSCS can perform both failure detection and the restarting of resources in a clustered environment, such as failover support for physical disks and IP addresses. (When the failed machine is online again, resources will not automatically fall back to it, unless you previously configure them to do so.) Before you enable DB2 UDB instances for failover support, perform the following planning steps: 1. Decide which disks you want to use for data storage. Each database server should be assigned at least one disk for its own use. The disk that you use to store data must be attached to a shared disk subsystem, and must be configured as an MSCS disk resource. 2. Ensure that you have one IP address for each database server that you want to use to support remote requests. When you set up failover support, it can be for an existing instance, or you can create a new instance when you implement the failover support. To enable failover support, perform the following steps: 1. Create an input file for the DB2MSCS utility. 2. Invoke the db2mscs command. 3. If you are using a partitioned database system, register database drive mapping to enable mutual takeover.
108
After you finish enabling the instance for failover support, your configuration will resemble Figure 4-7.
Machine A Machine B
DB2 Group 0
DB2 Group 1 Cluster disks in a disk lower
Figure 4-7 Example MSCS configuration
The following sections describe the different types of failover support, and how to implement them. Before performing any of the steps described below, you must already have the MSCS software installed on every machine that you want to use in an MSCS cluster. In addition, you must also have DB2 UDB installed on every machine.
MSCS configuration examples

Two types of configuration are available: Hot standby Mutual takeover Currently, MSCS supports clusters of two machines. The clusters do not all have to have the same type of configuration. You can have some clusters that are set up to use hot standby, and others that are set up for mutual takeover. For example, if your DB2 UDB instance consists of five workstations, you can have two machines set up to use a mutual takeover configuration, two to use a hot standby configuration, and one machine not configured for failover support.
:E
(Each machine has DB2 code installed on a local disk)
:C :F
:C
SQLLIB
Quorum disk used by MSCS
:D
SQLLIB
109
Hot standby configuration

In a hot standby configuration, one machine in the MSCS cluster provides dedicated failover support, and the other machine participates in the database system. If the machine participating in the database system fails, the database server on it will be started on the failover machine. If, in a partitioned database system, you are running multiple logical nodes on a machine and it fails, the logical nodes will be started on the failover machine. Figure 4-8 shows an example of a hot standby configuration.
Cluster
Workstation A
Workstation B
Instance A
Instance A
Figure 4-8 MSCS hot standby configuration
Mutual takeover configuration

In a mutual takeover configuration, both workstations participate in the database system (that is, each machine has at least one database server running on it). If one of the workstations in the MSCS cluster fails, the database server on the failing machine will be started to run on the other machine. In a mutual takeover configuration, a database server on one machine can fail independently of the database server on another machine. Any database server can be active on any machine at any given point in time. Figure 4-9 shows an example of a mutual takeover configuration.
110
Cluster
Workstation A
Workstation B
Instance A Instance B
Instance A Instance B
Figure 4-9 MSCS mutual takeover configuration
4.3 High Availability of media

In this section, we look at various techniques which you can exploit to keep your Data Warehouse or datamarts highly available by minimizing the chances of media failure (such as disk failure) and the time to recover from such failures. Disk failure is one of the most commonly experienced problem. If the warehouse system has, for example 100 physical drives, have you estimated with what frequency you might expect some kind of disk failures? Careful planning and understanding of all the things that can go wrong will protect you from losing information.
4.3.1 DB2 UDB support for High Availability of media

DB2 UDB support that helps you eliminate or minimize the need to restore and rollforward when disks fail includes: Consistency bits Automatic redundancy for critical control files such as log control file, tablespace files
Ping-pong algorithm to ensure integrity of log head

Support for mirroring, RAID
111
But that is not enough. You also need to: Assign logs and data to separate devices. The logs are vital. If you lose the log, you are not able to recover to the point of failure and you lose committed transactions. Keep dual logging. DB2 UDB now has the capability to mirror the active log files to protect databases from: Accidental deletion of an active log Data corruption caused by hardware failure A new registry variable, DB2_NEWLOGPATH2, allows the database to write an identical copy of the log files to a different path on a physically separate disk. Take advantage of DB2 UDBs granular tablespace recovery. It allows independent recovery by assigning tables to different tablespaces. Add redundancy to log and data devices through mirroring and RAID. Add redundancy to data path such as dual adaptors. Perform regular backups. Store archived logs and database backups in a safe place.
4.3.2 Disk mirroring in AIX

Disk mirroring is a useful technique for maximizing the availability of your database. In DB2 Warehouse Manager environment, you can use this technique for the higher availability of a target database built in AIX. Mirroring is the process of writing the same data to multiple storage devices at the same time. This is done either sequentially, when data is only written to the mirror once the master write is successful or, in parallel, when both master and mirror writes occur at the same time. The first method is slower but you are more likely to have at least one good copy of the data if a failure occurs. When reading from a mirrored logical volume, AIX will read from either the master or the mirror, whichever is quicker at the time. If a media failure occurs, operations are automatically switched to the good copy and AIX marks the faulty copy as stale. Mirroring allows your users to continue working even though a media failure has occurred. Mirroring can be implemented in either software or hardware. However, mirroring does not remove the need to back up databases. For example, disk mirroring will not allow you to restore a table that has been lost or damaged as a result of user error.
112
Also, although disk mirroring dramatically reduces the impact of media failures, there is still a risk of damage to both sides of the mirror. If a database is held on one set of physical volumes, and a mirror image of the same database is maintained on a separate set of physical volumes, it is possible for both sets of physical volumes to be damaged or destroyed. This could happen as a result of a disaster or it could just be bad luck. In such instances, it will be necessary to recover the database from backup copies. In DB2 UDB, you should at least consider mirroring the log file directory where the active logs reside. This will help ensure full recoverability to the point of failure.
4.3.3 RAID technology

RAID stands for Redundant Arrays of Inexpensive Disks, and provides a method of classifying the different ways of using multiple disks to increase availability and performance. In DB2 Warehouse Manager environment, you can use this technique wherever the possibility of disk failure exists. RAID can eliminate or minimize the chance of unavailability of the critical warehouse resources such as warehouse control database and target database tables and logs by allowing operation to continue in the degraded mode when a disk failure occurs.
Disk arrays and RAID are hardware solutions to improve some or all of the following disk characteristics:
Performance: Operating multiple disks in parallel can boost I/O performance.

Size: Instead of engineering expensive large disks, replace them with a number of small disks that together behave as a large disk.
Reliability: Using either mirroring or parity information allows operation to continue in the degraded mode, even in the event of a single disk failure. Variety: Disk arrays come in a large variety of configurations with various number of tunable alternatives.
113
Disk arrays
The capacity of single large disks has grown rapidly, but the performance improvements have been modest, when compared to the advances made in the other subsystems that make up a computer system. The reason for this is that disks are mechanical devices, affected by delays in seeks and the rotation time of the media. In addition, disks are often among the least reliable components of the computer systems, yet the failure of a disk can result in the unrecoverable loss of vital business data, or the need to restore a tape backup with consequent delays. The use of arrays of inexpensive disks can offer a solution to these concerns. There is nothing unusual about connecting several disks to a computer to increase the amount of storage. Mainframes and minicomputers have always had banks of disks. The disk subsystem is called a disk array when several disks are connected and accessed by the disk controller in predetermined patterns designed to optimize performance and/or reliability. The driving force behind disk array technology is the observation that it is cheaper to provide a given storage capacity or data rate with several small disks connected together than with a single disk.
RAID classifications
Disk arrays seem to have been invented independently by a variety of groups. One specifically, the Computer Architecture group at the University of California, Berkeley invented the term Redundant Arrays of Inexpensive Disk (RAID). The original RAID classification described five levels of RAID (RAID-1 through 5). To these have been added RAID-0 (data-striping), RAID-1 Enhanced (data stripe mirroring) and Orthogonal RAID-5 (which includes extra redundancy of components such as disk adapters). RAID-0 is not a pure RAID type, since it does not provide any redundancy. Different designs of arrays perform optimally in different environments. The two main environments are those where a high I/O rate is needed, that is: High transfer rates are very important. High I/O rates is needed - that is for applications requesting short length random records. Table 4-1 shows the RAID array classifications.
114
Table 4-1 RAID Classifications

RAID Level Description
RAID-0 RAID-1 RAID-1 Enhanced RAID-2 RAID-3 RAID-4 RAID-5 Orthogonal RAID-5
Block Interleave Data Striping without Parity Disk Mirroring/Duplexing Data Strip Mirroring Bit Interleave Data Striping with Hamming Code Bit Interleave Data Striping with Parity Check Block Interleave Data Striping with One Parity Disk Block Interleave Data Striping with Skewed Parity RAID-5 with Additional Redundancy (such as disk adapters)
Here we describe RAID-0, RAID-1, and RAID-5 and their function, as follows.
RAID-0
RAID-0 (Figure 4-10) is the data organization term used when striping data across multiple disk drives, without parity protection. Data striping improves performance with large files, since reads/writes are overlapped across all disks. However, reliability is decreased, as the failure of one disk will result in a complete failure of the disk subsystem.
Disk Controller
xxxxx yyyyy
xxxxx zzzzz
xxxxx
xxxxx
xxxxx
Block o
Block n Disk 1 Disk 2 Disk 3 Disk 4 Disk 5
xxxxx = Blocks belonging to long files yyyyy & zzzzz = Blocks belonging to short files
Figure 4-10 RAID-0 (Block interleave data striping without parity)
115
RAID-1
RAID-1 is the term used with disk mirroring (Figure 4-11) or duplexing (Figure 4-12) at the hardware level. Whenever the computer makes an update to a disk, it can arrange to duplicate that update to a second disk, thus mirroring the original. This level of RAID is implemented in LAN Server fault tolerance. Either disk can fail, and the data is still accessible. Additionally, because there are two disks, a read request can be serviced from either disk, thus leading to improved throughput and performance on the disk subsystem. However, the down side is the cost of using 50% of disk storage space for mirroring.
D is k C o n tr o lle r
D is k 1
D is k 2
Figure 4-11 RAID-1 (Disk mirroring)
In the case of the IBM RAID controller there are two separate disk SCSI channels available from one card and so duplexing is possible from one disk controller card. Some might argue this is not true duplexing, but is mirroring on separate disk control channels. In any case, the IBM RAID controller provides an extremely cost effective and adaptable method for arranging data on a disk subsystem using RAID-1.
D is k C o n tro lle r
D is k 1
D is k C o n tro lle r
D is k2
Figure 4-12 RAID-1 (Disk duplexing)
116
RAID-5
RAID-5 (Figure 4-13) is the term used when striping data across three or more disks with skewed parity. This means the data organization is essentially the same as RAID-0, but there is an additional element of parity checking. The parity checking is used to encode the data and guard it against loss, and is referred to as a checksum, disk parity or Error Correction Code (ECC). The principle is the same as memory parity, where the data is guarded against the loss of a single bit of data. In RAID-5, the parity information is stored on the disk array to guard against data loss, and skewing is used to remove the bottleneck that would be created by having all the parity information stored on a single drive.
Disk Controller
xxxxx xxxxx yyyyy

Disk 1
xxxxx xxxxx yyyyy

Disk 2
xxxxx xxxxx Party

Disk 3
xxxxx Party yyyyy

Disk 4
Party xxxxx yyyyy

Disk 5
Block o
Block n
xxxxx & yyyyy = Blocks belonging to long files
Figure 4-13 RAID-5 (Block interleave data striping with skewed parity)
Using RAID technology to provide striping of data across multiple disks will often improve performance as well as enhance data integrity in an environment where data is predominantly read off the disk without a subsequent update (write).
RAID performance characteristics

The following is a summary of each RAID type:
RAID-0: Block interleave data striping without parity:

Data split into a small chunks, I/O done in parallel Fastest data-rate performance Allows seek and drive latency to be performed in parallel Significantly outperforms single large disk
117
RAID-1: Disk mirroring/disk duplexing and data strip mirroring:

Data written to drives sequentially Recommended for database systems when cost is not a factor Mirroring Often used for logs Fast and reliable, but requires 100% disk space overhead Data copied to each set of drives With LAN Server, performance almost as fast as RAID-0 due to split seeks (split seeks send half of the operation to one side of the mirrored drives, and the other half to other side) No performance degradation with a single disk failure RAID-1 Enhanced provides mirroring with odd number of drives
RAID-5: Block interleave data striping with skewed parity:

Data and parity striped across several drives Frequently used for database and other applications Economical Striping Often used in databases Best for random transactions Poor for large sequential reads if request is larger than block size Better write performance than RAID-3 and RAID-4 Block size is key to performance, must be larger than typical request size Performance degrades in recovery mode that is when a single drive has failed
Orthogonal RAID-5: RAID-5 with multiple orthogonal disk adapters:

All the benefits of RAID-5 Mirroring and striping Often used in databases including logs Improved performance (due to load being spread across disk adapters) Improved reliability due to redundancy of disk adapters as well as disks Table 4-2 summarizes RAID performance characteristics.
Table 4-2 Summary of RAID performance characteristics
RAID Level Capacity Large Transfers High I/O Rate Data Availability
Single Disk RAID-0 RAID-1 RAID-5 Orthogonal RAID-5
Fixed (100%) Excellent Moderate (50%) Very Good Very Good
Good Very Good Good Very Good Very Good
Good Very Good Good Good Good Poor Good Good Very Good
118
DB2 UDB tuning tips with RAID-5

Since RAID-5 does its own internal striping, DB2 UDB striping is not strictly necessary. To effectively exploit RAID-5 devices with a single container, you want to: Make a tablespace's EXTENTSIZE match to the RAID stripe size (either equal or a whole multiple). Make the PREFETCHSIZE multiple of EXTENTSIZE and RAID stripe size * # of disks in RAID array. Enable parallel I/O within a single tablespace container. You can do this by using the db2set command. The examples are:
db2set db2_parallel_io=* (For all tablespaces) db2set db2_parallel_io=1,2,4 (For selected tablespace id)
With support for the DB2_PARALLEL_IO registry variable, it is no longer necessary to define multiple containers to achieve parallel I/O. Tell DB2 UDB that the container does it's own internal striping during the tablespace creation using the db2set command:
db2set db2_striped_containers=on
You need to do this for all DMS containers because they have a tag which marks the container as owned, and is placed at the start of the container. The presence of this tag means that each extent is not aligned on a RAID stripe boundary, even if the extent size and stripe size are equal. To avoid this undesirable problem, you want to set the DB2_STRIPED_CONTAINERS registry variable to ON. It only has effect on newly created or restored tablespaces.
4.3.4 High Availability through split mirror

In this section, we describe the split mirror feature of intelligent disk storage servers, such as the IBM Enterprise Storage Server (ESS), and some possible ways that DB2 Warehouse Manager can use this feature. You may find this split mirror technique very useful when your DB2 warehouse environment involves very large target databases and you cannot afford enough backup windows.
Split mirror
A backup always degrades the performance of the production system. Especially if the warehouse database is big or in a 7x24 hour environment, it is hard to find a time frame when to plan the backup to not interfere with the normal operation To free the production system from the overhead of backup, a copy or mirror of the database would be nice to be available for backup, report or other purposes.
119
Some intelligent storage servers such as IBM Enterprise Storage Server (ESS), support the split mirror feature. For example, the FlashCopy is the ESSs implementation of the split mirror feature. Split mirror means that identical and independent copies of disk volumes can be established within those storage servers. These copies can normally be established in a very short time (for example, 5 to 20 seconds depending on device). If the database resides on a storage server that supports the split mirror feature, a copy of the disk volumes can be established and assigned to another (backup) machine. On the backup machine, the (backup) database can then be accessed exclusively for backup or other purposes without reference to users. Split mirror allows you to establish an identical and independent copy of a disk volume (see Figure 4-14). The source and target volumes must reside in the storage server.
source system
target system
Storage Server
not assigned to any system
assigned to source system
assigned to target system
split mirror source volume split mirror target volume
Figure 4-14 Split mirror concept
120
As you can see in Figure 4-14, the target volumes should be assigned to another physical machine. If the target volumes are assigned to the same machine where the source volumes are, the operating system may have problems identifying the new disk volume, as it is an exact copy of the original disk. Unique disk identifiers, size of the disk, the copied data itself, and so on will appear twice on the same machine which may cause problems.
DB2 Suspended I/O feature

During the creation of the copy volumes, an important requirement is that the data on the disk volumes is consistent. One way to establish this is to shutdown the database and to synchronize all the data that may reside in the memory of the operating system to disk. After the split mirror is established the database can be started again. If the database cannot be stopped then the database itself must provide features to ensure that the data on the disk will be in a consistent state at the time of establishing the split mirror volumes. DB2 UDB provides the DB2 Suspended I/O feature for this purpose. The DB2 Suspended I/O supports continuous system availability by providing a full implementation for online split mirror handling; that is, splitting a mirror without shutting down the database. A split mirror is an instantaneous copy of the database that can be made by mirroring the disks containing the data, and splitting the mirror when a copy is required. Disk mirroring is the process of writing all of your data to two separate hard disks; one is the mirror of the other. Splitting a mirror is the process of making a backup copy of the mirror. If you would rather not back up a large database using the DB2 UDB backup utility, you can make copies from a mirrored image by using suspended I/O and the split mirror function. This approach also: Eliminates backup operation overhead from the production machine Represents a fast way to clone systems Represents a fast implementation of idle standby failover. There is no initial restore operation, and if a rollforward operation proves to be too slow, or encounters errors, reinitialization is very fast.
IBM Enterprise Storage Server (ESS)

You can fulfill the important demands of a high level of data warehouse availability, which extends to continuous availability, 24 hours a day and 7 days a week (24x7) operation with the Enterprise Storage Server and its Advanced Copy Services.
121
The ESS Copy Services provide replication of mission critical data; point-in-time with FlashCopy, dynamic synchronous mirroring to a remote with Peer-to-Peer Remote Copy (PPRC) and asynchronous copying to a remote site with Extended Remote Copying (XRC). This last function is only available in a z/OS and OS/390 environment. The IBM Enterprise Storage Server (ESS) is a member of the Seascape family of storage products. The ESS was announced in June 1999 and was generally available in September 1999. Since its announcement it has revolutionized the storage marketplace. It consists of a storage server and attached disk storage devices. The storage server provides integrated caching and RAID support for the attached disk devices. The disk devices are attached via a serial interface. The ESS can be configured in a variety of ways to provide scalability in capacity and performance. Redundancy within the ESS provides continuous availability. It is packaged in one or more enclosures, each with dual line cords and redundant power. The redundant power system allows the ESS to continue normal operation when one of the line cords is deactivated. The ESS provides the image of a set of logical disk devices to the attached servers. The logical devices are configured to emulate disk device types that are compatible with the attached servers. The logical devices access a logical volume that is implemented using multiple disk drives. The ESS can be broken down into different components. The storage server itself is composed of two clusters that provide the facilities with advanced functions to control and manage data transfer. Should one cluster fail, the remaining cluster can take over the functions of the failing cluster. A cluster is made up of the following subcomponents:
Host adapters:
Each cluster has one or more host adapters (HAs). Each host adapter provides one or more host I/O interfaces. A host adapter can communicate with either cluster complex.
Device adapters:
Each cluster has one or more Device Adapters (DAs). Each device adapter provides one or more storage device interfaces. Disk drives are attached to a pair of device adapters, one in each cluster, so that the drives are accessible from either cluster. At any given time, a disk drive is managed by only one device adapter.
122
Cluster complex:
The cluster complex provides the management functions for the ESS. It consists of cluster processors, cluster memory, cache, nonvolatile storage (NVS) and related logic: Cluster processors: The cluster complex contains four cluster processors (CP) configured as symmetrical multiprocessors (SMP). The cluster processors execute the licensed internal code that controls operation of the cluster. Cluster memory / cache: These are used to store instructions and data for the cluster processors. The cache memory is used to store cached data from the disk drives. The cache memory is accessible by the local cluster complex, by device adapters in the local cluster, and by host adapters in either cluster. Nonvolatile storage (NVS): This is used to store a nonvolatile copy of active written data. The NVS is accessible to either cluster-processor complex and to host adapters in either cluster. Data may also be transferred between the NVS and cache. The disk drives provide the primary nonvolatile storage medium for any host data stored within the ESS Storage devices. They are grouped into ranks and managed by the clusters. As a member of the IBM Seascape family, the ESS provides the outboard intelligence required by SAN solutions, off-loading key functions from host servers, which frees up valuable processing power for applications. As a comprehensive SAN-based storage solution, the ESS provides considerable management flexibility to meet the fast-paced requirements of the next century.
ESS Advanced Copy Services Copy Services is a separately sold feature of the Enterprise Storage Server. It
brings powerful data copying and mirroring technologies to Open Systems environments previously available only for mainframe storage. This book deals with the two features of Copy Services for the Open Systems environment: FlashCopy Peer-to-Peer Remote Copy (PPRC)
FlashCopy
FlashCopy provides an instant or point-in-time copy of an ESS logical volume. The point-in-time copy functions give you an instantaneous copy, or view, of what the original data looked like at a specific point-in-time. This is known as the T0 (time-zero) copy.
123
Both the source and target volumes reside in the ESS. Normally, the source volume is only visible to the source machine and the target volume only to the target machine When a FlashCopy is invoked, the command returns to the operating system as soon as the FlashCopy pair has been established and the necessary control bitmaps have been created. This process takes only a few seconds to complete. Thereafter, you have access to a T0 copy of the source volume. As soon as the pair has been established, you can read and write to both the source and the target volumes. The point-in-time copy created by FlashCopy is typically used where you need a copy of production data to be produced with minimal application downtime. It can be used for online backup, testing of new applications, or for creating a database for data-mining purposes. The copy looks exactly like the original source volume and is an instantly available, binary copy. See Figure 4-15 for an illustration of FlashCopy concepts.
FlashCopy provides a T0 copy
Source
Target
FlashCopy command issued
Copy immediately available
Time
Write
Read
Read
Write
Read and write to both source and copy possible
When copy is complete, relationship between source and target ends
Figure 4-15 Implementation of FlashCopy
124
FlashCopy is possible only between disk volumes. It also requires a target volume to be defined within the same Logical Subsystem (LSS) as the source volume. A source volume and the target can be involved in only one FlashCopy relationship at a time. When you set up the copy, a relationship is established between the source and the target volume and a bitmap of the source volume is created. Once this relationship is established and the bitmap created, the target volume copy can be accessed as though all the data had been physically copied. While a relationship between source and target volumes exists, a background task copies the tracks from the source to the target. The relationship ends when the physical background copy task has completed. You can suppress the background copy task using the Do not perform background copy (NOCOPY) option. This may be useful if you need the copy only for a short time, such as making a backup to tape. If you start a FlashCopy with the Do not perform background copy option, you must withdraw the pair (a function you can select) to end the relationship between source and target. At the time when FlashCopy is started, the target volume is basically empty. The background copy task copies data from the source to the target. The FlashCopy bitmap keeps track of which data has been copied from source to target. If an application wants to read some data from the target that has not yet been copied to the target, the data is read from the source; otherwise, the read is satisfied from the target volume. When the bitmap is updated for a particular piece of data, it signifies that source data has been copied to the target and updated on the source. Further updates to the same area are ignored by FlashCopy. This is the essence of the time-zero point-in-time copy mechanism. Before an application can update a track on the source that has not yet been copied, the track is copied to the target volume. Reads that are subsequently directed to this track on the target volume are now satisfied from the target volume instead of the source volume. After some time, all tracks will have been copied to the target volume, and the FlashCopy relationship will end. You cannot create a FlashCopy on one type of operating system and make it available to a different operating system. You can make the target available to another host running the same type of operating system.
125
Note 1: When establishing the split mirror the data on the source volume must be in a consistent state. This can either be achieved by stopping the application and flushing the data from memory to disk, or if the application cannot be stopped, then by application itself providing a feature to ensure that the data is consistent while the FlashCopy is being established and can recover from that state afterwards on the target side.
Note 2: DB2 UDB provides the suspended I/O feature and db2inidb tools to make use of the split mirror feature. PPRC
PPRC is a synchronous protocol that allows real-time mirroring of data from one Logical Unit (LUN) to another LUN in a second ESS. The secondary ESS can be located at another site some distance away. PPRC is application independent. Because the copying function occurs at the disk subsystem level, the application has no knowledge of its existence. The PPRC protocol guarantees that the secondary copy is up-to-date by ensuring that the primary copy will be written only if the primary system receives acknowledgement that the secondary copy has been written. Figure 4-16 shows the sequence of events.
Host server
1. Write (channel end) 2. Write to secondary 3. Write hit at secondary 4. Acknowledgement (device end)
Primary storage control ESS

Figure 4-16 PPRC write cycle
3 2
Secondary storage control ESS
126
1. The host server requests a write I/O to the primary ESS. The write is staged through cache into Non-Volatile Storage (NVS). 2. PPRC dispatches the write over an ESCON channel to the secondary ESS. The write hits the secondary ESSs NVS. 3. The primary then expects acknowledgment of the remote write. If the secondary write fails, the write does not return to the host server and is eventually aged from NVS. 4. The write returns to the host servers application. Once acknowledgement of the write has been received by the primary, both the primary and secondary write I/Os are eligible for destage to disk. Destage from the cache to the disk drives on both the primary and the secondary ESS is performed asynchronously. If acknowledgement of the remote write is not received within a fixed period of time, the write is considered to have failed, and is rendered ineligible for destage to disk. At this point, the application receives an I/O error, and in due course, the failed write I/O is aged-out of each NVS.
IBM StorWatch
IBM StorWatch, IBM's Enterprise Storage Resource Management (ESRM) solution, is a growing software family whose goal is to enable storage administrators to efficiently manage storage resources from any location within an enterprise. Widely dispersed, disparate storage resources will ultimately be viewed and managed through a single, cohesive control point. The StorWatch family of ESRM products addresses the storage challenges of today with a clear vision for the future. IBM StorWatch Expert is a program product in the IBM StorWatch software family and it helps you manage your Enterprise Storage Server (ESS) and Enterprise Tape Library (ETL) using a Web browser interface. The StorWatch Expert helps you do the following tasks: Performance management Asset management Capacity management Figure 4-17 shows how the StorWatch Expert works with your ESS and/or ETL. The StorWatch Expert communicates with your ESS/ETL through the TCP/IP network. Therefore, you can gather information on any ESS/ETL across the world as long as you have a communication path between the StorWatch Expert and the ESS/ETL through your intranet.
127
Figure 4-17 StorWatch Expert communicates ESS/ETL through TCP/IP network
The StorWatch Expert itself runs under a Windows NT 4.0 or AIX V4.3.3 operating system. However, you do not have to operate Expert right where it runs, since the user interface of the Expert is the Web browser interface. In other words, you can operate the Expert through Netscape Navigator or Microsoft Internet Explorer. The StorWatch Expert asks your ESS/ETL to send information about their capacity or performance. When the StorWatch Expert receives this information, it is stored into a DB2 UDB database. Thus, you can prepare and produce reports containing just the information you need.
Using split mirror features in conjunction with DB2

The DB2 suspended I/O feature and the db2inidb tool were introduced with DB2 UDB V7.1 fixpack2. We show how, and in which order, the commands must be used to make use of the split mirror features. We will describe three scenarios, which are the three parameters snapshot, standby, mirror of the db2inidb command.
128
The suspended I/O support is necessary to bring the database into a consistent state before establishing the split mirror. With the write suspend command, the database enters a well defined state where it can recover later using the db2inidb tool. During the suspend of the database, the DB2 UDB clients will see that their transactions will hang. As soon as the database is resumed again for writing (write resume), the transactions will proceed normally. The db2inidb tool is necessary in combination with the split mirror feature, to bring the target database into a running state again after the split mirror was established. The db2inidb tool can perform crash recovery or put a database in rollforward pending. There are three possible options:
Snapshot: This initiates crash recovery, making the database consistent. A new log chain starts, and the database will not be able to roll forward through any of the logs from the original database. The database is available for any operation, including backup. Standby: This places the database in rollforward pending state. Crash recovery is not performed, and the database remains inconsistent. Mirror: This causes a mirrored copy of the database to replace the original database. The database is placed in rollforward pending state, and the write suspend state is turned off. Crash recovery is not performed, and the database remains inconsistent.
Three scenarios of DB2 UDB using split mirror

Split mirror can be used with any DB2 UDB database. When using DB2 Warehouse Manager, it can be implemented mainly for target warehouses but might also be used with the warehouse control database in regards of the cost and the level of availability required. We now show three scenarios in which DB2 UDB can use the split mirror features of storage servers.It is assumed that the source and target DB2 UDB instance reside on different machines. Furthermore, the setup of the DB2 UDB instance must be the same on the source and on the target machine, for example, userid and groupid of the instance owner. Also, the operating system and DB2 UDB versions should be the same.
Split mirror to clone a database

The first scenario is to set up a clone database on the target system. A clone database can be used for taking an offline backup, generating a test database, or generating reports (Figure 4-18).
129
Active Database
Secondary Database
Snapshot for reporting Populate a test system
split
Master Data & Logs Mirror Copy
Figure 4-18 DB2 UDB clone database
These are the steps to follow to make a clone database: 1. Suspend I/O on source database. The following DB2 UDB command will suspend all write activities from DB2 UDB clients, but they will not be disconnected from the database:
db2 set write suspend for database
2. Establish split mirror. This step is dependent on the vendor. For example, issue FlashCopy on the IBM ESS. The whole database must be copied to use it on the target machine. 3. Resume I/O on source database. After a few seconds the split mirror is established and the database can be made available again for writing. The transactions will proceed as normal.
db2 set write resume for database
4. Make data visible on target machine. The target volumes will now contain all the data from the time the split mirror was established, but they may not yet be visible to the target operating system. The operating system on the target machine has to provide access to the data (import, mount, and so on). (For initial setup, the database needs to be cataloged at the target machine.) 5. Start target (clone) database Issue the db2start command at the instance at the target machine. 6. Bring the clone database into a consistent state (perform crash recovery) The following command will rollback every uncommitted transaction:
db2inidb <target-database> as snapshot
130
The clone database can now be used for an offline backup, or as a test database.
Note: Any DB2 UDB backup image taken on the cloned database can not be used for restore on the original database for the purpose of performing rollforward recovery using the log files produced on the original database after the split mirror.
Split mirror as a standby database

The second scenario is to set up a standby database. A standby database in this context is a database where the log files of the source database will be applied on the target (standby) database. The standby database will be in a permanent rollforward process as long as the source database produces logfiles (and as long as the source database is available). See Figure 4-19.
Active Database
Secondary Database
split
Master Data & Logs Mirror
Figure 4-19 DB2 UDB standby database
Hot standby Take backups that can be restored on the primary system
offloads cycles from primary
Snapshot for reporting
Copy
Note: Do not copy the logfiles using the split mirror feature. This will break your rollforward chain. The necessary logs for rollforward must all be acquired from the source database as soon as they are available (no longer used by the source database). If logfiles have been copied to the target machine using the split mirror feature, then they must be deleted.
The steps required to set up a standby database differ only at the last step of scenario one (setup a clone database). We only cover the last step here. Set the database into rollforward mode: db2inidb <target-database> as standby Now the logfiles from the source-database can be used to rollforward the target-database. As long as there are new logs available at the source-database, the rollforward step can be repeated. (To make the transfer of the logfiles easier, a user exit may be configured.)
131
db2 rollforward db <target-database> to end of logs As long as the database stays in rollforward pending mode, no database backups are allowed. If the source database crashes, the standby database can be activated for access. (Make sure that all data has already been copied to the target volumes.) The rollforward must be stopped with the stop option of the DB2 UDB rollforward command: db2 rollforward db <target-database> stop Then the users can switch over to the standby database to continue their work.
Split mirror as a backup image

The third scenario shows how to use the mirror option of the db2inidb tool. The purpose of this option is to provide the possibility of using a split mirror database for restore on the source database and then to rollforward the logs of the source database (Figure 4-20).
Active Database
Secondary Database
Mirror was never updated on the secondary system
split
Master Data Only Mirror
Figure 4-20 DB2 UDB mirror database
Copy
Note: Do not copy the logfiles using the split mirror feature. All necessary logs for recovery must stay at the source database or on another location, but not on the target database. If logfiles have been copied to the target machine using the split mirror feature, then they must be deleted.
The steps required are: 1. Suspend I/O on source database. The following DB2 UDB command will suspend all write activities from DB2 UDB clients, but they will not be disconnected them from the database.
db2 set write suspend for database
132
2. Establish split mirror. This step is dependent on the vendor. For example, issue FlashCopy on the IBM ESS. The logfiles should not be copied to the target database. Make sure that all data will be copied to the target machine sooner or later. 3. Resume I/O on source database. After a few seconds the split mirror is established and the database can be made available again for writing. The transactions will proceed as normal.
db2 set write resume for database
4. Make data visible on target machine. The target volumes will now contain all the data from the time the split mirror was established, but they may not yet be visible to the target operating system. The operating system on the target machine has to provide access to the data (import, mount, and so on). (For first time setup, the database needs to be cataloged at the target machine.)
Note: No DB2 UDB operations are allowed on the target (split mirror) database after the split mirror was established in order to use the target database for restore on the source database. The database needs to stay in this frozen state.
The target database cannot be backed up using DB2, but can be backed up using operating system tools. If the source database happens to crash it can be restored with the split mirror image at the target database. See the following steps: 1. Stop the source database (we want to restore this database).
db2stop
2. Restore the database. With operating system tools (for example, copy, xcopy, cp, tar and so on) copy the datafiles of the split mirror database over the original database. But do not copy the logfiles from the target database to the source database. (Make sure that all data has already been copied to the target database.) Another restore possibility would be to reverse the split mirror and copy the data on the target disk volumes back to the source disk volumes using the split mirror feature of the storage server again. 3. Start the source database again.
db2start
4. Reestablish the mirrored copy on the source database.

db2inidb <database> as mirror
133
Now a rollforward to end of logs of the database can be started. The logs from the original source database will be used.
Using AIX splitcopy feature for DB2 UDB backup

In this section, we look at the AIX splitcopy feature and how it can be used for DB2 UDB backup purposes (Figure 4-21). The AIX splitcopy feature was introduced with AIX 4.3.3.0. Its purpose is to provide an online backup of an AIX Journaled File System (JFS). DB2 UDB can use this splitcopy feature to create a DB2 UDB datafile backup. This backup is simply a copy of all the datafiles of a DB2 UDB database. This backup can be restored and then a rollforward can be done. This is almost the same as the third scenario of the split mirror usage That we just described in the previous section.
AIX splitcopy feature

Each AIX JFS must reside on a logical volume. All logical volumes are managed by the AIX Logical Volume Manager (LVM). With the AIX LVM, it is possible to mirror a JFS by mirroring the logical volumes the JFS is built on. A logical volume can have one (no mirror), two, or three physical copies of the same data (Figure 4-21).
/dbdata Filesystem view of the data
logical volume
Logical view of the data
physical volume
physical volume
physical volume
Physical data
Figure 4-21 AIX LVM mirroring with three physical copies
134
The following requirements are necessary to use the AIX splitcopy feature: Logical volume must be mirrored with AIX LVM Journaled files system log (JFS log) must also be mirrored The number of copies for the JFS log must be equal the number of copies for the filesystems logical volume If multiple filesystems needs to use the splitcopy feature at the same time, each filesystem must have its own mirrored journal file system log. Splitcopy separates one physical copy of the mirrored logical volume and assigns it to a new filesystem. This new splitcopy filesystem will be read only. The splitcopy can be established with the following command shown in Example 4-1.
Example 4-1 Splitcopy example
# chfs -a splitcopy=/dbcopy /dbdata splitlvcopy00 backup requested(0x100000)... log redo processing for /dev/splitlvcopy00 syncpt record at de9c08 syncpt record at d69b74 end of log e67488 syncpt record at de9c08 syncpt address d49074 number of log records = 10552 number of do blocks = 59 number of nodo blocks = 4 #lsvg -l rootvg |grep db splitlv jfs 6 12 2 splitlvcopy00 jfs 0 0 0
open/stale open???????
/dbdata /dbcopy
The original JFS will now be in a stale mirrored state, because the one physical disk that was split off the mirror will not be synchronized any more. But the relation between the split off physical volume and the LVM mirrored logical volume still exists. Refer to Figure 4-22.
135
/dbdata
/dbcopy Filesystem view of the data
logical volume
logical volume
Logical view of the data
physical volume
physical volume
physical volume
Physical data
Figure 4-22 Splitcopy read-only filesystem
For more information about splitcopy, see 5.8 Online JFS Backup in AIX Version 4.3 Differences Guide, SG24-2014. DB2 UDB can use this splitcopy feature to create a database backup image on the splitcopy filesystem that can be used to restore the DB2 UDB database and also rollforward the logs.
Backing up using AIX splitcopy

The following steps are required for backup using AIX splitcopy: 1. Create the mirrored filesystems and, if needed, the mirrored logfiles. The best practice is to use three copies, so that the production database will have at least two copies to provide availability after the splitcopy was established. 2. Create database on one or more (LVM) mirrored filesystems. We do not recommend for you to put the logfiles on this mirrored filesystem. The backup and the restore of the database will be easier. 3. Suspend I/O to database. Suspend access from clients to the database. The clients will not lose their connection, but the write and update transactions will hang. db2 set write suspend for database
136
4. Establish splitcopy. For each filesystem, issue the following command as root user. It is possible to specify which mirror copy to split off by using the -a copy option. The default is to split off the second copy. Valid options are 1,2, or 3. chfs -a splitcopy=/dbcopy -a copy=3 /dbdata 5. Resume I/O on database. db2 set write resume for database 6. Back up database. As instance owner (or root user) use an AIX backup command to back up the datafile of the database. Ensure that all files are being backed up. For example: cd /dbcopy; tar -cvf /tmp/db2inst1.tar ./db2inst1 7. Re-establish original mirror. This can be done by removing the generated splitcopy filesystem. As root user issue a command like the following: rmfs -r /dbcopy This will synchronize the splitcopy physical volume again into the original mirror.
Restoring a backup that was made using AIX splitcopy

The following steps are required for restoring a backup that was made using AIX splitcopy: 1. Stop the database. db2stop 2. Restore the backup image on the original destination. As instance owner (or root user) restore the backup image. Be sure that no original logs will be overwritten. For example: cd /dbdata; tar -xvf /tmp/db2inst1.tar 3. Start the database instance. db2start 4. Reestablish the mirrored copy. As instance owner issue following command: db2inidb <database> as mirror 5. Rollforward the original logs. db2 rollforward db <database> to end of logs and stop
137
4.3.5 Tivoli Storage Manager

In this section, we give an overview of Tivoli Storage Manager in order to assist database administrators who want to gain a basic understanding of Tivoli Storage Manager before configuring their system for backup and recovery of the critical DB2 UDB databases such as the warehouse control database and the target databases in DB2 UDB warehousing environment.
Overview
Tivoli Storage Manager, formerly known as ADSM, is a full-function storage software product that addresses the challenges of complex storage management across distributed environments. It protects and manages a broad range of data, from the workstation to the corporate server environment. Tivoli Storage Manager provides: Centralized administration for data and storage management Efficient management of information growth Customized backup solutions for major groupware and database products Tivoli Storage Manager is the core application of the Tivoli Storage Management solution set. Tivoli Storage Manager is an enterprise-wide storage management application for the network. It provides automated storage management services (including backup and restore, archive and retrieve, hierarchical space management and disaster recovery) to multi-vendor workstations, personal computers, mobile laptops and servers of all sizes and operating systems, which are connected via WAN, LAN, and SAN. Tivoli Storage Manager includes these components:
Server: The server provides backup, archive, and space management services to its defined clients. The server maintains its own database and recovery log for information about Tivoli Storage Manager resources, users, and user data including all backed-up, archived and migrated files. The client data itself is stored in server-controlled entities called storage pools. These are groups of random and sequential access media that store backed-up, archived, and space-managed files.
The Tivoli Storage Manager server is responsible for maintaining the integrity of client sessions, reliably receiving client data, storing client data in storage pools, and efficiently managing that data internally so that it can be restored or retrieved when required. You can set up multiple servers in your enterprise network to balance storage, processor, and network resources. Tivoli Storage Manager allows you to manage and control multiple servers from a single interface that runs in a Web browser (the enterprise console).
138
Administrative interface: The administrative interface allows administrators to control and monitor server activities, define management policies for client files, and set up schedules to provide services at regular intervals.
Administrative functions are available from an administrative client command line and from a Web browser interface. A server console is also available.
Backup-Archive client: The Backup-Archive client allows users to maintain backup versions of their files, which they can restore if the original files are lost or damaged.
Users can also archive files for long-term storage and retrieve the archived files when necessary. A command line interface, native GUI interface, and Web browser interface are available for the Backup-Archive clients.
Application program interface (API): The API allows users to enhance existing application programs with backup, archive, restore, and retrieve services. When users install the Tivoli Storage Manager API client on their clients, they can register as client nodes with a Tivoli Storage Manager server.
Tivoli Storage Manager Backup-Archive client

The Tivoli Storage Manager Backup-Archive client is designed to back up and restore, archive and retrieve client file system data. The client therefore can back up any non-database and database files. Tivoli Storage Manager clients use standard operating system functions to access files within file systems, but they do not require to understand any logical structure that might exist within a file. This affects how DB2 UDB and other database systems are backed up. Each database appears as an individual file on the server or client file systems. A Tivoli Storage Manager Backup-Archive client running on an DB2 UDB server or client can back up and restore, archive and retrieve entire DB2 UDB databases. It cannot back up smaller increments. Other than the issues of size and replication, using a Tivoli Storage Manager Backup-Archive client to back up DB2 UDB databases is straightforward. Each database is a self-contained data file that is backed up and restored. Tivoli Storage Manager restores a database in its entirety, because it is just a file for Tivoli Storage Manager. If a database is deleted or corrupted, it is a simple task for Tivoli Storage Manager to restore the most recent or any previous backup version of this database from the Tivoli Storage Manager server to the DB2 UDB server or client. The Tivoli Storage Manager Backup-Archive client, however, does not meet all requirements for an ideal storage management solution in a DB2 UDB environment.
139
Here are some of the drawbacks when using the Tivoli Storage Manager Backup-Archive client: Consider a 5 GB database that changes everyday. The Tivoli Storage Manager Backup-Archive client will back up the full 5 GB even if only a 2 MB document has changed. You waste a lot of time and storage space using this strategy. Many databases need to operate twenty four hours a day, seven days a week so they are in use all the time and a consistent backup cannot be taken. The alternative is to quiesce the DB2 UDB database and take backups, but this would result in server unavailability, which is not good for business.
Tivoli Storage Manager API

To overcome the above restrictions of the Backup-Archive client, DB2 UDB uses the Tivoli Storage Manager API code to facilitate database backup. DB2 UDB provides its own backup utility which allows backup at the tablespace level as well as a full database. The backup utility can be setup to use Tivoli Storage Manager as the backup media, as you will see later. Therefore, the two client types work together to provide full data protection for your DB2 UDB environment. The API client and the Tivoli Storage Manager Backup-Archive client can run simultaneously on the same DB2 UDB server, however, they are totally separate clients as far as the Tivoli Storage Manager server is concerned.
Day-to-day management
In this section we will discuss how to automate the DB2 UDB backup and how to verify the success of these backups.
db2adutl utility
The db2adutl is a very useful command that comes with DB2 UDB to manage old backups and logfiles that reside on the Tivoli Storage Manager server. The tool is located in the $HOME/sqllib/adsm directory of the instance owner in AIX environment. On Windows 2000, the tool is located in the ...\sqllib\bin directory. The db2adutl utility provides the ability to query, extract, delete, and verify backups, loadcopies, and log archives from Tivoli Storage Manager. All backup file names include a time stamp and therefore are unique for every backup to Tivoli Storage Manager. The backup copies in Tivoli Storage Manager are therefore always active and never deleted by normal Tivoli Storage Manager file expiration processing. The db2adutl utility provides a way of deleting backups and log files that are no longer required. Example 4-2 shows the syntax to invoke the db2adutl utility.
140
Example 4-2 db2adutl utility example

>>-db2adutl-----------------------------------------------------> >-----+-QUERY--+-------------------------------------+-----------------------------+> | +-+------------+---+---------------+--+ | | | +-TABLESPACE-+ '-SHOW INACTIVE-' | | | | +-FULL-------+ | | | | '-LOADCOPY---' | | | '-LOGS--+---------------------+-------' | | '-BETWEEN sn1 AND sn2-' | +-EXTRACT--+--------------------------------------------------------------+--+ | +-+------------+--+---------------+----+---------------------+-+ | | | +-TABLESPACE-+ '-SHOW INACTIVE-' '-TAKEN AT-timestamp--' | | | | +-FULL-------+ | | | | '-LOADCOPY---' | | | '-LOGS--+----------------------+-------------------------------' | | '-BETWEEN sn1 AND sn2--' | +-DELETE--+-------------------------------------------------------+----------+ | +-+------------+---+----------------------------------+-+ | | | +-TABLESPACE-+ +-KEEP--n--------------------------+ | | | | +-FULL-------+ +-OLDER--+-------+---+-timestamp-+-+ | | | | '-LOADCOPY---' | '-THAN--' '-n--days---' | | | | | '-TAKEN AT--timestamp--------------' | | | '-LOGS--+-------------------------+---------------------' | | '-BETWEEN--sn1--AND--sn2--' | '-VERIFY--+----------------------------------------------------------------+-' +-+------------+---+----------------+---+----------------------+-+ | +-TABLESPACE-+ '-SHOW INACTIVE--' '-TAKEN AT--timestamp--' | | '-FULL-------' | '-LOGS--+----------------------+---------------------------------' '-BETWEEN sn1 AND sn2--' >-----+--------------------------------+------------------------> '--+-DATABASE-+--database_name---' '-DB-------' >-----+--------------------+---+---------------------+----------> '-NODE--node_number--' '-PASSWORD--password--' >-----+----------------------+---+--------------------+---------> '-NODENAME--node_name--' '-WITHOUT PROMPTING--' >-----+---------------+---------------------------------------->< '-OWNER--owner--'
QUERY queries either tablespace, full backup, loadcopy images, or log archives. If you type query only, it returns all four. EXTRACT, if you do not qualify it, displays the name of each backup image and prompts you to extract the image. If you qualify the command you reduce the scope of the search. For this option, time stamps do not apply to log archives.
141
DELETE, if you do not qualify it, works similarly to extract, except that backup images are marked inactive and archive objects (log files) are deleted.
The VERIFY qualifier, performs consistency checking on the tablespace or backup copy that is on the Tivoli Storage Manager server. This parameter causes the entire backup image to be transferred over the network. The KEEP qualifier n keeps only the newest n images. The OLDER qualifier (THAN is optional) deletes any images older than the specified time stamp (this must be the complete time stamp string) or the specified number of days. Warning: If you automated the deletion of backups using this qualifier, be aware backups may fail and your automation may delete your last valid backups. The TAKEN AT <timestamp> qualifier is the time stamp with which you want to perform the operation. This qualifier is not allowed with LOGS, because LOGS have no time or date association. The SHOW INACTIVE qualifier shows also the inactive versions of full database backup, tablespace or loadcopy images that have been deleted with db2adutl and therefore marked deleted on the Tivoli Storage Manager server. This qualifier will only work if the Tivoli Storage Manager server is configured to allow inactive versions. The DATABASE qualifier restricts the search to a particular database. The WITHOUT PROMPTING qualifier disables the prompt. Be very careful when using this option because it may delete more than you have expected to delete.
Verification of DB2 UDB backups

The most important and, of course, the most reliable way is to see whether a backup is recoverable - to actually restore it. At least once, a typical backup should be restored to test if the backup and the restore process itself is valid. There is no way to restore an unsuccessful, incomplete or even missing backup. Once the backup process itself has been validated by doing a restore, the main purpose of this chapter is to outline which other possibilities there are to verify the success of the backup operation and to verify the backup object itself.
Verify backup using db2ckbkp

The db2ckbkp checks the integrity of the backup image and determines whether or not it can be restored. Some or all parts of the backup can be checked. The backup image must reside physically on the disk (see Example 4-3).
142
Example 4-3 Example db2chkbk

$ db2ckbkp TBC_MD.0.db2inst1.NODE0000.CATN0000.20010308132920.001 [1] Buffers processed: #### Image Verification Complete - successful. $
Verification using db2 list history

The DB2 UDB history file ( db2rhist.asc) contains logged events from different administrative events like backup, load, drop table, reorganize and so on. The success of backups can be verified by listing the backup information using the db2 list history command, as shown in Example 4-4.
Example 4-4 DB2 list history
$ db2 list history backup all for TBC_MD Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------B D 20010315070920001 F A S0000032.LOG S0000032.LOG ---------------------------------------------------------------------------Contains 2 tablespace(s): 00001 SYSCATSPACE 00002 USERSPACE1 ---------------------------------------------------------------------------Comment: DB2 BACKUP TBC_MD OFFLINE Start Time: 20010315070920 End Time: 20010315070953 ---------------------------------------------------------------------------00005 Location: adsm/libadsm.a
The relevant information is contained within the first line of each stanza. These lines show the following information: Operation (Op:B=backup) Object (Obj:D=Database) Type (Type:F=Offline,N=Online) Device (Dev:A=TSM) For a more detailed description of the list history command see the DB2 UDB Command Reference, SC09-2951.
143
Verification using db2adutl

In this section, we will focus on some options of the db2adutl command and how they may be used to verify the DB2 UDB backups. The QUERY option of the db2adutl utility can be used to get general information about which database backups, logfiles and load copies reside on the Tivoli Storage Manager server. An alternative test would be to only download one file from the Tivoli Storage Manager server and see if the whole file is readable. The db2adutl extract option must be used. The best way to verify the backup is the verify option of the db2adutl utility. It is basically a way to run the db2ckbkp functionality through Tivoli Storage Manager to verify if the image is going to be restorable. The whole image will be read from the Tivoli Storage Manager server into a local memory buffer (not the whole image at once but piece by piece). There is nothing written to local disk, but the actual data transfer over the local area network (LAN) should be considered. Issuing db2adutl verify will cause the utility to read through the image in the same way that restore does. It will verify that all of the required objects (media header, DB configuration, DB history, list of tablespace, tablespace definitions, and so on) exist in the object. The tool performs numerous checks but it will not, and cannot, actually check the integrity of any of the userdata that exists in the backup image.
Sample script for backup and verification

Example 4-5 is a sample script that can be used to back up and afterwards verify the backup image. This script must be adapted to fit the specific environment and is only an example of one way to do this.
Example 4-5 Sample backup and verification script
#!/bin/ksh LOG=/home/db2inst1/tsm/db2backup.log DB=TBC_MD # # First backup the database # db2 backup db $DB use tsm >> $LOG 2>&1 # # Verify if return code from backup is 0 # RC=$? if [ $RC != 0 ]
144
then echo "Backup was not successful" | mail db2inst1@brazil exit 1 fi # # Get the timestamp from backup out of logfile # TIMESTAMP=$(tail -3 $LOG | awk '{print $11}') # # Verify backup # db2adutl verify full taken at $TIMESTAMP db $DB without prompting >> $LOG 2>&1 # # Check if verification is ok # RC=$? if [ $RC != 0 ] then echo "Verify was not successful" | mail db2inst1@brazil exit 1 fi exit 0
145
146
Chapter 5.
Warehouse control database recovery scenarios

The warehouse control database contains metadata which represents all the information about the data warehouse and the population subsystem. Without accessing the warehouse control database, you cannot perform any of the data warehouse population activities. It is very important that you are able to recover your database when things go wrong. Problems may include power failures, application failures, as well as media and storage failures. To ensure that you can recover when problems like this happen, you need to have backups or copies of the entire warehouse control database or of the table spaces that make up the database. These backups can then be used following a database problem to restore the database. Backup of the warehouse control database, both development and production environment, should be done regularly. As more and more transformations are defined over time, the work done by development has to be protected from any corruption on the warehouse control database. Before building these recovery scenarios on the warehouse control database with DB2 UDB or Data Warehouse Center tools, some basic recommendations should apply when starting administering a DB2 Warehouse Manager environment.
147
5.1 Before starting to use DB2 Warehouse Manager

This section describes some basic recommendations on configuration when starting using DB2 Warehouse Manager to get an environment that provides a reasonably high service level and a relatively High Availability of the latest data.
5.1.1 Configure at least two warehouse servers

To prevent your production warehouse server from being impacted by the development work, you must set up a separate warehouse server dedicated to all the development work. Besides, when the production warehouse server is unavailable due to a severe hardware error, the machine used for development can take over the production workload.
5.1.2 Back up the warehouse control database

Backup of warehouse control database, both development and production environment, should be done regularly. As more and more transformations are defined over time, it gets more important to ensure the development work to be protected from any undesirable situation that causes the corruption of the warehouse control database.
5.1.3 Maintain multiple administrative clients

Consider keeping at least one more Data Warehouse Center readily available for use in case something happens to your primary Data Warehouse Center. The Warehouse Manager server can be connected to one or more DB2 UDB administrative clients. With the Data Warehouse Center tool (a component of DB2 UDB Control Center in the administrative client) the users are able to: Identify the source systems from which data will be extracted for inclusion in the warehouse. Identify the specific tables and columns that will serve as input to the warehouse. Specify data transformations to be performed on the extracted data. Specify which agent to perform a given extraction task. Specify any user-supplied programs that will be used to further manipulate the data prior to its being loaded into the warehouse. Schedule extraction, transformation and load jobs for execution at a specific time of the day.
148
Monitor the execution of agent jobs, ensuring successful completion, and allowing administrators to tune the warehouse build process over time using statistics collected as part of the execution process. Sometimes your primary Data Warehouse Center could fail and no longer function as the user interface to perform the tasks described above. Assuming that there is nothing wrong with your warehouse control database, you can resume your interrupted administrative task by quickly switching to another Data Warehouse Center.
5.1.4 Enable the warehouse control database for rollforward recovery

You can choose to use either circular logging or archive logging for your warehouse control database. But bearing in mind that updates can be made online via the DB2 Control Center graphical user interface, we strongly recommend that you enable rollforward recovery by setting up either the database configuration parameter logretain to RECOVERY or the userexit parameter to ON with an user exit properly compiled for your warehouse control database.
Note: If all the updates to DB2 Warehouse Manager are done in a standalone session in a special administration window, then this is a very controlled environment, and an offline backup could be used satisfactorily for the warehouse control database.
Example 5-1 shows how you can use the DB2 command line interface to issue the update command.
Example 5-1 Update database configuration parameters for rollforward recovery
C:\SQLLIB\BIN>db2 update db cfg for tbc_md using logretain recovery DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully. DB21026I For most configuration parameters, all applications must disconnect from this database before the changes become effective. C:\SQLLIB\BIN>db2 update db cfg for tbc_md using userexit on DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully. DB21026I For most configuration parameters, all applications must disconnect from this database before the changes become effective.
Chapter 5. Warehouse control database recovery scenarios
149
You can accomplish the same thing through the DB2 Control Center, as shown in Figure 5-1.
Attention: All applications need to disconnect from the database before the change becomes effective. After all applications are disconnected, the database is in backup pending state, and no new connections are allowed to the database until a full database backup is made.
Select the Control Database and click the Configure... database option and then click the Logs tab to display the Logs page.
Figure 5-1 Logretain database configuration parameter
5.1.5 Allocate enough logging space

The biggest problem we have seen in warehouse control database, especially in a migration situation, is that the database runs out of logging space and causes the migration to fail. Unfortunately once this happens the you are stuck since part of the metadata is migrated and parts are still at previous levels. We always recommend to start using 4096k log file size, 10 primary logs, and 100 secondary logs for the warehouse control database before the migration.
150
Example 5-2 shows how you can use the DB2 command line interface to issue the update command.
Example 5-2 Update database configuration parameter to increase logging space
C:\SQLLIB\BIN>db2 update db cfg for tbc_md using logfilsiz 4096 logprimary 10 logsecond 100 DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully. DB21026I For most configuration parameters, all applications must disconnect from this database before the changes become effective. C:\SQLLIB\BIN>
Or you can accomplish the same thing through the DB2 Control Center, as shown in Figure 5-1 on page 150.
5.2 Warehouse control database recovery overview

You can use either of two methods to back up and recover the warehouse control database: Use the standard DB2 UDB backup, restore and rollforward utility. To back up your warehouse control database, you can use the standard procedures for DB2 UDB backup and recovery. You can backup and restore the entire metadata in the warehouse control database quickly. With this method, you can not backup and recover a specific warehouse object individually. Export and import all of the data into and from tag language files. Metadata in the warehouse control database can be exported using a built-in export utility in Data Warehouse Center. The export utility enables administrator to select specific metadata to be exported. The output is a flat file in tag format. The tag file will be imported to the same or different warehouse control database via Data Warehouse Center import utility. The advantage of this method is that you can selectively backup and restore at the warehouse control database object level without having to restore the entire warehouse control database. This method is also useful as a tool to migrate the metadata defined in the development warehouse control database to the production warehouse control database.
151
5.3 Backing up the control database using DB2 UDB

In this section, we discuss how to back up the warehouse control database using the DB2 UDB utilities. You must back up your warehouse control database, as well as all your warehouse target databases, at regular intervals. We cover: Full offline backup Full online backup Incremental backup Tablespace backup The typical backup strategy consists of a combination of the backup types listed above. For example, you might want to take offline database backup periodically with incremental backups in between. There could be many different backup strategies that could fit your Warehouse environment. We also discuss how to verify backup image files and how to automate backup.
5.3.1 Full offline backup

The DB2 UDB backup utility can be used from any of the following: DB2 command line interface DB2 Control Center graphical user interface Your own C, COBOL, or Fortran language program Our examples use the DB2 command line and DB2 Control Center. To back up the warehouse control database in full offline backup mode, you must stop the Data Warehouse Center and the DB2 warehouse management services. To shut down the Data Warehouse Center: 1. On the workstation that contains the warehouse server, click Settings >Control Panel > Administrative Tools> Services. The Services window opens (Figure 5-2).
152
Figure 5-2 Services window
2. Select a warehouse service, and click Stop. 3. Repeat this step for each warehouse service that is listed in the Services window. The warehouse logger and server are linked; if you stop the warehouse logger, the server also stops. However, if you stop the warehouse server, the logger does not stop. When all warehouse services are stopped, click Close. Before you can start a backup or recovery operation, make sure that the database manager is online. In the ...\sqllib\bin directory is an executable db2start.exe. This executable will start the database manager, if it is not running; or if it is running, it will indicate this, as shown in Example 5-3.
Example 5-3 db2start.exe
C:\SQLLIB\BIN>db2start SQL1026N The database manager is already active.
Another way to check is to see if the corresponding Windows service is running. With this method the Windows OS command net start will list the services running (see Example 5-4). Our instance is named DB2, so the corresponding Windows service is DB2 - DB2.
153
Example 5-4 Net start command

C:\SQLLIB\BIN>net start These Windows 2000 services are started: COM+ Event System Computer Browser DB2 - DB2 DB2 - DB2CTLSV DB2 - DB2DAS00 DB2 JDBC Applet Server DB2 License Server The command completed successfully. C:\SQLLIB\BIN>
Offline backup using the DB2 command line

This example shows an offline backup using the DB2 command line interface. Log in to Windows with a user that has rights to run the DB2 Command Line Processor (CLP) and make sure that all applications are logged off from the warehouse control database you want to back up. Use the following DB2 UDB command to verify that all applications are logged off: db2 list applications for database tbc_md The result should appear as shown in Example 5-5.
Example 5-5 List applications command
C:\SQLLIB\BIN>db2 list applications for database tbc_md Auth Id Application Name Appl. Handle Application Id DB Name # of Agents
-------- -------------- ---------- ------------------------------ -------- ----KIMJAY db2bp.exe 3 *LOCAL.DB2.011210211529 TBC_MD 1
C:\SQLLIB\BIN>
In our example, a user is connected to our warehouse control database TBC_MD. You can use the following db2 force command (shown in Example 5-6) to force the user off: db2 force application ( 3 )
154
Example 5-6 Force application command

C:\SQLLIB\BIN>db2 force application ( 3 ) DB20000I The FORCE APPLICATION command completed successfully. DB21024I This command is asynchronous and may not be effective immediately. C:\SQLLIB\BIN>
If there are too many connections to the database, use the following command to force all applications off the instance: db2 force application all
Attention: if you have multiple databases, this command will force off all applications from all databases within the same instance:
Now you can start the backup operation from the DB2 command line: db2 backup database tbc_md to c:\db2backup The to c:\db2backup option tells DB2 UDB to write the output backup file to a directory c:\db2backup. Wait until a message like Example 5-7 appears.
Example 5-7 Backup command
C:\SQLLIB\BIN>db2 backup database tbc_md to c:\db2backup Backup successful. The timestamp for this backup image is : 20011210150125 C:\SQLLIB\BIN>
The backup operation has created an image file of your warehouse control database, TBC_MD.
Offline backup using the DB2 Control Center

This example shows an offline backup using the DB2 Control Center: 1. Start the DB2 Control Center. On a Windows system, you will find the DB2 Control Center under Start>Programs>IBM DB2. 2. Click down the tree until you reach the database folder. 3. Select the database you want to back up and right-click the database name and select Backup>Database, as shown in Figure 5-3.
155
Figure 5-3 Offline backup using DB2 Control Center
The Backup Database window appears (Figure 5-4).
Figure 5-4 Backup Database window in DB2 Control Center
156
Select Media Type: Directories or Tapes. Next, click Backup Now to execute the backup. If there is a problem starting the backup, an error message will appear. Depending on the system where the DB2 Control Center was started, the following windows may differ slightly (Figure 5-5).
Figure 5-5 DB2 start backup window
Note: The Close button must be selected to actually start the backup. If the window stays open, the backup will not start.
After the backup is finished, another window appears (Figure 5-6).
Figure 5-6 End of DB2 offline backup message
157
The DB2 Journal will provide additional information about the success of the backup, as shown in Figure 5-7. The DB2 Journal is in the Tools menu of the DB2 Control Center.
Figure 5-7 DB2 Journal shows the result of the backup
158
5.3.2 Full online backup

DB2 UDB online backup and recovery require the use of logfiles to enable rollforward recovery. Our example shows both DB2 command line and DB2 Control Center GUI interfaces.
Online backup using the DB2 command line

Start the backup operation from the DB2 command line: db2 backup database tbc_md online to c:\db2backup The to c:\db2backup option tells DB2 UDB to write the output backup file to local storage. Wait until a message, as shown in Example 5-8 appears.
Example 5-8 Backup online command
C:\SQLLIB\BIN>db2 backup db tbc_md online to c:\db2backup Backup successful. The timestamp for this backup image is : 20011210162138 C:\SQLLIB\BIN>
The backup operation created an image file of your warehouse control database, TBC_MD, in c:\db2backup directory.
Online backup using the DB2 Control Center

This example shows an online backup using the DB2 Control Center: 1. Start the DB2 Control Center. On a Windows system, you will find the DB2 Control Center under Start>Programs>IBM DB2. For UNIX systems, type db2cc. 2. Click down the tree until you reach the database folder.
159
3. Select the database you want to back up and right-click the database name and select Backup>Database, as shown in Figure 5-8.
Figure 5-8 Online backup using DB2 Control Center
160
The Backup Database window appears (Figure 5-9). Select Type: Directories or Tapes.
Figure 5-9 Backup database online in DB2 Control Center
Click the Options tab and select Online, as shown in Figure 5-10.
Figure 5-10 Select Online in Backup Database Options window
161
Then click Backup Now to execute the backup. If there is a problem starting the backup, an error message will be generated. Depending on the system where the DB2 Control Center was started, the following windows may differ slightly (Figure 5-11).
The Close button must be clicked to actually start the backup. If the window stays open, the backup will not start. After the backup is finished, another window appears (Figure 5-12).
Figure 5-12 End of DB2 online backup message
162
The DB2 Journal will provide additional information about the success of the backup, as shown in Figure 5-13.
Figure 5-13 DB2 Journal shows the result of the backup
5.3.3 Tablespace backup

Tablespace backup is faster than full database backup. For this reason, instead of taking periodic database backups, you might want to take frequent tablespace backups of the tablespace where your warehouse control database tables are located, especially when you do not want to lose any of your development work. Tablespace backups can be done either offline or online. The only difference is the option on the backup command. In this chapter, we only discuss the online mode. For the offline mode, remove the online option from the command or select the Offline mode in the GUI. Remember when using the online option that the warehouse control database must be enabled for roll-forward recovery.
163
Tablespace backup using the DB2 command line

Start the backup operation from the DB2 command line: backup db tbc_md tablespace ( userspace1 ) online to c:\db2backup Wait until a message like Example 5-9 appears:
Example 5-9 Tablespace backup using the DB2 command line window
C:\SQLLIB\BIN>db2 backup db tbc_md tablespace userspace1 online to c:\db2backup Backup successful. The timestamp for this backup image is : 20011211135646 C:\SQLLIB\BIN>
The backup operation created an backup image file of the userspace1 tablespace in your warehouse control database and put it in c:\db2backup directory.
Tablespace backup using the DB2 Control Center

This example shows an online backup using the DB2 Control Center: 1. Start the DB2 Control Center. On a Windows system, you will find the DB2 Control Center under Start>Programs>IBM DB2. 2. Click down the tree until you reach the database folder.
164
3. Select the folder of your warehouse control database, then click the tablespace on the expanded tree in the left frame, as shown in Figure 5-14. It displays all the tablespaces that belong to your warehouse control database in the right pane. Right-click the tablespace you want to back up.
Figure 5-14 Select tablespace backup menu
165
The Backup Database window appears (Figure 5-15). Select Media Type: Directories or Tapes and the directory where you want to create the backup image file.
Figure 5-15 Select directories or tapes on the backup table space window
Click the Options tab and select the Online radio button, as shown in Figure 5-16.
Figure 5-16 Select Offline or Online in options window
166
Then click Backup Now to execute the backup. If there is a problem starting, the backup an error message will be generated. Depending on the system where the DB2 Control Center was started, the following windows may differ slightly (Figure 5-17).
You need to click the Close button to actually start the backup. If you leave the window open, the backup will not start. After the backup is finished, another window appears (Figure 5-18).
Figure 5-18 End of tablespace backup message
167
The DB2 Journal will provide additional information about the success of the backup, as shown in Figure 5-19. The DB2 Journal is in the Tools menu of the DB2 Control Center.
Figure 5-19 DB2 Journal shows the result of the tablespace backup
5.3.4 Incremental backup

The larger your warehouse control database gets, the larger the database backup image gets as well. Every time you take a full database backup, you tie up the resources like network or disk I/O required to perform the backup for the duration of its execution. Also even when only a few percent of data in the database has changed, the whole database will be backed up again when doing a full database backup. To overcome this kind of problem, you can use incremental backups. There are three types of backups:
Full: This is a classical full database backup. Also known as level 0 backup. Incremental: This is a database backup that contains only the changes since the last full (level 0) database backup. Incremental backups are also known as level 1 backups. Delta: This is a database backup that contains only the changes since the last backup of every type (full, incremental or delta).
168
To enable the use of incremental backups, the DB2 database configuration parameter TRACKMOD must be set to on, as shown in Example 5-10. To activate this parameter, a db2stop and db2start may be necessary.
Example 5-10 Setting up TRACKMOD parameter ON using update command
C:\SQLLIB\BIN>db2 update db cfg for tbc_md using TRACKMOD ON DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully. DB21026I For most configuration parameters, all applications must disconnect from this database before the changes become effective. C:\SQLLIB\BIN>
You have to take a full database backup before you start a chain of backups. This can be an offline or online backup, as shown in Example 5-11.
Example 5-11 A full backup before the first incremental backup
C:\SQLLIB\BIN>db2 backup db tbc_md online to c:\db2backup Backup successful. The timestamp for this backup image is : 20011211152529 C:\SQLLIB\BIN>
Now we are able to generate incremental or delta backups. To start an incremental backup, the incremental option of the backup command must be specified, as shown in Example 5-12.
Example 5-12 Incremental backup
C:\SQLLIB\BIN>db2 backup db tbc_md online incremental Backup successful. The timestamp for this backup image is : 20011211153137 C:\SQLLIB\BIN>
5.3.5 Verifying DB2 UDB backup image files

There is no way to restore an unsuccessful, incomplete or missing backup. For this reason, you want to test the integrity of a backup image and to determine whether or not the image can be restored. The best way to see if a backup can be used for restore is to actually restore the backup. In this section, we introduce some other alternatives available for you.
Verification using db2ckbkp command

You can use db2ckbkp to check the integrity of the backup image and determine whether or not it can be restored. Some or all parts of the backups can be checked. The backup image must reside physically on the disk.
169
In Example 5-13, we ran the db2ckbkp against the two backup images. The second backup image file does not seem to be useable for recovery.
Example 5-13 Checking backup image files using db2ckbckp
C:\SQLLIB\BIN>db2ckbkp c:\db2backup\TBC_MD.0\DB2\NODE0000\CATN0000\20011211\1525 29.001 c:\db2backup\TBC_MD.0\DB2\NODE0000\CATN0000\20011210\162138.001 [1] Buffers processed: ##### [2] Buffers processed: ##### ERROR! More than one BACKUP.START.RECORD.MARKER! ERROR! More than one LIST.OF.DATA.POOLS! ERROR! More than one history file! ERROR! More than one DATA.POOL.TABLE! ERROR! More than one DB configuration! ERROR! More than one BUFFER.POOL.FILE! ERROR! More than one LOG.FILE.HEADER! ERROR! More than one backup tail! Image Verification Complete - ERRORS DETECTED: 8 C:\SQLLIB\BIN>
Verification using the db2 list history command

The DB2 history file (db2rhist.asc) contains logged events from different administrative events like backup, load, drop table, reorganize, and so on. You can verify the success of backups by listing the backup information using the db2 list history command. In Example 5-14, you see only one entry which shows there is a tablespace backup image taken online at 2001-12-11-13:56:46.
Example 5-14 The db2 list history backup command
C:\SQLLIB\BIN>db2 list history backup since 20011211 for tbc_md List History File for tbc_md Number of matching file entries = 4 Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------B P 20011211135646001 N D S0000018.LOG S0000019.LOG ---------------------------------------------------------------------------Contains 1 tablespace(s): 00001 USERSPACE1 ---------------------------------------------------------------------------Comment: DB2 BACKUP TBC_MD ONLINE Start Time: 20011211135646 End Time: 20011211135716 ---------------------------------------------------------------------------00001 Location: c:\db2backup\TBC_MD.3\DB2\NODE0000\CATN0000\20011211 C:\SQLLIB\BIN>
170
The interesting information is contained within the first line of each stanza. These lines show the Operation (Op:B=backup); the Object (Obj:P=Tablespace); the Type (Type:F=Offline,N=Online); and the Device (Dev:D=Directory). For an in-depth description of the db2 list history command, see the DB2 UDB Command Reference, SC09-2951.
Verification using db2adutl command

If you created your warehouse control database using Tivoli Storage Manager (TSM), you can use the db2adutl command to verify the backups. The query option of the db2adutl utility can be used to get general information about which database backups, logfiles, and loadcopies reside on the Tivoli Storage Manager server (see Example 5-15).
Example 5-15 db2adutl query
C:\SQLLIB\bin>db2adutl query Query for database TBC_MD Retrieving full database backup information. full database backup image: 1, Time: 20010328113323 Oldest log: S0000000.LOG, Node: 0, Sessions used: 2 Retrieving tablespace backup information. No tablespace backup images found for TBC_MD Retrieving load copy information. No load copy images found for TBC_MD Retrieving log archive information. Log file: S0000000.LOG, Node: 0, Taken at: 2001-03-28-13.29.31
To do a test, you would download the backup from the Tivoli Storage Manager server first and then would run db2adutl with extract option to see if the whole file is readable (see Example 5-16). However, this is not a very practical method.
Example 5-16 db2adutl extract
C:\SQLLIB\bin>db2adutl extract Query for database TBC_MD Retrieving full database backup information. Please wait. full database backup image: .\TBC_MD.0\DB2\NODE0000\CATN0000\20010328\113323.001, Node: 0 .\TBC_MD.0\DB2\NODE0000\CATN0000\20010328\113323.002, Node: 0 Do you wish to extract ALL of these images (Y/N)? y
171
Writing to file: .\TBC_MD.0\DB2\NODE0000\CATN0000\20010328\113323.001 Writing to file: .\TBC_MD.0\DB2\NODE0000\CATN0000\20010328\113323.002
The best way to verify the backup is to use the verify option of the db2adutl utility. It is basically a way to run the db2ckbkp functionality through Tivoli Storage Manager to verify if the image is restorable. The whole image will be read from the Tivoli Storage Manager server into local memory buffer, not to local disk, piece by piece instead of the whole image at once. db2adutl verify attempts to read through the image in the same way that a true restore does. It will verify that all of the required objects (such as, media header, DB configuration, DB history, list of tablespace, tablespace definitions, and so on) exist in the object (see Example 5-17). The tool performs numerous checks, but it will not and cannot actually check the integrity of the userdata that exists in the backup image.
Example 5-17 db2adutl verify
C:\SQLLIB\bin>db2adutl verify full taken at 20010328113323 db tbc_md Query for database TBC_MD Retrieving full database backup information. Please wait. full database backup image: .\TBC_MD.0\DB2\NODE0000\CATN0000\20010328\113323.001, Node: 0 .\TBC_MD.0\DB2\NODE0000\CATN0000\20010328\113323.002, Node: 0 Do you wish to verify this image (Y/N)? y Verifying file: .\TBC_MD.0\DB2\NODE0000\CATN0000\20010328\113323.001 ================== BUFFER 0 ================== ***BUFFERS 1 - 41 deleted to save space in this redbook.*** ================== BUFFER 42 ================== Read 0 bytes, assuming we are at the end of the image Image Verification Complete - successful. .\TBC_MD.0\DB2\NODE0000\CATN0000\20010328\113323.002 ================== BUFFER 0 ================== ***BUFFERS 1-42 deleted to save space in this redbook.***
172
================== BUFFER 43 ================== WARNING only partial image read, bytes read: 28672 of 4194304 Read 0 bytes, assuming we are at the end of the image Image Verification Complete - successful.
The warning message shown in Example 5-17 is misleading and can be ignored. The last line is the important one and it indicates that the verification was successful.
5.3.6 Automating DB2 UDB backup

In this section, we describe ways to automate the backup using Windows, DB2 UDB and Tivoli Storage Manager utilities. The examples shown here include some simple command files written to automate parts of the backup process. We do not include in the scripts any error checking or automated recovery because those process logics heavily depend on the configuration of your databases and how you want to structure your backup strategy.
Automation using Windows

You can use the Windows utility AT to automate the backup of your warehouse control database. The Windows AT utility runs shell commands at specified dates and times. You can type at /? to view Help on the command. Additional Help with examples for this command can be found by searching Start>Help from Windows. We created a script file called bckpcdb.cmd in c:\scripts folder, as shown in Example 5-18. The script bckpcdb.cmd executes a DB2 UDB online backup on the warehouse control database. You must specify the /c option when invoking these scripts. Otherwise, the db2cmd keeps running in the background.
Example 5-18 backpcdb.cmd
set DB2INSTANCE=DB2 "C:\SQLLIB\bin\db2cmd.exe" /c DB2.EXE backup db tbc_md user db2admin using itsosj online to c:\db2backup
173
Example 5-19 shows how to schedule backups to run at every 8:00 AM in the morning using AT command.
Example 5-19 Scheduling backup using AT command
C:\SQLLIB\BIN>at 08:00 /every:M,T,W,Th,F,S,Su c:\scripts\bckpcdb.cmd Added a new job with job ID = 1 C:\SQLLIB\BIN>
If you type AT without specifying any parameters, you see the current scheduled jobs, as shown in Example 5-20.
Example 5-20 Displaying backup schedule
C:\SQLLIB\BIN>at Status ID Day Time Command Line ------------------------------------------------------------------------------1 Each M T W Th F S Su 8:00 AM c:\scripts\bckpcdb.cmd C:\SQLLIB\BIN>
Automation using DB2 UDB

This section covers how to automate DB2 UDB backups using the DB2 Journal, the DB2 Scheduler, and the DB2 Script Center. You can access the DB2 Journal, the DB2 Scheduler, or the DB2 Script Center from the DB2 Control Center.
174
Open the DB2 Control Center and click down the tree until reaching the database folder. Select the database to be backed up and right-click the database name and select Backup and Database, as shown in Figure 5-20.
Figure 5-20 Select backup database in the DB2 Control Center
175
Choose whether you want to do an online or offline, full or tablespace backup. After you make your selection, click the Schedule button, as shown in Figure 5-21.
Figure 5-21 DB2 UDB backup window
The DB2 UDB schedule window opens, as shown in Figure 5-22.
Figure 5-22 DB2 UDB schedule window
176
This window is very useful. Click the desired actions when the backup should take place and for how long, and the backup should be started automatically. The userid and password are required and the userid must have enough privileges to do a backup. When you click the OK button, the DB2 UDB message window shown in Figure 5-23 confirms that the job has been scheduled.
Figure 5-23 DB2 UDB message window
177
To monitor the success of scheduled backups the DB2 Journal can be used. The DB2 Journal is located under the Tools Menu on the DB2 Control Center. Once started a window will open like the one in Figure 5-24.
Figure 5-24 DB2 journal
The system name must be selected first. By clicking the corresponding button, information about the jobs that are already finished (Job History), running (Running Jobs) or are expected to run (Pending Jobs) will appear. In order to read all the information, resize the column by moving the border of the column description. To manage a job, the job must be highlighted. The possible options will be selectable either from the Jobs menu or by right-clicking on the highlighted job. Sometimes, you need to customize the DB2 commands or run a series of commands as a script or batch files. You can use the DB2 Script Center to create a script. This is located under the Tools menu of the DB2 Control Center. Figure 5-25 shows the DB2 Script Center, and the scripts available.
178
Figure 5-25 DB2 script center
You can create a new script from the Scripts>New menu. Figure 5-26 shows a backup script using the buffers and parallelism option.
Figure 5-26 New command script window
179
To schedule the script, highlight the entry in the Script Center, and right-click to display the drop-down menu, as shown in Figure 5-25 on page 179. You can then select Schedule to schedule the script.
5.4 Recovering the control database using DB2 UDB

In this section, we look at different methods of recovering the warehouse control database. Depending on the situation, you can use one of the following recovery methods: Restore recovery Rollforward recovery Point-in-time recovery Tablespace redirected restore Dropped table recovery Restarting restore and rollforward Restore without rolling forward We also discuss how to restore a warehouse control database to a new DB2 UDB instance. The DB2 recovery history file plays a critical role in every recovery situation. Sometimes you might find the recovery history file itself damaged. We show how to recover the history file as well.
5.4.1 Restore recovery

Sometimes this type of recovery is referred to as a version recovery. You have to use only this recovery method when your warehouse control database is not enabled for rollforward recovery. Restore recovery restores a previous version of the database using an image created by a backup. No log files are applied, and any transactions committed after the backup are lost. All users must disconnect from the database for restore recovery. Any changes in the warehouse control database since the last backup will be lost and this is why we strongly recommend enabling the rollforward recovery with the warehouse control database by turning the archive logging on. This method to recover the warehouse control database should be used exceptionally when a full recovery on the warehouse control database is not possible.
180
Restore recovery scenario

Your warehouse control database does not do archive logging. A container C:\SW-N100\TBC_MD\DATASPACE1-1.DAT for a tablespace DATASPACE1 was accidently deleted. DB2 UDB reports the error on its diagnostic file db2diag.log, as shown in Example 5-21.
Example 5-21 Container error in db2diag.log
2001-12-12-16.11.23.712000 Instance:DB2 Node:000 PID:1456(db2syscs.exe) TID:2076 Appid:*LOCAL.DB2.011213001106 buffer_pool_services sqlbDMSDoContainerOp Probe:810 Database:TBC_MD DIA9999E An internal error occurred. Report the following error code : "0xFFFFC11E". 2001-12-12-16.11.23.722000 Instance:DB2 Node:000 PID:1456(db2syscs.exe) TID:2076 Appid:*LOCAL.DB2.011213001106 buffer_pool_services sqlbDMSDoContainerOp Probe:810 Database:TBC_MD Error checking container 0 (C:\SW-N100\TBC_MD\DATASPACE1-1.DAT) for tbsp 4. Rc = FFFFE60A 2001-12-12-16.11.23.722001 Instance:DB2 Node:000 PID:1456(db2syscs.exe) TID:2076 Appid:*LOCAL.DB2.011213001106 buffer_pool_services sqlbDMSStartPool Probe:800 Database:TBC_MD DIA9999E An internal error occurred. Report the following error code : "0xFFFFC11E". 2001-12-12-16.11.23.732000 Instance:DB2 Node:000 PID:1456(db2syscs.exe) TID:2076 Appid:*LOCAL.DB2.011213001106 buffer_pool_services sqlbDMSStartPool Probe:800 Database:TBC_MD Tablespace 4 (DATASPACE1)
2001-12-12-16.11.23.742000 Instance:DB2 Node:000 PID:1456(db2syscs.exe) TID:2076 Appid:*LOCAL.DB2.011213001106 buffer_pool_services sqlbStartPools Probe:30 Database:TBC_MD Tablespace DATASPACE1 (4) is in state x4000. Starting it failed. rc=1ec1 ffff
Restore recovery using the DB2 command line

In order for you to recover the database: 1. Restoring a database would require you to select an image taken from a backup, using a timestamp to identify an image. you use the list history command to identify which backup image to use.
181
The command lists entries in the history file, which contain recovery events, such as backups, restore and roll-forward. It is also contains entries of additional events, such as load, run statistics, drop table, reorganize table, and alter tablespace. Use the list history command to select a backup image you want to use for the restore, as shown in Example 5-22, which also shows the results. There can be several entries, so you need to scroll to select the backup image you need. You need the timestamp portion of the Timestamp+Sequence field or you can use the value of the Start Time field.
Example 5-22 The list history command results
C:\SQLLIB\BIN>db2 list history backup all for tbc_md List History File for tbc_md Number of matching file entries = 2 Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------B D 20011211153137001 O D S0000024.LOG S0000025.LOG ---------------------------------------------------------------------------Contains 4 tablespace(s): 00001 SYSCATSPACE 00002 USERSPACE1 00003 FLG32K 00004 DATASPACE1 ---------------------------------------------------------------------------Comment: DB2 BACKUP TBC_MD ONLINE Start Time: 20011211153137 End Time: 20011211153147 ---------------------------------------------------------------------------00001 Location: C:\SQLLIB\BIN\TBC_MD.0\DB2\NODE0000\CATN0000\20011211 Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------B D 20011212134004001 N D S0000026.LOG S0000027.LOG ---------------------------------------------------------------------------Contains 4 tablespace(s): 00001 SYSCATSPACE 00002 USERSPACE1 00003 FLG32K 00004 DATASPACE1 ---------------------------------------------------------------------------Comment: DB2 BACKUP TBC_MD ONLINE Start Time: 20011212134004 End Time: 20011212134046 ---------------------------------------------------------------------------00002 Location: c:\db2backup\TBC_MD.0\DB2\NODE0000\CATN0000\20011212 C:\SQLLIB\BIN>
182
2. Use the restore command (see Example 5-23), specifying the timestamp of the image you want to restore, and then reply y to continue:
Example 5-23 Restoring the warehouse control database
C:\SQLLIB\BIN>db2 restore db tbc_md from c:\db2backup taken at 20011212134004 SQL2539W Warning! Restoring to an existing database that is the same as the backup image database. The database files will be deleted. Do you want to continue ? (y/n) y DB20000I The RESTORE DATABASE command completed successfully. C:\SQLLIB\BIN>
Restore recovery using the DB2 Control Center

Alternatively you can use the DB2 Control Center to restore the warehouse control database. Follow the steps described below: 1. Right-click the database, and then from the drop-down menu, select Restore>Database, as shown in Figure 5-27.
Figure 5-27 Restore recovery using the DB2 Control Center
183
2. When the database image selection page appears, as shown in Figure 5-28, select the database image you want to use for the restore, then select OK.
Figure 5-28 Backup image selection window
The dialog box in Figure 5-29 appears to tell you that the restore job has been scheduled. Click the Close button to launch the execution of the job.
Figure 5-29 DB2 message - Restore job created
184
3. When the restore successfully completes, a DB2 message window appears, as shown in Figure 5-30, confirming that the restore was successful. Click Close.
Figure 5-30 DB2 message window - successful completion
5.4.2 Rollforward recovery

If you enable your warehouse control database to use the archive logging method, you can do rollforward recovery up to the point of failure without losing any transactions. You call this type of database a recoverable database. With recoverable databases, you have the option of recovering the whole database or only the tablespaces that need recovery. You can recover tablespaces either offline or online. You can use an image taken from either a previous tablespace backup or a database backup. A tablespace rollforward recovery takes less time than a database recovery. For that reason, you may want to attempt to recover your warehouse control database by a tablespace rollforward method instead of a database rollforward recovery as long as you can isolate the damaged area limited to a tablespace.
Rollforward recovery scenario

Users complain that they cannot access a tablespace when querying a table. You did a list tablespaces, as shown in Example 5-24, and discovered that one of the tablespaces is offline. Then looking at the db2diag.log, you see an error when checking a container, and the tablespace 4 (DATASPACE1) could not be started. The db2diag.log would be similar to the one shown in Example 5-21 on page 181.
185
Example 5-24 List tablespaces to check the state

C:\SQLLIB\BIN>db2 list tablespaces Tablespaces for Current Database Tablespace ID = 0 Name = SYSCATSPACE Type = System managed space Contents = Any data State = 0x0000 Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Offline C:\SQLLIB\BIN> = = = = = 1 TEMPSPACE1 System managed space System Temporary data 0x0000
= = = = =
2 USERSPACE1 System managed space Any data 0x0000
= = = = =
3 FLG32K System managed space Any data 0x0000
= = = = =
4 DATASPACE1 Database managed space Any data 0x4000
In this case, you only need to recover the tablespace DATASPACE1 because all other tablespaces are in normal state.
186
Tablespace rollforward using DB2 command line

The steps to recover are as follows: 1. Use the list history command to determine which backup image is to be restored. There can be a few entries, as shown in Example 5-25, so you need to scroll to select the backup image you need. You need the timestamp portion of the Timestamp+Sequence field or you can use the value in the Start Time field.
Example 5-25 The list history backup command
C:\SQLLIB\BIN>db2 list history backup all for tbc_md List History File for tbc_md Number of matching file entries = 4 Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------B D 20011211153137001 O D S0000024.LOG S0000025.LOG ---------------------------------------------------------------------------Contains 4 tablespace(s): 00001 SYSCATSPACE 00002 USERSPACE1 00003 FLG32K 00004 DATASPACE1 ---------------------------------------------------------------------------Comment: DB2 BACKUP TBC_MD ONLINE Start Time: 20011211153137 End Time: 20011211153147 ---------------------------------------------------------------------------00001 Location: C:\SQLLIB\BIN\TBC_MD.0\DB2\NODE0000\CATN0000\20011211 Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------B D 20011212134004001 N D S0000026.LOG S0000027.LOG ---------------------------------------------------------------------------Contains 4 tablespace(s): 00001 SYSCATSPACE 00002 USERSPACE1 00003 FLG32K 00004 DATASPACE1 ---------------------------------------------------------------------------Comment: DB2 BACKUP TBC_MD ONLINE Start Time: 20011212134004 End Time: 20011212134046 ---------------------------------------------------------------------------00002 Location: c:\db2backup\TBC_MD.0\DB2\NODE0000\CATN0000\20011212
187
Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------R D 20011212162233001 F 20011212134004 ---------------------------------------------------------------------------Contains 4 tablespace(s): 00001 SYSCATSPACE 00002 USERSPACE1 00003 FLG32K 00004 DATASPACE1 ---------------------------------------------------------------------------Comment: RESTORE TBC_MD WITH RF Start Time: 20011212162233 End Time: 20011212162309 ---------------------------------------------------------------------------00003 Location: Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------B P 20011213131258001 F D S0000028.LOG S0000028.LOG ---------------------------------------------------------------------------Contains 1 tablespace(s): 00001 DATASPACE1 ---------------------------------------------------------------------------Comment: DB2 BACKUP TBC_MD OFFLINE Start Time: 20011213131258 End Time: 20011213131321 ---------------------------------------------------------------------------00004 Location: C:\db2backup\TBC_MD.3\DB2\NODE0000\CATN0000\20011213
In this example, there are three database backups and one tablespace backup image. You are going to use only the tablespace backup image to recover the tablespace DATASPACE1. 2. Issue the restore command, as shown in Example 5-26, specifying the timestamp of the image file that you selected.
Example 5-26 Restore tablespace command
C:\SQLLIB\BIN>db2 restore db tbc_md tablespace ( dataspace1 ) online from c:\db2backup taken at 20011213131258 DB20000I The RESTORE DATABASE command completed successfully. C:\SQLLIB\BIN>
3. Now you issue a list tablespaces command again to check the state of DATASPACE1 and find it in rollforward pending state, as shown in Example 5-27.
188
Example 5-27 List tablespaces command to check the state again

C:\SQLLIB\BIN>db2 list tablespaces Tablespaces for Current Database Tablespace ID = 0 Name = SYSCATSPACE Type = System managed space Contents = Any data State = 0x0000 Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Roll forward pending = = = = = 1 TEMPSPACE1 System managed space System Temporary data 0x0000
= = = = =
2 USERSPACE1 System managed space Any data 0x0000
= = = = =
3 FLG32K System managed space Any data 0x0000
= = = = =
4 DATASPACE1 Database managed space Any data 0x0080
C:\SQLLIB\BIN>
4. Use the rollforward command shown in Example 5-28 to apply the logs to completion.
189
Example 5-28 Rollforward command

C:\SQLLIB\BIN>db2 rollforward db tbc_md to end of logs and stop tablespace (dataspace1) online Rollforward Status Input database alias = tbc_md Number of nodes have returned status = 1 Node number = 0 Rollforward status = not pending Next log file to be read = Log files processed = Last committed transaction = 2001-12-12-21.40.45.000000 DB20000I The ROLLFORWARD command completed successfully. C:\SQLLIB\BIN>
As you can see in Example 5-29, the tablespace DATASPACE1 is now recovered and in normal state.
Example 5-29 List tablespaces command again
C:\SQLLIB\BIN>db2 list tablespaces Tablespaces for Current Database Tablespace ID = 0 Name = SYSCATSPACE Type = System managed space Contents = Any data State = 0x0000 Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Normal Tablespace ID Name Type Contents State = = = = = 1 TEMPSPACE1 System managed space System Temporary data 0x0000
= 2 = USERSPACE1 = System managed space = Any data = 0x0000
= 3 = FLG32K = System managed space = Any data = 0x0000
190
Detailed explanation: Normal Tablespace ID Name Type Contents State Detailed explanation: Normal C:\SQLLIB\BIN> = 4 = DATASPACE1 = Database managed space = Any data = 0x0000
Tablespace rollforward using the DB2 Control Center

This time we show you how to do a tablespace rollforward recovery using the DB2 Control Center. You can also use the DB2 Control Center to do a database rollforward recovery too. 1. First you check the state of each tablespaces by clicking on Table Spaces under your warehouse control database, as shown in Figure 5-31. You find that the state of tablespace DATASPACE1 is offline.
Figure 5-31 Tablespaces window
191
2. Right-click the database, and then from the drop-down menu select Restore>Database, as shown in Figure 5-32.
Figure 5-32 Restore using Database drop-down menu
3. When the backup image selection window appears, as shown in Figure 5-33, select a backup image in the list. You choose the backup image for the tablespace DATASPACE1.
192
Figure 5-33 Backup image selection
4. Next go on to the Table Spaces tab. You get the warning message shown Figure 5-34. You can ignore it and click Close.
Figure 5-34 DB2 message warning
193
5. Check the Use selected table spaces only. And then select DATASPACE1 and click the > button to move to Use table spaces panel (see Figure 5-35).
Figure 5-35 Table Spaces page
6. Then move on to the Roll-forward tab. Check the Roll-forward (reapply the logs) and the Roll-forward to the end of the logs option, and uncheck the Leave in roll-forward pending state option, as shown in Figure 5-36.
Figure 5-36 Roll-forward page
7. Go on to the Options tab, as shown in Figure 5-37, and select Online for an online restore. Select OK.
194
Figure 5-37 Options page
You are presented with the DB2 message window, as shown in Figure 5-38. Click Close to launch the execution of the job.
Figure 5-38 DB2 message - Restore job created
8. A DB2 message box appears, as shown in Figure 5-39, to notify you that the rollforward recovery was successful. Click Close.
Figure 5-39 DB2 message - Successful completion
195
9. Go back to the DB2 Control Center and click the Table Spaces under your warehouse control database. Refresh it and make sure that the tablespace DATASPACE1 is in normal state.
Figure 5-40 Table Spaces state
5.4.3 Point-in-time recovery

Sometimes you might run into a situation where you want to roll back transactions that you inadvertently committed. For example, you deleted a Data Warehouse object such as a Process and soon realized that you should not have deleted it. However, DB2 UDB rollback removes only uncommitted transactions in the database. Once the transaction is committed, rollback is no longer possible. To remove or re-instate unwanted data changes that are mostly due to user errors, you have to do point-in-time recovery. Point-in-time recovery can also be used if the archive logs are unusable, and roll forwarding produces errors when reading the archive logs.
196
Point-in-time recovery considerations

During an online backup of a database, logs are still being updated when updating transactions are active. When recovering using such online backups, there is a minimum recovery time that must be specified. This is usually the time when the online backup completed. For tablespace point-in-time recovery, there are some more considerations: You cannot do point-in-time recovery on the tablespace containing the system catalogs. If a table spans more than one tablespace, for example, data and indexes are in separate tablespaces, you must roll forward those tablespaces to the same point-in-time. Tables in a tablespace being rolled forward to a point-in-time might have referential integrity constraints in tables in other tablespaces. The tables having such relationships will be in check pending state. You must resolve the inconsistencies caused by the point-in-time recovery by using the set constraints command. If tables in tablespaces being rolled forward to a point-in-time execute triggers that insert, update, or delete rows from tables in other tablespaces, rows from tables from tablespaces not being rolled forward may become inconsistent with the tables in tablespaces being rolled forward. You must correct the inconsistencies after the point-in-time recovery, or you can include the related tablespaces in the point-in-time recovery. Because all data must correlate with data in the system catalogs, there may be a minimum point-in-time to recover for a tablespace. For example, when a table is created, the system catalogs are updated to reflect this. You cannot do point-in-time recovery prior to the table creation, because the system catalogs have information for this table and is not removed by the tablespace point-in-time recovery process. This also applies to dropping tables. You cannot recover a dropped table unless you specify the DROP TABLE RECOVERY option turned on for the tablespace where the table resides. To determine the minimum recovery time for a tablespace to be recovered, use the db2 list tablespaces show detail command. After the tablespace is rolled forward to a point-in-time, it will be in backup pending state. You must back up the tablespace.
197
Point-in-time recovery scenario

You accidently deleted a data warehouse process and soon realized that you did not mean it. Now you want to get it back. In this case you really have no choice other than performing a point-in-time recovery to a time just before the deletion took place. You would need to recover only a tablespace USERSPACE1 where you know all the warehouse control database tables reside.
Point-in-time recovery using the DB2 command line

Follow the steps described below: 1. Determine the minimum recovery time for USERSPACE1 by using the db2 list tablespaces show detail command, as shown in Example 5-30.
Example 5-30 List tablespaces show detail results
C:\SQLLIB\BIN>db2 list tablespaces show detail Tablespaces for Current Database Tablespace ID Name Type Contents State Detailed explanation: Normal Total pages Useable pages Used pages Free pages High water mark (pages) Page size (bytes) Extent size (pages) Prefetch size (pages) Number of containers Tablespace ID Name Type Contents State Detailed explanation: Normal Total pages Useable pages Used pages Free pages High water mark (pages) Page size (bytes) Extent size (pages) = = = = = 0 SYSCATSPACE System managed space Any data 0x0000
= = = = = = = = = = = = = =
2518 2518 2518 Not applicable Not applicable 4096 32 16 1 1 TEMPSPACE1 System managed space System Temporary data 0x0000
= = = = = = =
1 1 1 Not applicable Not applicable 4096 32
198
Prefetch size (pages) Number of containers Tablespace ID Name Type Contents State Detailed explanation: Normal Total pages Useable pages Used pages Free pages High water mark (pages) Page size (bytes) Extent size (pages) Prefetch size (pages) Number of containers Tablespace ID Name Type Contents State Detailed explanation: Normal Total pages Useable pages Used pages Free pages High water mark (pages) Page size (bytes) Extent size (pages) Prefetch size (pages) Number of containers Minimum recovery time Tablespace ID Name Type Contents State Detailed explanation: Normal Total pages Useable pages Used pages Free pages
= 16 = 1 = = = = = 2 USERSPACE1 System managed space Any data 0x0000
= = = = = = = = = = = = = =
538 538 538 Not applicable Not applicable 4096 32 16 1 3 FLG32K System managed space Any data 0x0000
= = = = = = = = = = = = = = =
29 29 29 Not applicable Not applicable 4096 32 16 1 2001-12-12-20.40.19.000000 4 DATASPACE1 Database managed space Any data 0x0000
= = = =
512 480 160 320
199
High water mark (pages) Page size (bytes) Extent size (pages) Prefetch size (pages) Number of containers Minimum recovery time C:\SQLLIB\BIN>
= = = = = =
160 4096 32 16 1 2001-12-07-22.12.54.000000
Note that the minimum recovery time of USERSPACE1 is 2001-12-12-20.40.19.000000. the timestamp is in Coordinated Universal Time (CUT) format. In this example, CUT is eight hours ahead of the local time. This means that you need to pick a time after 2001-12-12-12.40.19.000000 in local time. 2. Use the db2 list history command (see Example 5-31) to select a backup image that you want to use for the restore.
Example 5-31 The list history command results
C:\SQLLIB\BIN>db2 list history backup all for tbc_md List History File for tbc_md Number of matching file entries = 6 Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------B D 20011211153137001 O D S0000024.LOG S0000025.LOG ---------------------------------------------------------------------------Contains 4 tablespace(s): 00001 SYSCATSPACE 00002 USERSPACE1 00003 FLG32K 00004 DATASPACE1 ---------------------------------------------------------------------------Comment: DB2 BACKUP TBC_MD ONLINE Start Time: 20011211153137 End Time: 20011211153147 ---------------------------------------------------------------------------00001 Location: C:\SQLLIB\BIN\TBC_MD.0\DB2\NODE0000\CATN0000\20011211 Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------B D 20011212134004001 N D S0000026.LOG S0000027.LOG ---------------------------------------------------------------------------Contains 4 tablespace(s): 00001 SYSCATSPACE 00002 USERSPACE1 00003 FLG32K 00004 DATASPACE1 ----------------------------------------------------------------------------
200
Comment: DB2 BACKUP TBC_MD ONLINE Start Time: 20011212134004 End Time: 20011212134046 ---------------------------------------------------------------------------00002 Location: c:\db2backup\TBC_MD.0\DB2\NODE0000\CATN0000\20011212 Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------R D 20011212162233001 F 20011212134004 ---------------------------------------------------------------------------Contains 4 tablespace(s): 00001 SYSCATSPACE 00002 USERSPACE1 00003 FLG32K 00004 DATASPACE1 ---------------------------------------------------------------------------Comment: RESTORE TBC_MD WITH RF Start Time: 20011212162233 End Time: 20011212162309 ---------------------------------------------------------------------------00003 Location: Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------B P 20011213131258001 F D S0000028.LOG S0000028.LOG ---------------------------------------------------------------------------Contains 1 tablespace(s): 00001 DATASPACE1 ---------------------------------------------------------------------------Comment: DB2 BACKUP TBC_MD OFFLINE Start Time: 20011213131258 End Time: 20011213131321 ---------------------------------------------------------------------------00004 Location: C:\db2backup\TBC_MD.3\DB2\NODE0000\CATN0000\20011213 Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------B P 20011213131258001 F D S0000028.LOG S0000028.LOG ---------------------------------------------------------------------------Contains 1 tablespace(s): 00001 DATASPACE1 ---------------------------------------------------------------------------Comment: DB2 BACKUP TBC_MD OFFLINE Start Time: 20011213131258 End Time: 20011213131321 ---------------------------------------------------------------------------00005 Location: C:\db2backup\TBC_MD.3\DB2\NODE0000\CATN0000\20011213 Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID
201
-- --- ------------------ ---- --- ------------ ------------ -------------B D 20011213193727001 F D S0000030.LOG S0000030.LOG ---------------------------------------------------------------------------Contains 4 tablespace(s): 00001 SYSCATSPACE 00002 USERSPACE1 00003 FLG32K 00004 DATASPACE1 ---------------------------------------------------------------------------Comment: DB2 BACKUP TBC_MD OFFLINE Start Time: 20011213193727 End Time: 20011213193810 ---------------------------------------------------------------------------00006 Location: C:\db2backup\TBC_MD.0\DB2\NODE0000\CATN0000\20011213 C:\SQLLIB\BIN>
3. Issue the db2 restore command, specifying the timestamp of the backup image, as shown in Example 5-32.
Example 5-32 Restore using the last backup image
C:\SQLLIB\BIN>db2 restore db tbc_md tablespace (userspace1) online from c:\db2backup taken at 20011213193727 DB20000I The RESTORE DATABASE command completed successfully. C:\SQLLIB\BIN>
4. Use the db2 rollforward command to apply the logs. Let us assume that you deleted the Data Warehouse Process object before 2001-12-13-19.41.00 and you decided to select 2001-12-13-19.40.00 in local time as the recovery time. Specify the point-in-time in CUT format, and it is past the minimum recovery time, as shown in Example 5-33.
Example 5-33 Rollforward to a point-in-time past the minimum recovery time
C:\SQLLIB\BIN>db2 rollforward db tbc_md to 2001-12-14-03.40.00 tablespace (users pace1) online Rollforward Status Input database alias Number of nodes have returned status Node number Rollforward status Next log file to be read Log files processed Last committed transaction = tbc_md = 1 = 0 = TBS working = = = 2001-12-12-21.40.45.000000
DB20000I The ROLLFORWARD command completed successfully. C:\SQLLIB\BIN>
202
5. Notice that you did not do a rollforward stop on the previous command. So issue another rollforward command specifying stop, as shown in Example 5-34.
Example 5-34 Rollforward stop command
C:\SQLLIB\BIN>db2 rollforward db tbc_md stop tablespace (userspace1) online Rollforward Status Input database alias Number of nodes have returned status Node number Rollforward status Next log file to be read Log files processed Last committed transaction = tbc_md = 1 = = = = = 0 not pending S0000032.LOG 2001-12-12-21.40.45.000000
DB20000I The ROLLFORWARD command completed successfully. C:\SQLLIB\BIN>
The USERSPACE1 is in backup pending state, as shown in Example 5-35.

Example 5-35 Tablespace in backup pending state
C:\SQLLIB\BIN>db2 list tablespaces Tablespaces for Current Database Tablespace ID Name Type Contents State Detailed explanation: Backup pending C:\SQLLIB\BIN> = = = = = . 2 USERSPACE1 System managed space Any data 0x0020
6. Finally, take a backup of the USERSPACE1 to clear the backup pending state.
Example 5-36 Tablespace backup
C:\SQLLIB\BIN>db2 backup db tbc_md tablespace (userspace1) online to c:\db2backup Backup successful. The timestamp for this backup image is : 20011213202234 C:\SQLLIB\BIN>
203
Point-in-time recovery using the DB2 Control Center

The following steps show how to recover USERSPACE1 tablespace to a point-in-time using the DB2 Control Center: 1. Right-click the database, and then from the drop-down menu, select Restore>Database, as shown in Figure 5-41.
Figure 5-41 Tablespace point-in-time in the DB2 Control Center
204
2. When the backup image selection window appears, as shown in Figure 5-42, select a backup image in the list. This can be a backup image or a tablespace image.
205
3. If you select a database backup image, on the Table Spaces tab, you can select which tablespaces you want to restore, as shown in Figure 5-43. In this example, all the warehouse control database tables reside in USERSPACE1.
Figure 5-43 Table Spaces window
4. On the Roll-forward tab, as shown in Figure 5-44, check the Roll-forward (re-apply logs) option, select Roll-forward to a point-in-time, enter the date and time in the Roll-forward to transaction fields, and uncheck the Leave in roll-forward pending state option.
Figure 5-44 Rollforward options
206
5. Go to the Options tab, as shown in Figure 5-45, and select Online for an online restore. Select OK. After completion, a DB2 message box is presented to tell you whether or not the rollforward is successful.
Figure 5-45 Options window
6. Finally take a backup of the database or the tablespace to get it out of the backup pending state.
5.4.4 Tablespace redirected recovery

You can use tablespace redirected restore when tablespace containers cannot be restored to their original location for some reason. This may be due to disk or hardware errors. Redirected restore can also be used if you want to reorganize the layout of the containers in the tablespace where you can add, change, remove containers. You can move containers to another device, for example, to distribute I/O. Since ALTER TABLESPACE does not allow you to add containers once an SMS tablespace is created, you can add containers by using redirected restore.
Tablespace redirected recovery scenario

You have over-allocated the space for the DATASPACE1 tablespace, and you want to resize the container to free up space, so it can be used by other objects in the database.
207
Tablespace redirected recovery using the DB2 command line

These are the steps to follow to perform a redirected recovery for DATASPACE1: 1. Determine the tablespace ID of DATASPACE1 by the list tablespaces command. As shown in Example 5-37, the tablespace ID is 4.
Example 5-37 List tablespaces to determine the tablespace ID
C:\SQLLIB\BIN>db2 list tablespaces Tablespaces for Current Database Tablespace ID Name Type Contents State Detailed explanation: Normal C:\SQLLIB\BIN> = = = = = 4 DATASPACE1 Database managed space Any data 0x0000
2. Using the tablespace ID, list the containers for DATASPACE1, as shown in Example 5-38.
Example 5-38 List containers for DATASPACE1
C:\SQLLIB\BIN>db2 list tablespace containers for 4 show detail Tablespace Containers for Tablespace 4 Container ID = 0 Name = C:\SW-N100\TBC_MD\DATASPACE1-1.DAT Type = File Total pages = 512 Useable pages = 480 Accessible = Yes Container ID = 1 Name = C:\SW-N100\TBC_MD\DATASPACE1-2.DAT Type = File Total pages = 5120 Useable pages = 5088 Accessible = Yes C:\SQLLIB\BIN>
You see two containers assigned to the DATASPACE1. You decide that the second container is too big and want to reduce the size of containers to 1280 pages in total. 3. Use the restore command with the redirect option, specifying the DATASPACE1 to be restored and the timestamp of the image you want to use for the restore.
208
Example 5-39 Restore redirect command

C:\SQLLIB\BIN>db2 restore db tbc_md tablespace (dataspace1) online from c:\db2backup taken at 20011213193727 redirect SQL1277N Restore has detected that one or more table space containers are inaccessible, or has set their state to 'storage must be defined'. DB20000I The RESTORE DATABASE command completed successfully. C:\SQLLIB\BIN>
4. Issue the command set tablespace containers to resize the container for tablespace ID 4, as shown in Example 5-40. The command ignore rollforward container operations means that the ALTER TABLESPACE operations in the logs will be ignored during the roll-forward.
Example 5-40 Set tablespace containers command
C:\SQLLIB\BIN>db2 set tablespace containers for 4 ignore rollforward container operations using (file 'C:\SW-N100\TBC_MD\dataspace1-3.dat' 1280) DB20000I The SET TABLESPACE CONTAINERS command completed successfully. C:\SQLLIB\BIN>
5. Continue the restore with the continue option, as shown in Example 5-41.
Example 5-41 Restore continue command
C:\SQLLIB\BIN>db2 restore db tbc_md continue DB20000I The RESTORE DATABASE command completed successfully. C:\SQLLIB\BIN>
6. Do a rollforward for the tablespace, as shown in Example 5-42.

Example 5-42 Rollforward to end of logs and stop
C:\SQLLIB\BIN>db2 rollforward db tbc_md to end of logs and stop tablespace (dataspace1) online Rollforward Status Input database alias Number of nodes have returned status = tbc_md = 1
Node number = 0 Rollforward status = not pending Next log file to be read = S0000034.LOG Log files processed = Last committed transaction = 2001-12-14-04.40.51.000000 DB20000I The ROLLFORWARD command completed successfully. C:\SQLLIB\BIN>
209
7. The following list tablespace containers command confirms that the resizing of DATASPACE1 to 1280 pages was successful completed.
Example 5-43 List tablespace containers for DATASPACE1
C:\SQLLIB\BIN>db2 list tablespace containers for 4 show detail Tablespace Containers for Tablespace 4 Container ID Name Type Total pages Useable pages Accessible C:\SQLLIB\BIN> = = = = = = 0 C:\SW-N100\TBC_MD\dataspace1-3.dat File 1280 1248 Yes
Tablespace redirected recovery using DB2 Control Center

Follow these steps to do a tablespace redirected recovery using the DB2 Control Center: 1. Right-click the database, and then from the drop-down menu, select Restore>Database as shown in Figure 5-46.
210
Figure 5-46 Tablespace redirected restore in the DB2 Control Center
211
2. When the backup image selection window appears, as shown in Figure 5-47, select the database image you want to use for the restore, and then click the Table Spaces tab.
3. On the Table Spaces window, as shown in Figure 5-48, click the Use selected tablespaces only radio button, select DATASPACE1, click the > button, and then select the Containers tab.
Figure 5-48 Table Spaces selection
212
4. On the Containers page, when you check the Redirect table space containers, and it displays the list of containers for the tablespace, as shown in Figure 5-49, highlight the container on the lower frame, and then click the Change button.
Figure 5-49 Container selection
213
5. A Change Container window appears, as shown in Figure 5-50. Change the file size, and then select OK.
Figure 5-50 Change container
6. You are now back to the Container page, as shown previously in Figure 5-49 on page 213. Select the Roll-forward tab. Check the Roll-forward (re-apply logs) option, uncheck the Leave in roll-forward pending state option, as shown in Figure 5-51. And then select the Options tab.
214
Figure 5-51 Rollforward option
7. On the Options window, as shown in Figure 5-52, select Online if you want to do recovery online. Select OK when done.
Figure 5-52 Options selection
8. A DB2 message window appears to tell you that the tablespace redirected restore is successful.
215
5.4.5 Dropped table recovery

You may accidently drop a table whose data you still need. To prevent such accidents, you should consider making your warehouse control database tables recoverable following a drop table operation. You could recover the table data by invoking a database restore operation, followed by a database rollforward operation to a point-in-time before the table was dropped. This may be time consuming if the database is large, and your data will be unavailable during recovery. DB2s dropped table recovery feature lets you recover your dropped table data using table space-level restore and rollforward operations. This will be faster than database-level recovery, and your database will remain available to users. For a dropped table to be recoverable, the table space in which the table resides must have the DROPPED TABLE RECOVERY option turned on. You can do this during table space creation, or by invoking the ALTER TABLESPACE statement, as shown in Example 5-44. The DROPPED TABLE RECOVERY option is table space specific and limited to regular table spaces.
Example 5-44 Alter tablespace with dropped table recover option
C:\SQLLIB\BIN>db2 alter tablespace userspace1 dropped table recovery on DB20000I The SQL command completed successfully. C:\SQLLIB\BIN>
To determine if a table space is enabled for dropped table recovery, you can query the DROP_RECOVERY column in the SYSCAT.TABLESPACES catalog table, as shown in Example 5-45.
Example 5-45 Checking to see if the dropped table recovery is enabled
C:\SQLLIB\BIN>db2 select tbspace, drop_recovery from syscat.tablespaces where tbspace='USERSPACE1' TBSPACE DROP_RECOVERY ------------------ ------------USERSPACE1 Y 1 record(s) selected. C:\SQLLIB\BIN>
When a DROP TABLE statement is run against a table whose table space is enabled for dropped table recovery, an additional entry (identifying the dropped table) is made in the log files. An entry is also made in the recovery history file, containing information that can be used to recreate the table. (See Figure 5-53.)
216
Figure 5-53 Additional information written in Log and Recovery History FIle
Only one dropped table can be recovered at a time. You can recover a dropped table by doing the following: 1. Identify the dropped table by invoking the db2 list history dropped table command (see Example 5-46). The dropped table ID is listed in the Backup ID column.
Example 5-46 List history dropped table
C:\SQLLIB\BIN>db2 list history dropped table all for tbc_md List History File for tbc_md Number of matching file entries = 1 Op Obj Timestamp+Sequence Type Dev Earliest Log Current Log Backup ID -- --- ------------------ ---- --- ------------ ------------ -------------D T 20011214133215 00000000000079180002002d ---------------------------------------------------------------------------"IWH "."DATARESOURCE" resides in 1 tablespace(s): 00001 USERSPACE1 ---------------------------------------------------------------------------Comment: DROP TABLE Start Time: 20011214133215 End Time: 20011214133215 ---------------------------------------------------------------------------00001 DDL: CREATE TABLE "IWH "."DATARESOURCE" ( "IWHID" CHAR(10) NOT NULL , "UPDA TED_BY" CHAR(128) NOT NULL , "UPDATE_TS" TIMESTAMP NOT NULL , "NAME" VARCHAR(80) NOT NULL , "SHORT_DESCRIPTION" VARCHAR(254) , "IRNAME" VARCHAR(80) NOT NULL , " FILETYPE" VARCHAR(80) NOT NULL , "KEYLOC" INTEGER NOT NULL , "KEYLEN" INTEGER NO T NULL , "RECLEN" INTEGER NOT NULL , "PCBIND" INTEGER NOT NULL , "MAXSEGLEN" INT EGER NOT NULL , "BLOCKSIZE" INTEGER NOT NULL , "CREATED" TIMESTAMP , "TYPEOFCREA TION" VARCHAR(90) NOT NULL , "PASSWORD" VARCHAR(8) NOT NULL , "DB2TABLENAMESPACE " VARCHAR(90) NOT NULL , "BINARYFLAG" SMALLINT , "KSDSFLAG" SMALLINT , "FIRSTLIN ENAMES" SMALLINT , "CHARDELM" VARCHAR(1) NOT NULL , "VIEWFLAG" SMALLINT , "ALIAS FLAG" SMALLINT , "LARGESCHEMANAME" VARCHAR(128) NOT NULL , "LARGETABLENAME" VARC HAR(255) NOT NULL , "PERSISTENTFLAG" SMALLINT , "MAX_EDITIONS" INTEGER , "GRANTT OPUBLIC" SMALLINT , "LASTCOLSEQNMCREATE" INTEGER , "LASTCOLSEQNMALTER" INTEGER , "CREATETARTAB" SMALLINT , "PRIMARYKEYNEW" SMALLINT , "AUTOGENCREATESTMT" SMALLI NT , "TABLECREATED" SMALLINT , "DIMENSION_TABLE" SMALLINT , "FACT_TABLE" SMALLIN T , "CDTABLENAME" VARCHAR(128) , "CDTABLESCHEMA" VARCHAR(128) , "BEFORE_IMG_PREF IX" VARCHAR(4) , "REPL_ENABLED" SMALLINT , "DB2TABLENAMEINDEX" VARCHAR(90) NOT N
217
ULL , "INPARTTBSP" SMALLINT , "DBNAME390" VARCHAR(8) , "HIGHESTSTATUS" INTEGER N OT NULL , "DATAIMPORTFLAG" SMALLINT , "DATAIMPORTID" INTEGER , "SITEID" INTEGER ) IN "USERSPACE1" ; ---------------------------------------------------------------------------C:\SQLLIB\BIN>
2. Restore a database or table space level backup image taken before the table was dropped, as shown in Example 5-47.
Example 5-47 Restore from backup taken before the drop
C:\SQLLIB\BIN>db2 restore db tbc_md tablespace (userspace1) online from c:\db2backup taken at 20011213202234 DB20000I The RESTORE DATABASE command completed successfully. C:\SQLLIB\BIN>
3. Create an export directory to which files containing the table data are to be written. This directory must either be accessible to all database partitions, or exist on each partition. Subdirectories under this export directory are created automatically by each database partition. These subdirectories are named NODE nnnn, where nnnn represents the database partition or node number. Data files containing the dropped table data as it existed on each database partition are exported to a lower subdirectory called data. For example, \export_directory \NODE0000 \data. As shown In Example 5-48, we use an existing directory c:\db2backup. 4. Roll forward to a point-in-time after the table was dropped, using the RECOVER DROPPED TABLE option on the db2 rollforward command. Alternatively, roll forward to the end of the logs, so that updates to other tables in the table space or database are not lost.
Example 5-48 Rollforward with Recover Dropped Table option
C:\SQLLIB\BIN>db2 rollforward db tbc_md to end of logs and stop tablespace (userspace1) online recover dropped table 00000000000079180002002d to c:\db2backup Rollforward Status Input database alias = tbc_md Number of nodes have returned status = 1 Node number = 0 Rollforward status = not pending Next log file to be read = S0000036.LOG Log files processed = Last committed transaction = 2001-12-14-21.32.16.000000 DB20000I The ROLLFORWARD command completed successfully. C:\SQLLIB\BIN>
218
5. Recreate the table using the CREATE TABLE statement from the recovery history file (see Example 5-49).
Example 5-49 Redefine the table
C:\SQLLIB\BIN>db2 CREATE TABLE "IWH "."DATARESOURCE" ( "IWHID" CHAR(10) NOT NULL , "UPDATED_BY" CHAR(128) NOT NULL , "UPDATE_TS" TIMESTAMP NOT NULL , "NAME" VARCHAR(80) NOT NULL , "SHORT_DESCRIPTION" VARCHAR(254) , "IRNAME" VARCHAR(80) NOT NULL , "FILETYPE" VARCHAR(80) NOT NULL , "KEYLOC" INTEGER NOT NULL , "KEYLE N" INTEGER NOT NULL , "RECLEN" INTEGER NOT NULL , "PCBIND" INTEGER NOT NULL , "M AXSEGLEN" INTEGER NOT NULL , "BLOCKSIZE" INTEGER NOT NULL , "CREATED" TIMESTAMP , "TYPEOFCREATION" VARCHAR(90) NOT NULL , "PASSWORD" VARCHAR(8) NOT NULL , "DB2T ABLENAMESPACE" VARCHAR(90) NOT NULL , "BINARYFLAG" SMALLINT , "KSDSFLAG" SMALLIN T , "FIRSTLINENAMES" SMALLINT , "CHARDELM" VARCHAR(1) NOT NULL , "VIEWFLAG" SMAL LINT , "ALIASFLAG" SMALLINT , "LARGESCHEMANAME" VARCHAR(128) NOT NULL , "LARGETA BLENAME" VARCHAR(255) NOT NULL , "PERSISTENTFLAG" SMALLINT , "MAX_EDITIONS" INTE GER , "GRANTTOPUBLIC" SMALLINT , "LASTCOLSEQNMCREATE" INTEGER , "LASTCOLSEQNMALT ER" INTEGER , "CREATETARTAB" SMALLINT , "PRIMARYKEYNEW" SMALLINT , "AUTOGENCREAT ESTMT" SMALLINT , "TABLECREATED" SMALLINT , "DIMENSION_TABLE" SMALLINT , "FACT_T ABLE" SMALLINT , "CDTABLENAME" VARCHAR(128) , "CDTABLESCHEMA" VARCHAR(128) , "BE FORE_IMG_PREFIX" VARCHAR(4) , "REPL_ENABLED" SMALLINT , "DB2TABLENAMEINDEX" VARC HAR(90) NOT NULL , "INPARTTBSP" SMALLINT , "DBNAME390" VARCHAR(8) , "HIGHESTSTAT US" INTEGER NOT NULL , "DATAIMPORTFLAG" SMALLINT , "DATAIMPORTID" INTEGER , "SIT EID" INTEGER) IN "USERSPACE1" DB20000I The SQL command completed successfully. C:\SQLLIB\BIN>
6. Import the table data that was exported during the rollforward operation into the table. There are some restrictions on the type of data that is recoverable from a dropped table. It is not possible to recover: Large object (LOB) or long field data. The DROPPED TABLE RECOVERY option is not supported for long table spaces. If you attempt to recover a dropped table that contains LOB or LONG VARCHAR columns, these columns will be set to NULL in the generated export file. The DROPPED TABLE RECOVERY option can only be used for regular table spaces, not for temporary or long table spaces. The metadata associated with row types. (The data is recovered, but not the metadata.) The data in the hierarchy table of the typed table will be recovered. This data may contain more information than appeared in the typed table that was dropped.
5.4.6 Restoring your control database into a new DB2 UDB instance
Sometimes you might want to create another warehouse control database for testing purposes. Or you might want to have a standby warehouse control database which serves as a quick backup, in case the original warehouse control database gets corrupted to the extent that the recovery is expected to take quite a long time.
219
In this section, we discuss how to restore your warehouse control database that was backed up on another instance as a new database. The target instance can be on the same system or on another system. These are the steps to follow: 1. Create new target DB2 UDB database. The target database must be created first. Any containers and required disk space for the database must also be available before the restore. If the logfiles are also to be restored, then the new target database name must be the same as that of the original database. Otherwise, the newly created database name can be different. Use a CREATE DATABASE statement, as shown in Example 5-50, to create a database.
Example 5-50 Create a database
C:\SQLLIB\BIN>db2 create database tbc_md2 DB20000I The CREATE DATABASE command completed successfully. C:\SQLLIB\BIN>
2. Update the new database configuration. Use the db2 update database configuration command to change the parameter values from the default to the copying warehouse control database values. 3. Restore the database. Select the database to restore, and use a restore command to restore the database. In our example, we restore a database image from database TBC_MD to database TBC_MD2, as shown in Example 5-51.
Example 5-51 Restore into a new database
C:\SQLLIB\BIN>db2 restore db tbc_md from c:\db2backup taken at 20011213193727 into tbc_md2 SQL2529W Warning! Restoring to an existing database that is different from the backup image database, and the alias name "TBC_MD2" of the existing database does not match the alias name "TBC_MD" of the backup image, and the database name "TBC_MD2" of the existing database does not match the database name "TBC_MD" of the backup image. The target database will be overwritten by the backup version. The Roll-forward recovery logs associated with the target database will be deleted. Do you want to continue ? (y/n) y DB20000I The RESTORE DATABASE command completed successfully.
220
4. Do a rollforward with stop option, as shown in Example 5-52.

Example 5-52 Rollforward stop for a new database
C:\SQLLIB\BIN>db2 rollforward db tbc_md2 stop Rollforward Status Input database alias = tbc_md2 Number of nodes have returned status = 1 Node number = 0 Rollforward status = not pending Next log file to be read = Log files processed = Last committed transaction = 2001-12-12-21.40.45.000000 DB20000I The ROLLFORWARD command completed successfully. C:\SQLLIB\BIN>
Now you have a new warehouse control database created by restoring a backup image file taken from your existing warehouse control database.
5.4.7 Recovering the recovery history file

The recovery history file is used to get information about database backup image you can use to do recovery. The DB2 Control Center also uses this file on the Backup Image page, as shown in Figure 5-33 on page 193, when giving the user the option to select a backup image. The history file also contains information about load operations and backups and restore operations. The history file is always backed up during a database or tablespace backup. For this reason, you can recover the history file from a backup image if the file db2rhist.asc has been corrupted or the entries were accidentally deleted using the db2 prune command. To recover the history file, issue a recover command, as shown in Example 5-53.
Example 5-53 Recover history file command
C:\SQLLIB\BIN>db2 restore db tbc_md history file online from c:\db2backup taken at 20011213193727 DB20000I The RESTORE DATABASE command completed successfully. C:\SQLLIB\BIN>
221
5.4.8 Restarting restore and rollforward

During a restore or rollforward operation, you may find that you want to use a different backup image and decide to cancel the current restore or rollforward operation and start again. Suppose you initiated a restore of the backup image taken at 20011213202234, as shown in Example 5-54.
Example 5-54 Restore USERSPACE1 with a backup image
C:\SQLLIB\BIN>db2 restore db tbc_md tablespace (userspace1) online from c:\db2backup taken at 20011213202234 DB20000I The RESTORE DATABASE command completed successfully.
Before you finish up the restore by running a rollforward, you realized that you should have used the latest backup image which was taken at 20011217101537. In order for you to be able to restart the restore with the latest backup image, you first have to cancel the current restore, as shown in Example 5-55.
Example 5-55 Rollforward cancel to use different backup image
C:\SQLLIB\BIN>db2 rollforward db tbc_md cancel tablespace (userspace1) SQL4975W Roll-forward operation was cancelled successfully. The database or selected table spaces have to be restored on node(s) "0".
The tablespace that was previously restored will be placed in a restore pending state, as shown in Example 5-56.
Example 5-56 USERSPACE1 is in restore pending
C:\SQLLIB\BIN>db2 list tablespaces Tablespaces for Current Database Tablespace ID Name Type Contents State Detailed explanation: Restore pending = = = = = 2 USERSPACE1 System managed space Any data 0x0100
You can now restart the restoring the tablespace USERSPACE1 by using the backup image taken at 20011217101537 (see Example 5-57), so that the tablespace that was previously in a restore pending state will be in a rollforward pending state.
222
Example 5-57 Restore USERSPACE1 with a different backup image

C:\SQLLIB\BIN>db2 restore db tbc_md tablespace (userspace1) online from c:\db2backup taken at 20011217101537 DB20000I The RESTORE DATABASE command completed successfully.
Run a rollforward, as shown in Example 5-58, to apply logs and finish up the recovery of the tablespace.
Example 5-58 Rollforward to end of logs
C:\SQLLIB\BIN>db2 rollforward db tbc_md to end of logs and stop tablespace (userspace1) Rollforward Status Input database alias = tbc_md Number of nodes have returned status = 1 Node number = 0 Rollforward status = not pending Next log file to be read = S0000040.LOG Log files processed = Last committed transaction = 2001-12-17-18.16.02.000000 DB20000I The ROLLFORWARD command completed successfully.
5.4.9 Restore without rolling forward

If the database is rollforward enabled, a restore will always put the database or tablespaces in a rollforward pending state. Depending on your situation in a point-in-time recovery, you may only want to do a restore and stop there without applying the logs. You must consider the minimum recovery time, as discussed in Point-in-time recovery on page 196. You can use the restore command with the option, without rolling forward, to accomplish this, as shown in Example 5-59.
Example 5-59 Restore without rolling forward
C:\SQLLIB\BIN>db2 restore db tbc_md from c:\db2backup taken at 20011217133440 without rolling forward SQL2539W Warning! Restoring to an existing database that is the same as the backup image database. The database files will be deleted. Do you want to continue ? (y/n) y DB20000I The RESTORE DATABASE command completed successfully.
The database will not be in a rollforward pending state most of the time if you use this option. You can query the rollforward state, as shown in Example 5-60.
223
Example 5-60 Rollforward query status

C:\SQLLIB\BIN>db2 rollforward db tbc_md query status Rollforward Status Input database alias = tbc_md Number of nodes have returned status = 1 Node number = 0 Rollforward status = not pending Next log file to be read = Log files processed = Last committed transaction = 2001-12-17-18.16.02.000000
This does not work all the time. Sometimes, DB2 UDB still puts it in rollforward pending state. To remove the rollforward pending state, stop the rollforward, as shown in Example 5-61.
Example 5-61 Rollforward with stop
C:\SQLLIB\BIN>db2 rollforward db tbc_md stop Rollforward Status Input database alias = tbc_md Number of nodes have returned status = 1 Node number = 0 Rollforward status = not pending Next log file to be read = Log files processed = Last committed transaction = 2001-12-17-18.16.02.000000 DB20000I The ROLLFORWARD command completed successfully.
5.5 Backup and recovery using Data Warehouse Center

In most cases, you can recover your warehouse control database using standard DB2 UDB backup, restore, and rollforward utility. And it is faster. But sometimes it is handy to use Data Warehouse Center built-in export and import utilities when you have to selectively recover some of the Warehouse objects. For example, you can re-create a Warehouse Process individually without restoring the entire warehouse control database.
5.5.1 Exporting and importing Data Warehouse Center metadata

This method is also useful when you migrate the metadata defined in the development warehouse control database to the production warehouse control database.
224
Export processes and import processes use a large amount of system resources. You might want to limit the use of other programs while you are exporting and importing object definitions. Because the import and export formats are release-dependent, you cannot use exported files from a previous release to migrate from one release of the Data Warehouse Center to another. If you want to migrate the Data Warehouse Center, see the DB2 Universal Database Quick Beginnings for your operating system. Alternatively, if you want to make a copy of your warehouse control database (such as when you want separate test and production systems), you can use the DB2 UDB export and load utilities to copy the data. However, both databases must have a user ID and password that matches the default warehouse user, so that users can log on to the Data Warehouse Center.
Exporting metadata to another Data Warehouse Center

When you export metadata to a tag language file, the Data Warehouse Center finds the objects that you want to export and produces tag language statements to represent the objects. It then places the tag language statements into files that you can import into another Data Warehouse Center. If you export a process, you might export a large volume of metadata. The following objects are exported with a process: All steps contained within a process. You cannot export individual steps. The agent site for the process. Any user-defined programs that are associated with steps in the process. Internal objects that are required by the Data Warehouse Center, such as column maps for SQL steps. Schedule information for processes and steps. You can specify not to export this information. Any warehouse source or warehouse target definitions that are used by the process and underlying steps. You can choose not to export the warehouse source table definitions. Any other processes and steps whose output is used by the process that is being exported. All table definitions in a selected warehouse source or warehouse targets, if you export a warehouse source or warehouse target. All program definitions within a program group, if you export a program group. All table definitions within a warehouse schema, if you export a warehouse schema.
225
Importing metadata
You can import object definitions for use in your Data Warehouse Center system. When you import a tag language file, the metadata that it contains is stored in the warehouse control database. If you are using the import utility to establish a new Data Warehouse Center, you must initialize a new warehouse control database in the target system. After you complete this task, you can import as many tag language files as you want. After you complete the export process, and a tag language file is created, the file is ready to import to a target system. Before you the import the file, consider the following issues:
Naming considerations: If you are using the import utility to move a warehouse source from a test environment to a production environment, make sure that the production environment does not already have a warehouse source with the same warehouse source name unless you want to over write the definition of the warehouse source.
If you import a step into a system that contains a step with the same name, then you must either delete the step that you want to overwrite, or change the step to development mode. Otherwise, the step cannot be updated, and an error will occur.
Ordering the import of objects: You use a logical order to import objects. An object that is referred to but not defined in a tag language file must be defined in the destination warehouse control database. You can do this by first importing the object definition that is referred to in a separate tag language file, or you can define the object in the Data Warehouse Center tree view.
For example, you can import warehouse source definitions first. When the warehouse source definitions are ready for use, you can import subject areas, warehouse target definitions, and any corresponding steps individually.
Other considerations: If you move a tag language file from one system to another, you must move all the associated files along with it and they must reside in the same directory.
Do not use the import function to migrate from a previous version of the Data Warehouse Center. You must always import and export tag language files using the same version of the Data Warehouse Center.
226
How to run the metadata export utility faster

When you export metadata, the export utility retrieves all the metadata needed to define the objects you select and all the metadata needed to create the relationships for those objects. Therefore, many objects are exported in addition to those you select. This process ensures that the objects are valid. All of this metadata is exported to a tag language file. Most steps are related not only to input and output tables and column mapping objects, but also to other steps (either through task flow relationships or data dependency relationships). These steps and their related objects are also exported to a tag language file, even if they belong to a different process. The export utility maintains a list of processed objects which it uses to prevent any object from being exported more than once. As the number of objects on this list increases, performance is impacted. Here are some recommendations to improve performance: Use more than one tag language file. Tag language files smaller than 5 MB are recommended. Tag language files larger than 10 MB are prohibitive. Leave the export dependent source properties box in the Export Metadata window unchecked. Use this option only if you are creating a tag language file in order to update metadata and the definition of your warehouse sources is unchanged. This will decrease the size of your tag language file. Run export in a maintenance window when no other applications are using the machine. Stop the warehouse server and warehouse logger services and run the export utility from the DB2 command line. For more information about this, on the DB2 command line, enter: iwh2exp2 Increase the virtual memory on the machine where the export is running. A minimum of 200 MB to 300 MB is recommended. Use the RUNSTATS, REORG TABLE, and REORGCHK commands against your warehouse control database and the tables within it.
How to run the metadata import utility faster

The import utility re-creates all of the objects that are exported to the selected tag language file in a new warehouse control database. This process is database intensive, requiring large numbers of queries and updates against the warehouse control database. Since tag language represents each object and relationship independently, each metadata object and relationship is added or updated using a separate operation on the database. This has a large impact on performance.
227
In addition, the import utility maintains a list of processed objects that it uses to retrieve objects when processing relationships. As the number of objects on this list increases, performance is impacted. Here are some recommendations to improve performance: Use multiple small tag language files instead of one large file. Tag language files smaller than 5 MB are recommended. Tag language files larger than 10 MB are prohibitive. Run import in a maintenance window when no other applications are using the machine. Stop the warehouse server and warehouse logger services and run the import utility from the DB2 command line. For more information about this, on the DB2 command line, enter: iwh2imp2. Increase the virtual memory on the machine where the import is running. A minimum of 200 MB to 300 MB is recommended. Make sure that the application heapsize parameter (APPLHEAP) for the warehouse control database is set at 4096 as a minimum. Use the RUNSTATS, REORG TABLE, and REORGCHK commands against your warehouse control database and the tables within it.
228
Chapter 6.
Problem determination methodology

In order to help DB2 Warehouse Manager administrators to know what to do when a problem happens, this chapter develops a problem determination methodology. When using DB2 Warehouse Manager, two major failure cases may occur: Hardware failures (facilities, systems, disks) DB2 Warehouse Manager component failures When hardware failures occur, we will apply the High Availability techniques. Which technique to use with DB2 Warehouse Manager hardware failures has already been described in the following sections: When hardware and system failures occur, see 4.2, Clustering and failover support on page 99. When disk failures occur, see 4.3, High Availability of media on page 111. However, in this chapter, we focus on DB2 Warehouse Manager component failures and describe a problem determination methodology, going from identifying the problem to solving it or to being able to restart or recover the environment. We give some keys and recommendations to restart or recover the DB2 warehouse application or environment.
229
6.1 A methodology in three phases

To solve any DB2 Warehouse Manager components problem or failure, we describe three main phases:
Problem determination phase:

First, you discover that something is not going the way it should. This is the problem determination phase. In this phase, you gather information about the external and internal aspects of the problem. Locating and resolving problems in the distributed warehouse manager environment can be similar to solving a crossword puzzle. Puzzles look different from each other, but after working on a few of them, you begin to see a pattern. Each person has a slightly different method of approaching it, depending on the clues that are presented. As long as you work systematically through all of the clues, the solution eventually presents itself. In this redbook, the problem determination techniques that may be used to produce detailed problem information are shown with step-by-step instructions. The use of the various displays and menus are explained to assist in gathering information for IBM support.
Problem source identification phase:

In the second phase, you study the information you have collected to isolate the system components that are indicated in the failure. This is the problem source identification phase. Some problems are easy to limit to a single component. But some situations may be more complex, involving relationships between various components. In some cases, it is not possible to give a complete explanation of the cause of a problem. The support professionals with the assistance of appropriate diagnostic information will be able to recommend a course of action to recover from a problem.
Problem fixing phase:

The third phase involves fixing the problem, if possible, or at least circumventing it. If the root cause of the problem is well defined and understood, the solution to the problem is certainly not far away. Some problems will point to the core code of either the warehouse agent or the warehouse server. Only the developers maintaining those systems can provide a modification that will solve such problems. Other problems will point to user-written or third-party applications or operational procedures. This methodology is system-independent, when the problems that need to be fixed are linked to the logic of the warehouse server itself. However, some of the tools and techniques used may differ according to the operating system on which you run an agent.
230
6.2 The generic troubleshooting methodology

This section describes the generic troubleshooting methodology which describes the logical flow you go through to solve any problem, as shown in Figure 6-1: Problem determination: Start looking at return codes. Read the error messages from DB2 Information Center. Problem source identification: Identify the problem source. When the problem requires more investigation, be able to understand the DB2 Warehouse Manager architecture components and their interaction. Determine which trace or additional tool to use to get a better diagnostic. Determine the specific component concerned by the problem: GUI Warehouse agent (also called agent in this chapter) Warehouse server Warehouse control database Warehouse source databases Warehouse target databases Information Catalog Manager and Information Catalog database Application Operations Performances Network
Problem fixing: Correcting a user error Installing DB2 Warehouse Manager fixpacks or patches Looking at the specific problem areas.
Chapter 6. Problem determination methodology
231
How to start?
Start looking at return codes supplied
PROBLEM DETERMINATION
Read the DWC error messages from DB2 Information Center
Yes Understand the likely cause? No
WM components Kernel agent Memory
Where to look at?
Understand WM Architecture
Additional error information required ?

1)Use appropriate traces
PROBLEM SOURCE IDENTIFICATION
Warehouse client location Warehouse Server location Agent location Target DB location
Control Center trace
CLI, Server traces (Level 5) Agent logs, Msg queue trace, VWP and transformer traces CLI, ODBC traces
2)Use additional resources
What kind of issue/failure?
No Yes
User error?
OS error?
No
Area of failure identified?
Yes
Yes
No
PROBLEM FIXING
Server failure
Database issues
Network failure
How to solve issues/ restart from failures?
Cor rect an d restart
Res tart from Server cras h
Solv e is su es : E rror symptom Process does not complete in the ex pected time
Restart from commun ication s errors betw een: Serv er, Agen t, Sou rce, Target
Raise PMR, A pply Serv er & A gent trace level 5
Figure 6-1 The troubleshooting generic methodology
6.3 The problem determination phase

Problem determination basically is a technique of isolating a problem which involves understanding the problem and defining it in terms meaningful to those who support the system. The first objective of problem isolation is to define the external symptoms accurately. The external symptoms are the attributes of the incident that drew attention to the existence of a problem. The messages are the most important symptom that will be present in case of a problem. The basic problem determination steps are shown in Figure 6-2. This phase includes the initial diagnosis of the problem, using: Error messages DB2 Information Center
232
Problem determination: How to start?
Start looking at return codes supplied
Read the DWC error messages from DB2 Information Center Yes
Understand the likely cause?

No
Cor re ct a nd r es tar t
Figure 6-2 Problem determination steps
6.3.1 A problem occurs

When a problem occurs, the DB2 Warehouse Manager user gets an error message code. Messages provide answers to many problems and assist you with most of the problem isolation. When the system detects an error, it usually provides a message that describes the problem and the actions needed to solve the problem. A message can contain the following information: The Data Warehouse Center error token Information that explains the problem Return codes and error codes.
233
For example, the user gets the error message in Figure 6-3 when they try to run a warehouse step which may be used to populate a target table or to run a transformer.
Figure 6-3 Error message
A detailed log of the execution of each step is maintained, so if a step failed, this would be the first place to look for troubleshooting: 1. Right-click the failure step, then select Show Log to get the log screen. 2. Locate the first occurrence of the Run Time Error, then right-click, and select Show Details to obtain additional information on the failure.
6.3.2 How to start problem determination

For the example case we discussed in Figure 6-3, the best approach would be to try to understand the brief information provided with the error token. In most of the cases, information provided with the error token is not enough to pinpoint a problem. In those cases you should see the explanation and user response for the return codes in the DB2 Warehouse Manager documentation. The Data Warehouse Center messages can be viewed online in the DB2 Information Center, which is shipped with DB2 Warehouse Manager.
234
Here are the various steps to follow: 1. Go to the Help MenuBar and open Information Center (Figure 6-4).
Figure 6-4 DB2 Information Center
235
2. Click the Troubleshooting tab (Figure 6-5).
Figure 6-5 Troubleshooting Tab in DB2 Information Center
236
3. Expand All messages (Figure 6-6).
Figure 6-6 DB2 Messages
4. Double-click DB2 Messages, which opens a Web browser on your system. 5. In the Contents page in the left-hand side, click Data Warehouse Center Message. Place the mouse cursor in the right side frame in the browser and then press the Ctrl-F key. 6. This opens a Find window. Type the 4-digit return code and click the Find Next button (Figure 6-7).
237
Figure 6-7 Search in DB2 documentation
7. The search result should take you to the appropriate error message reference (Figure 6-8).
Figure 6-8 Error message reference
238
Remember that the secondary return code (RC2) may display an error number that is returned by the operating system. This gives a better idea of what error actually occurred within the cover. It is clear from the RC1 and RC2 messages in Figure 6-3 on page 234 that either the warehouse agent is not running on the agent system, or something to do with IP sockets has been set up incorrectly, probably on the agent side. The first message says that warehouse server could not establish communication with the agent daemon; the second said it was an error, something to do with sockets. The next step for the user should be to log on to the server, where the warehouse agent is installed, and verify if the agent daemon is up and running At that point, you may also contact the IBM Support Center to receive assistance. You may try, with their help, to recreate the problem, run commands, collect trace data, perform tests, or take whatever steps are suitable to pinpoint what is not working.
6.3.3 Understanding DB2 Warehouse Manager architecture

To be able to go ahead in problem determination and to find out the components involved, it is important to have a full understanding of DB2 Warehouse Manager architecture. DB2 Warehouse Manager is a leading data warehouse development environment with the distributed agent-based architecture (Figure 1-2 on page 7 shows the agent-based architecture and the interaction between different components of the products). The warehouse agent executes the transformation and data movement requests, allowing the administrator to select the correct platform for transformation processes and enabling point-to-point, distributed, and parallel data movement. Before looking at problem determination tools and techniques, we must discuss architecture information about the DB2 Warehouse Manager server to help you with problem determination. The current release of DB2 Warehouse Manager is Version 7 Release 2.
DB2 Warehouse Manager architecture: who does what?

The main component of DB2 Warehouse Manager is the warehouse server. In the current architecture, warehouse server can only be installed on Windows NT/2000. The warehouse server basically is an operating system process started from operating system services.The warehouse server runs with several threads like a scheduler, a listener through which it sends and receives messages, and a set of jobs which control the warehouse agent execution.
239
The warehouse agent receives a command from the warehouse server in the form of agent messages, executes them, and then returns a reply to the warehouse server. The warehouse agent process is controlled by the warehouse server process and works as a conduit for data flow between the warehouse source and target databases (see Figure 6-9).
Agent role during population

Server Site
Step Execution
Message
11 12 10 9 8 7 6 5 1 2 3 4
Agent Site
Agent
SQL SELECT
Source Site
SQL SELECT
Data
DDD Editions
ODBC Driver
Data
Source Database Server
Source Database
Control Database
Target Database
SQL INSERT
Data
Warehouse Server
Schedules wake up Reads Step definition from Control Database Starts Agent (through Agent Daemon) Receives and logs result (update operational metadata)
Agent
Acknowledges to Warehouse Server Logic: - Receive command - Execute command - Report status
Figure 6-9 Warehouse agent role during population
The warehouse agent can also issue system commands and run user defined programs on the machine which it resides. The warehouse agent can be local or remote. A warehouse agent that runs on an agent machine which is the same as warehouse server is called the local agent. The remote agent is an agent machine that is different from the warehouse server. The important difference between the two agents is that the local agent processes are started by the warehouse server; whereas the remote agent processes are started by an agent daemon. The agent daemon is a continuously running process which listens for agent start or kill requests from the warehouse server, and it is responsible for starting warehouse agent processes on a remote agent machine. It also has capability to forcibly kill a warehouse agent process if the warehouse agent ceases to respond to the warehouse server request.
240
To summarize the event, let us see who does what: 1. The agent daemon starts at the agent machine, listening at port 11001. 2. The warehouse server sends a request to the agent daemon. 3. The agent daemon creates a warehouse agent process listening at a new port, and the warehouse agent responds to the warehouse server with the port number at which it is listening. 4. The warehouse server sends request to the warehouse agent process. 5. The warehouse agent process handles the request and returns the results to the warehouse server. It handles the commit/rollback based on the data movements success. 6. The warehouse agent process tells the warehouse server it is done and goes away.
6.4 The problem source identification phase

The terms problem determination and problem source identification are often joined together. While this may seem to be an unnecessary duplication of terms, it conveys the idea that there is an important distinction between the two components of problem analysis. Figure 6-10 explains the process of finding out what has caused the problem.
241
Problem source identification: Where to look

PROBLEM DETERMINATION
Understand the likely cause? No

WM components Understand WM Architecture Kernel agent Memory
Additional error information required ?
Warehouse client location Warehouse Server location Agent location
Control Center trace
1) Use appropriate traces
CLI, Server traces (Level 5) Agent logs, Msg queue trace, VWP and transformer traces CLI, ODBC traces
2) Use additional resources
Target DB location
PROBLEM FIXING
Figure 6-10 Problem source identification steps
6.4.1 Where to find additional error information

Having identified the messages and obtained their explanation, together with an understanding of the architecture, the next step is to identify the area in which the problem is occurring. There are a variety of traces which can be applied to the system to enable you to get more information. One approach is to put every tracing facility on and inspect the lot. However, it is far more productive if you can deduce the nature of the problem from the message and then apply the appropriate traces.
242
For example, if we take the previously described problem, where the warehouse server was unable to send a request to the agent daemon to spawn a warehouse agent process, then we know that a warehouse agent trace would not be likely to yield any relevant information (because no warehouse agent got spawned). We do know that a warehouse server trace would be likely to show that it is sending a request and getting an error back, so this would be a good piece of information to collect. We also know from the messages that an inspection of the TCP/IP definitions at the server and remote ends would provide some relevant information too. This section shows the source of information that DB2 Warehouse Manager provides through Data Warehouse Center traces and logs (see Figure 6-11) that is helpful in problem determination phase.
DWC Tracing
Clients
Warehouse Server
Warehouse Agents
Databases
Control Center
CLI Trace A G E N T Server Trace CLI Trace

Control Database DB2 Log Editions Configuration
DB2 Target
CC Trace
O D B C T R A C E
Relational Source
T R A C E
NonRelational Source
non-DB2 Target
Typ titl e e
. 1Typeext t
Flat Files
Figure 6-11 Data Warehouse Center traces
243
The following logs or traces are available and are described in this section: Warehouse client location CC Trace or Control Center trace Warehouse server location CLI traces Warehouse server traces Agent site locations Agent logs Message queue traces UDP traces ODBC connectivity traces Connector traces
Target database location CLI traces ODBC traces Transformer traces
Warehouse client location

The Control Center traces can be very useful to isolate a User Interface (UI) problem or error. However, the Control Center trace can be of only one level and there is no filtering available. The trace can be started from the DB2 Control Center or from the DB2 command window: 1. You can start the Control Center trace from the DB2 Control Center user interface as follows: a. Use Tools -> Setting interface, as shown in Figure 6-12
Figure 6-12 Tools settings
b. Check the box Add Tools Tracing to DB2 Trace, as shown in Figure 6-13.
244
Figure 6-13 Starting the Control Center trace
2. Optionally, you can also use DB2 command line to start the Control Center trace. Type: db2cc -tf <filename> The trace written in the file may not be formatted. A trace formatting utility is provided in DB2. To run this utility, open a command line window. Change the directory to SQLLIB/CC and run this command: jre TraceFormatter <infile> <outfile>
245
The formatted trace output may look like that shown in Figure 6-14.
CC
Thread name
05/2/00 7:42 AM |CC:run() - entry 5/2/00 7:42 AM ||CC:construct(localhost,6790) - entry 5/2/00 7:42 AM |||CommonImageRepository:CommonImageRepository(TreeNav) - entry 5/2/00 7:42 AM |||CommonImageRepository:CommonImageRepository(TreeNav) - return() 5/2/00 7:42 AM |||CommonImageRepository:CommonImageRepository(TreeNav) - entry 5/2/00 7:42 AM |||CommonImageRepository:CommonImageRepository(TreeNav) - return() 5/2/00 7:42 AM |||ImageRepository:ImageRepository(TreeNav) - entry 5/2/00 7:42 AM |||ImageRepository:ImageRepository(TreeNav) - return() 5/2/00 7:42 AM |||CC:initialize(localhost,6790) - entry 5/2/00 7:42 AM ||||CC:initializeWebccServer(hostname,port,parent) - entry 5/2/00 7:42 AM |||||AdminConnection:AdminConnection - entry 5/2/00 7:42 AM ||||||AdminConnection_App:AdminConnection_App(connection) - entry 5/2/00 7:42 AM |||||||CCServerDriver:getConnection() - entry 5/2/00 7:42 AM |||||||CCServerDriver:getConnection() - return(Handle=0 rc=0) 5/2/00 7:42 AM ||||||AdminConnection_App:AdminConnection_App(connection) - return() 5/2/00 7:42 AM ||||||AdminConnection:openConnection() - entry 5/2/00 7:42 AM |||||||JDBC_CC_ExtensionsDriver:call(mid) - entry 5/2/00 7:42 AM ||||||||JDBC_CC_ExtensionsDriver:call(mid,msg,retry,true) - entry 5/2/00 7:42 AM |||||||||JDBC_CC_ExtensionsDriver:runRequest() - entry
Classname
Method name Timestamp
Figure 6-14 Trace output example
Warehouse server location

Both DB2 CLI traces and warehouse traces may be used.
DB2 CLI trace

Warehouse users are not required to generate CLI traces to pinpoint a problem. But occasionally IBM support might ask the user to generate the CLI trace to isolate problems like these: Corrupted warehouse control database Uncaught metadata errors Errors which occurred in migrating the warehouse control database while running a backlevel migration against a later warehouse control database (for example, running the DB2 UDB V7.2 CONTROL DATABASE MIGRATION utility on a warehouse control database that is at a later level)
246
To get the CLI trace for the warehouse control database, follow these steps: 1. Shut down all applications to the warehouse control database. 2. Open the DB2 command line processor and connect to the warehouse control database. 3. Type the command: DB2 UPDATE CLI CFG FOR SECTION COMMON USING TRACE 1 TRACEPATHNAME <filename> TRACEFLUSH 1 and press Enter. 4. Restart the DB2 services from the Windows Control Panel - Services. 5. Recreate the problem. To stop the CLI trace, follow these steps: 1. Shut down DB2 services and client. 2. Open the DB2 command line processor and connect to the warehouse control database. 3. Type the command: 4. DB2 UPDATE CLI CFG FOR SECTION COMMON USING TRACE NULL TRACEPATHNAME NULL TRACEFLUSH NULL and press Enter.
Warehouse server traces

The warehouse server traces show server processing of a warehouse step. They are created in the SQLLIB/LOGGING directory on the system where the warehouse server is running. The DB2 Warehouse Manager user can get different levels of warehouse server traces, depending on the severity of the problem and the level of information needed. Levels 1 through 4 can be set in the Data Warehouse Center: 1. Right-click Warehouse. 2. Select the Properties panel. 3. Select the first notebook page to set up the trace. They provide different levels of information for problem determination:
Level 1: The level 1 warehouse server trace provides the entry and exit point of a function in the application code. Level 2: The level 2 trace warehouse server provides level 1 trace and some extra warning and error information. Level 3: The level 3 warehouse server trace shows level 2 information and add SQL statements and parameter substitution. Level 4: The level 4 warehouse server trace shows level 3 information and adds the command list execution.
247
There is a level 5 trace which is not documented and can only be set at the DB2 command line window. This level of trace displays the contents of data and communication buffers and may be very useful in isolating problems like user data error or memory errors. This trace should be set up exceptionally if required by IBM Support and should be administered under IBM Support assistance.
Agent site location

Multiple files may be used on agent site location for problem source identification: Agent logs Message queue traces on iSeries VWP traces Transformers traces ODBC connectivity traces Connector traces
Agent logs
The agent log shows all the information that is passed to the warehouse agent from the warehouse server and the command steps that are completed. For example, the agent log shows information such as these activities: Connect to target database Fetch and Insert statements SQL execute statements with parameter substitutions The file AGNTxxxx.LOG contains tracing information, where xxxx is the process ID of the warehouse agent process. Except iSeries system, there is another log file AGNTxxxx.SET created on warehouse agent platforms. This file contains environment information.
.SET files are very helpful in understanding the environment that the warehouse agent is running under and reviewing can resolve many problems. For example, if you are trying to run a step that calls a User Defined Program (UDP) and you get an error that the program was not found, you can examine the .SET file to see what the PATH environment is that the warehouse agent is running under.
Depending on the level of information needed, the agent logs also have 5 different level of trace information. Like the warehouse server trace, the fifth level of agent trace can not be set through warehouse properties panel. This fifth level of trace should be set up exceptionally if required by the IBM support and should be administered under IBM support assistance. The agent logs are created in different locations on different agent platforms. Table 6-1 shows the path of the warehouse traces on different agent platforms.
248
Table 6-1 Log file locations

Agent sites Location of log files
Windows/OS2 Unix iSeries (AS/400) zSeries (S/390)
x:\program files\sqllib\logging /var/IWH /qibm/userdata/IWH environment variable vws_logging specifies the directory
Message queue traces

Message queue traces are only generated on iSeries system. They are very useful for communications problems that cannot be diagnosed with the error code information alone. They can also be used for Cancel and Get Row Count problems. This trace is started automatically when tracing is turned on for either the warehouse agent or the daemon. To read this trace, use Microsoft Wordpad or any other Unicode enabled editor. This trace might contain non-printable characters and will not format correctly on a basic text editor like Notepad. The message queue trace file name looks like VWxxxxx.IWH4MSGQ where xxxxx is the process ID of the process that started the message queue process. An additional trace file can be produced by the message queue process. The msgq_err.log file is a cumulative trace file that records all nonrecoverable message queue errors. This file is useful for tracking down terminations of the message queue process that cannot be recorded in the regular message queue trace file.
Important: Trace slows performance. It should only be used for debugging.
UDP traces
These are trace files for sample programs shipped with DB2 Warehouse Manager. They display functional traces of the major events during the execution of the programs. The file name of these trace files on UNIX, Windows NT/2000 and zSeries (OS/390) agent is TRCxxxx.log where xxxx is either the process ID that the warehouse program was started or a timestamp indicating the time the warehouse program started. The file name of these trace files on iSeries agent is VWxxxxx.yyyyyy where xxxxx is the process ID under which the warehouse program was started and yyyyy is the name of the warehouse program.
249
If the warehouse program is started by the warehouse agent process, it will run in the same job as the warehouse agent process, so it will share the same process ID. The message queue trace, agent trace and the warehouse program trace will all share the same xxxxx value. The following table shows the location of UDP traces on different agent platform:
Table 6-2 UDP trace location
Agent sites Location of log files
Windows or OS2 UNIX iSeries (AS/400) zSeries (z/OS and OS/390)
\sqllib\logging /var/IWH /qibm/userdata/IWH and /qsys.lib/qiwh.lib/ftpcmd.file/MSxxxxx.mbr environment variable vwp_logs specifies the directory
Example: Finding UDP traces with iSeries agent

DB2 Warehouse Manager for iSeries agent supports three different types of sample programs: FTP programs iSeries (AS/400) load programs: OLAP Server programs for iSeries: These programs are described in detail in the following three subsections.
FTP programs: These programs generate three kinds of diagnostic information.

The return code, as documented in the DB2 Warehouse Manager online help. The FTP message log: FTP message logs are members in the QIWH/FTPCMD source physical file. They follow the naming convention MSxxxxxx, where xxxxx is the process ID of the FTP program execution that produced the file. To view the FTP message logs enter the following command at an iSeries command line: WRKMBRPDM QIWH/FTPCMD.
Important: Successful completion of this program does not guarantee that the data was transferred correctly. It only indicates that the FTP command ran. To ensure that the data was transferred correctly, you must check the FTP message logs for warning and non-fatal errors.
The FTP program trace file: The FTP command program trace file can be found in /QIBM/USERDATA/IWH directory on iSeries agent system.
250
iSeries (AS/400) load programs: These programs provide two kinds of diagnostic information:
The return code, as documented in the DB2 Warehouse Manager online help. The load program trace file: The Load program trace files are located in the /QIBM/USERDATA/IWH directory. The Load program trace file has the name format of VWxxxxxx.VWPLOADI or VWPxxxxxx.VWPLOADR depending on the program used. These trace files can be viewed on native iSeries environment or via Client Access Express for Windows on your PC. Here is how to view the trace file on your PC using Client Access Express: 1. Set up a Client Access Express connection to your iSeries system over TCP/IP. 2. Open the Windows Explorer on your PC. 3. From the Explorer menu, select Tools -> Map Network Drive. 4. Type the path name: \\hostname\QIBM where hostname is the fully qualified TCP/IP host name of your iSeries system. 5. Click OK.
Important: If you use Client Access Express to access trace file, you must define the file extension of the trace file to Client Access Express. Defining the extension allows Client Access to translate the contents of files with this extension from EBCDIC to ASCII.
To define a file extension to Client Access Express:

1. From Windows, select Start -> Programs -> IBM AS400 Client Access Express -> AS400 Operation Navigator. The Client Access Operation Navigator notebook opens.
251
2. Right-click Integrated file system by expanding your iSeries system name, as shown in Figure 6-15.
Figure 6-15 AS/400 Operations Navigator
252
3. Click Properties. This should open the properties notebook, as shown in Figure 6-16.
Figure 6-16 Integrated file systems properties
4. Type VWPLOADI in the File Extension field. 5. Click Add. 6. Click OK. You should now be able to load the file into an ASCII editor or word processor. If there was a failure of any of the system commands issued by Load programs, then there will be an exception code recorded in the Load program trace file. Here is how to get an explanation for the iSeries exception: 1. At an iSeries command line, enter DSPMSGD RANGE(xxxxxxx) where xxxxxxx is the exception code. For example, you might enter DSPMSGD RANGE(CPF2817) at the iSeries command line. The Select Message Details to Display panel is displayed.
253
2. Select option 30 to display all information. A message similar to the following message is displayed: Message ID: Message file: Library: Message: Cause: Recovery: Specify CPF2817 QCPFMSG QSYS Copy command ended because of error. An error occurred while the file was being copied. See the messages previously listed. Correct the errors, and then try the request again.
OLAP Server programs for iSeries: The OLAP Server programs provides three kinds of diagnostic information:
The return code, as documented in the DB2 Warehouse Manager online help The OLAP Server program trace: These trace files are located in /qibm/userdata/iwh directory on iSeries agent system. The file format is trcxxxxx.log where xxxx is the process ID of the program execution which produces the trace file. This trace is very useful in diagnosing the IBM DB2 OLAP Server or Hyperion Essbase Server problems The feedback log: The feedback log contains the specific Return code returned from DB2 OLAP or Essbase Server API and contains the relevant error information. The file is located in the same location as trace file and appear as vwpfdbk.log
Important: To diagnose the OLAP server program, first look at the feedback log and then look for the return code message in the trace file.
Transformer traces
DB2 Warehouse Manager transformers provide all diagnostic information in a database table called IWH.LOGTABLE. You can specify your own logging table name in the processing options for a transformer step. The default log table is created in the IWH schema on the agent/target database system, when a transformer step is executed for the first time. After that, the trace information is appended for each execution if trace is ON. It is a good idea to delete the contents of the logtable before you execute the step with diagnostic information. To clear the logtable, simply open the SQL interface on the target database system and run an SQL statement: DELETE FROM IWH.LOGTABLE
254
As with traces, IWH.LOGTABLE can contain different levels of trace information based on the level of trace set in the Properties page of the Transformer step. To turn the trace on, click the Processing Options tab on the Properties notebook, as shown in Figure 6-17.
Figure 6-17 How to turn the transformer trace on
The different levels of traces are described in Table 6-3.

Table 6-3 Levels of trace
Level of trace Information available
1 2 3
Only error information Level 1 and warnings Level 2 and informational messages
The column definition of the LOGTABLE are described in Table 6-4.

Table 6-4 Logtable column definition
Column Name Type Schema Type Name Length Scale Nulls
TSTAMP NAME RUN_ID MSG_ID
SYSIBM SYSIBM SYSIBM SYSIBM
TIMESTAMP CHARACTER INTEGER INTEGER
10 25 4 4
0 0 0 0
Yes Yes Yes Yes
255
Column Name
Type Schema
Type Name
Length
Scale
Nulls
MSG_TYPE MSG_TEXT
SYSIBM SYSIBM
CHARACTER VARCHAR
1 2000
0 0
Yes Yes
Message Type field (MSG_TYPE) contains a character value which corresponds to the logging level. For example, Message Types E (Errors), W (Warning) and I (Informational) correspond to the logging levels 1, 2 and 3 respectively. This makes easy for the user to extract the relevant information from the logtable. Users can write a simple SQL statement like this to extract all Data Warehouse Center errors logged in the logtable: SELECT MSG_TEXT FROM IWH.LOGTABLE WHERE MSG_TYPE = E The message text can be longer than the MSG_TEXT column. If the message text exceeds one row limit, it goes to second row and the MSG_TYPE field contains C to specify the continuation of the line.
ODBC connectivity trace

DB2 Warehouse Manager provides a test program, called odbctest, which you can run at your Windows and UNIX warehouse agent sites to validate connectivity from your Windows and UNIX warehouse agent sites to your ODBC data sources. This program attempts to connect to the database you specify and lists the contents of the database catalog. If test program is able to connect to the database, then the connectivity is properly set up and the warehouse agent should be able to connect. If the test program is not able to connect to the source database, then the error code that was encountered will be displayed, as will any ODBC driver messages. These messages will help you to configure and fix the connection to the source database. The location of the odbctest program varies with the operating system (see Table 6-5), where xx or x refers to the version and yy or y to the release level of the product.
Table 6-5 Odbctest program location
Operating system Location of odbctest
Windows AIX Solaris Operating Environment or Linux
\sqllib\bin /usr/opt/db2_xx_yy /opt/IBM/db2/Vx.y
256
Here is how to validate the connectivity of your ODBC data source for warehouse agents: 1. From a DOS or UNIX command line, type the following: On AIX, run this command: usr/opt/db2_xx_yy/bin/IWH.environment On the Solaris Operating Environment and Linux, run this command: opt/IBM/db2/Vx.y/bin/IWH.environment 2. Type odbctest <dsn> <uid> <pw>, where: <dsn> is the ODBC system (Windows) database to which you are attempting to connect <uid> is a valid user ID to connect to the <dsn> database <pw> is a valid password for user <uid> 3. To verify the connection to the system ODBC data source (called in this example 'target'), type: C:\Program Files\SQLLIB\bin>odbctest target resident my1pw 4. If there is something wrong with either the definition or the connectivity, an error message should be output to the screen.
Attention: Please be aware of the following considerations:

An error may sometimes occur when setting connection options. The Windows agent must have the database catalog as a system ODBC data source. Database connectivity does not require the use of user environment variables. UNIX agents must have an entry in the .odbc.ini file (located in the home directory of the user ID under which the UNIX agent is executing).
DB2 Warehouse Manager Connectors traces

Here we will discuss debugging and trace information for DB2 Warehouse Manager Connectors. DB2 Warehouse Manager V7 supports Connector for the Web and Connector for SAP R/3.
Connector for the Web: Figure 6-18 explains the debugging strategy for a Web Connector integrated with DB2 Warehouse Manager. To understand how the Web Connectors work, please refer to Connector for the Web on page 23.
257
DWC Error with Site Analyzer Token

Yes
DWC Error
DWC Error without Site Analyzer Token
Go To Site Analyzer Documentation
Go To the DWC Infocenter Yes
No
User Error
Generate Web connector trace / report to customer support
Correct the error
Figure 6-18 Debugging procedure for Web Connector
Error messages from the Connector for the Web are documented in the Data Warehouse Center messages in the following range: DWC08900X - DWC08929X. If you are not getting the WebSphere Site Analyzer (WSA) error token along with the Data Warehouse Center error taken, the first place to look for the error explanation is the Data Warehouse Center Infocenter. The WSA documentation comes with the WSA product. The WSA documentation can also be found on the Web at:
http://www.ibm.com/software/webservers/siteanalyzer/library.html
Table 6-6 lists the different traces you can generate to debug the Web Connector.
Table 6-6 Traces to use with Web Connector
Trace Name Usage
Web Connector trace for import source Web Connector trace for polling step Agent log
WSAxxxx.log WSAxxxx.log AGNTxxxx.log
To debug problems with data import To debug problems with polling step To debug Data Warehouse Center related problems
258
Connector for SAP R/3: The problem source identification methodology for the SAP Connector is pretty much same as Web Connector. The user should follow the procedure described below to find the cause of the problem:
If the user gets a Data Warehouse Center error message, they should look at the documentation for Data Warehouse Center messages. The range of the message tokens for the SAP Connector is: DWC08930X - DWC08941X The users may be able to find extra error information logged by an RFC module ( librfc32.dll) or by SAP R/3 servers. Such information can be found in the SAP documentation. If the error message explanation does not help you to reach the cause of the problem, you should turn on appropriate trace levels to generate two types of trace information, as shown in Table 6-7.
Note: The 4th or 5th level of agent trace generates SAP Connector trace files.
Table 6-7 Traces to use with SAP Connector
Trace Name Usage
SAP Connector trace Agent log
SAPxxxx.log AGNTxxxx.log
To debug SAP Connector specific problems To debug Data Warehouse Center agent related problems
If that does not help solve the problem, you will need to contact IBM support with the trace information you generated.
6.4.2 Other resources/tools to use

To use the information you gather, and to check if any answer to a problem exists, you can consult the IBM DB2 Warehouse Manager Web site at:
http://www-4.ibm.com/software/data/db2/datawarehouse/support.html
IBM provides a great deal of technical information free of charge on the Web. This DB2 Warehouse Manager support Web page is the first place you should look for the answer to a problem, especially if it appears to be a software bug.
Note: Any latest changes or quick update to the product user guide are posted in a FAQ or Technote on the support Web site.
The Web site window is shown in Figure 6-19.
259
Figure 6-19 DB2 Warehouse Manager Web site
In addition to the latest FAQ and Technical notes, also known as Technotes, the support Web site contains several types of information and tools, including: Product news Release notes written by Development and Technical support staff White papers/online publications Case Studies Support downloads A complete catalog of books, manuals, redbooks and supplemental materials
260
Finding information is easy. You can navigate using the menus and views, or use the search feature in the support database. For example, here is how you can find the latest FAQ on the import/export problem in DB2 Warehouse Manager: 1. Connect to:
http://www-4.ibm.com/software/data/db2/datawarehouse/support.html
2. Click Frequently Asked Questions (FAQs). 3. In the drop-down box, select Import /Export. 4. Click Go. 5. You can filter the result by searching with keywords. The list of FAQs appears on the Web, as shown in Figure 6-20.
Figure 6-20 List of FAQs on the Web site
261
6.5 The problem fixing phase

As explained in the previous section, different kind of problems can have different kind of solutions. A solution for a problem can be found by: Correcting a user error Installing Operating System fixes either on the warehouse server or the warehouse agent platform Installing the DB2 Warehouse Manager product fixes or patches Looking at the specific problem area: Warehouse server failure Database access issues Network failure
Problem fixing: How to solve the issue or restart?
No Yes
User error?
OS error?
No
Area of failure identified?
Yes
Yes
No
Server failure
Database issues
Network failure
Correct and restart
Restart from Server crash
Solve issues: E rror symptom Process does not complete in the expected time
Restart from communications errors between: Serv er, Agent, Source, Target
Raise PMR, A pply Serv er & A gent trace lev el 5
Figure 6-21 Problem fixing steps
6.5.1 User errors

User errors must not be overlooked as a possible cause of the problem. Here are a few examples of user errors in DB2 Warehouse Manager scenario:
262
A DB2 Warehouse Manager uses a remote agent to execute a warehouse step and the remote agent daemon is not started on the remote system. The warehouse agent calls an external program (user defined program) and passes an invalid number of parameters. DB2 Warehouse Manager user does not acquire appropriate authority on database objects.
Note: Follow the steps defined in problem source identification phase to verify it is a user error and then take corrective action.
Basic recommendations to avoid user errors

The major user errors come from the logon issue when connecting to the Data Warehouse Center GUI.
How to logon to the Data Warehouse Center

The Data Warehouse Center can only be started from the DB2 Control Center. The link to the Data Warehouse Center from the DB2 Control Center is shown in Figure 6-22.
Figure 6-22 Linking to the Data Warehouse Center
263
The logon settings are shown in Figure 6-23. During Data Warehouse Center logon, use the Advanced button to specify the warehouse server to which you want to connect. If the warehouse server is installed on the same system, the server host name should point to localhost. But if you are connecting to the warehouse server from a DB2 Administrative client, the server host name field should point to the fully qualified hostname or TCP/IP address of the warehouse server, as shown in Figure 6-23.
Figure 6-23 Logon settings
Subsequent invocations from the same DB2 Control Center session do not require you to logon again. The warehouse control database name must match the current warehouse control database of the warehouse server. If you are trying to connect to a different or a new warehouse control database, make sure that it is initialized before you connect. If the warehouse control database does not exist or it is not initialized, you may get the error, as shown in Figure 6-24.
264
Figure 6-24 Logon error example
To isolate the problem, please use the following users checklist:
User's checklist:
1. Verify that the warehouse control database under the logon advanced window matches the current warehouse control database for the warehouse server. To check if the warehouse control database exists already, use the following command on the DB2 command window: db2 list db directory This should list all the databases cataloged locally or remotely, as shown in Figure 6-25.
Figure 6-25 db2 list database result
2. Reinitialize the warehouse control database. To do this, select Start => Programs => IBM DB2 => Warehouse Control Database Management
265
3. Type the new/existing warehouse control database name, your existing user ID and password, and click OK. If initialized successfully, you should see the messages shown in Figure 6-26.
Figure 6-26 Warehouse control database initialization
4. Click Cancel to exit. 5. Now logon to Data Warehouse Center using the new warehouse control database name.
Note: If you have recently switched between a LAN connected environment and a standalone warehouse server environment without rebooting, the logon issue usually occurs.
You may be able to get away with just stopping and starting all the DB2 Warehouse Manager services, but a reboot will likely be more effective. If you cannot solve the error, try reinitializing the warehouse control database.
Adding a new user in DB2 Warehouse Manager

Adding a new user to access the warehouse server is a very easy process: 1. Logon to Data Warehouse Center using an existing user ID. 2. Expand Administration in the warehouse tree.
266
3. Expand Warehouse Users and Groups. 4. Right-click Warehouse Users to define a new user. 5. Define a new User, password, and email address, as shown in Figure 6-27. 6. Select the Active user check box, and click OK. 7. Close the Data Warehouse Center and Control Center, and logon with the new user ID.
Figure 6-27 Adding a new user
6.5.2 Operating system errors

Sometimes the users attention is focused on a software bug in the product code, and the problem is found in the operating system running on the warehouse server or the warehouse agent platform. For example, suppose the warehouse transformers fail to run on the warehouse agent platform. A closer look at the log table generated on the warehouse agent system might provide some clues about this problem. We recommend that the user verifies that they have the latest patches or fixpacks for their database on the respective platforms.
267
How to check the latest fixpack

Contact your database administrator or the operating system vendor (in the case of an iSeries agent system, the database is integrated with operating system) to find out if the necessary patches or fixpacks are applied.
Product fixpack apply recommendations

When you install a fixpack for DB2 Warehouse Manager, make sure you install same level of fixpacks for all components of it. For example it is necessary to apply fixpacks for warehouse agent, if you have applied fixpack for warehouse server. DB2 Warehouse Manager fixpacks are available on Web:
http://ftp.software.ibm.com/ps/products/db2/fixes
You can also contact IBM support for location of the fixpacks. It is recommended to read the Readme file for the fixpacks before you plan to apply fixes on any of the warehouse components.
How to check that each component has the right fixpack?

To verify if DB2 Warehouse Manager on NT/2000 has the latest fixpack applied: 1. Open a DB2 command window. 2. Type db2level and press Enter. The result should look like Example 6-1.
Example 6-1 Using db2level command
C:\>db2level DB21085I Instance DB2 uses DB2 code release SQL07023 with level identifier 03040105 and informational tokens DB2 v7.1.0.55, n011126 and WR21294.
3. Write down the word WRxxxx (WR21294 in this case) and call IBM Support to find out what fixpack level of code you have. You can also verify with the second informational token in the above window which corresponds to the date when this fixpack was built.
Will my warehouse control database be affected when I install the product fixpack?
When you apply the fixpack on Windows NT and Windows 2000 systems, you are asked whether you want to migrate the metadata in your warehouse control database. You must migrate your metadata in order to have an operational warehouse environment. The migration can be done during the fixpack installation or later using the Warehouse Control Database Management window on Windows NT or Windows 2000 systems which have the DB2 Administration Client at the fixpack level.
268
It is important for you to make a backup copy of your warehouse control database before your warehouse metadata is migrated. You should also update the configuration of the warehouse control database so that logging space is sufficient for the metadata migration and for use of the database by the Data Warehouse Center. The recommended settings are: Log file size (logfilsiz) = 4096 Number of primary logs (logprimary) = 5 Number of secondary logs (logsecond) = 100 This is how to change the metadata configuration settings: Open the DB2 Control Center and find your warehouse control database. Right-click the warehouse control database (TESTDB in this example) and click Configure, as shown in Figure 6-28.
Figure 6-28 Metadata configuration
The Configure Database page opens. Click the Logs tab. Select one parameter at a time and change the value in the Value Text box, as shown in Figure 6-29. Click OK to save.
269
Figure 6-29 Change metadata configuration
6.5.3 Looking at the problem areas identified

This section of this redbook discusses the details on how to handle problems in the following areas: Warehouse server failure Database problems Network problems
Warehouse server failure

Here is a quick checklist of what to do first if your warehouse server fails: Are you on the latest software levels and fixpacks available? Which data has to be collected to get an overview of the problem? Was it a user error? Is it a severe problem? What is the essential information in the data you need collect and provide, as a problem summary, to the IBM Support specialist? Read on to learn how to handle a warehouse server crash.
270
What is a warehouse server failure?

A warehouse server failure is the term we use when, for some reason or other, the warehouse server is not able to schedule new jobs or is not able to respond to administrative requests. This could happen for any number of reasons, ranging from possible user error to some warehouse server induced error to an operating system or hardware failure of some kind.
User's checklist for warehouse server failures: Here is a quick checklist of what to do in the unlikely event that the warehouse server stops for some reason:
1. Stop all components of the DB2 Warehouse Manager, the remote agent (when it exists), the agent daemon (local Windows NT/2000 agent), warehouse logger, and warehouse server services, as shown in Figure 6-30.
Figure 6-30 Warehouse server services
2. Check that all log files are saved in ..\sqllib\logging. 3. Check the Windows NT/2000 Event viewer to see if any logs are generated: a. Select Start -> Settings -> Control Panel -> Administrative Tools -> Event Viewer (see Figure 6-31). b. Select Application Log, Security Log, and System Log to check the error reported. 4. Analyze and report the error to IBM support. 5. Restart the warehouse logger and warehouse server in the NT/2000 Services dialog. 6. Restart the client. After restarting all components, if you are not able to run a step, then rerun the metadata initialization utility mentioned in 6.5.1 User errors on page 262.
271
Figure 6-31 Event log example
Database problems
In this section, we focus on database access problems in general when using the warehouse agent and also on encountering corrupted data problems with the warehouse control database.
Database access problems using the warehouse agents

Warehouse agents are predominantly manipulating sources and targets via an SQL statement. In the case of a warehouse agent failure of any kind or a database access failure, it is useful to understand the way in which the warehouse agent is accessing the (DB2) data. Most of the time, this is achieved through the DB2 Call Level Interface (CLI). In addition, if the connection is remote to iSeries or zSeries servers, the communication is likely to be through Distributed Relational Database Architecture (DRDA). The following sections describe more about each of these: DB2 CLI problems DB2 DRDA problems
DB2 CLI problems

All data access or program calls that happen from the DB2 Warehouse Manager to the relational database go through either the DB2 CLI or the ODBC interface.
272
Call Level Interface (CLI): The DB2 CLI is the IBM callable SQL interface to the DB2 family of database servers. It is an application programming interface for relational database access and uses function calls to pass dynamic SQL statements as function arguments. DB2 CLI is based on Microsoft ODBC specification and the International Standard for SQL/CLI.
Using this interface, an application developer can code, compile, and ship an application without targeting a specific Relational Database Management System (RDBMS). The DB2 CLI is an alternative to an embedded dynamic SQL. For the most part, the SQL CLI is syntactically and semantically equivalent to ODBC. This is not to say that everything in the latest level of ODBC is supported, but an application that uses the most common routines will likely run against the CLI with few changes. The SQL CLI is a standard created by the X/Open standards group, and the ODBC is the Microsoft implementation of the X/Open CLI standard.
DB2 CLI problem determination: The DB2 Call Level Interface (or CLI) is at the heart of any access to the database. If there is a problem here, database or operating system problem determination tools should be used. If the problem is in DB2 Warehouse Manager, you can expect it to be:
An application coding error A bug in the function that accesses the DB2 Call Level Interface Instructions on how to set up CLI connectivity are provided in DB2 Warehouse Manager Installation Guide, GC26-9998.
CLI tracing in UNIX and Intel environments: Data Warehouse Center ODBC drivers for several non-DB2 databases are installed when you install a warehouse agent.
To get the CLI trace, follow these steps for a specific database: 1. Shut down all applications to the database. Open the DB2 command line processor and connect to the database 2. Type the command:
DB2 UPDATE CLI CFG FOR SECTION COMMON USING TRACE 1 TRACEPATHNAME <filename> TRACEFLUSH 1
Then press Enter. 3. Restart the DB2 services from Windows Control Panel - Services. 4. Recreate the problem.
273
To stop the CLI trace, follow these steps: 1. Shut down DB2 services and client. 2. Open the DB2 command line processor and connect to the database. 3. Type the command:
DB2 UPDATE CLI CFG FOR SECTION COMMON USING TRACE NULL TRACEPATHNAME NULL TRACEFLUSH NULL
Then press Enter.
CLI Tracing in iSeries (AS/400) environment: The following information is from the CLI Frequently Asked Questions document available at the Web site:
http://www.as400.ibm.com/db2/clifaq.htm#header_16
The CLI has a trace function built into the code, starting with OS/400 V4R3. To turn on the trace, use the following command: CHGCPATRC JOBNUMBER(*CURRENT) SEV(*INFO) This example turns on *INFO level tracing for the current job. The *INFO level is what is needed for the CLI to start writing out trace records. For a long-running program that uses many CLI calls, this trace can fill up quickly. The CHGCPATRC also has parameters for specifying the buffer size for the trace, whether the trace should wrap when full, and whether the buffer should be resized.
Note: The trace can be turned on for jobs other than *CURRENT, by simply specifying the job number. This allows you to run the CLI trace against batch jobs, prestart jobs, and so forth.
After turning on the trace, and running a CLI program, use the DSPCPATRC command to see the trace records. You will see every CLI function call, along with a display of the parameters. This may be helpful in debugging problems, or just to identify how many times a certain function is called. An alternative for those who do not have the CHGCPATRC and DSPCPATRC commands on their system is to use the CHGUSRTRC and DMPUSRTRC commands instead. However, there is one extra step. When doing the CHGUSRTRC command to set the buffer options, you will notice that this command does not have a way to set the level of tracing. This is accomplished by doing the following: CALL QSYS/QP0WUSRT parm('-l 2' '-c 0' 'xxxxxx') Here, 'xxxxxx' is the job number of the job you want to trace. The program is spelled Q, P, then the number zero, and the first parameter is a lower case l.
274
Make sure you specify the program name correctly. You can simply cut and paste the command line shown above. All the CLI trace records start with 'SQL CLI:' in the text. Since there may be trace records from non-CLI functions, such as thread APIs, you can use those values to quickly locate CLI trace records. Another option is to query the temporary file that contains the data, and use search criteria to just display the CLI trace records, for instance, if your DSPCPATRC output looks as shown in Example 6-2.
Example 6-2 DSPCATRC output
Display Physical File Member File . . . : QAP0ZDMP Library . . : QTEMP Member . . .: QP0Z886434 Record . .: 1 Control . . . . . Column . .. : 1 Find ....... *...+....1....+....2....+....3....+....4....+....5....+....6....+....7....+... 886434:222784 SQL CLI: SQLAllocHandle ( 1 , 0 , SQLINTEGER* ) 886434:246288 Qp0zSGetEnv(QIBM_USE_DESCRIPTOR_STDIO) 886434:246976 Qp0zSGetEnv(QIBM_USE_DESCRIPTOR_STDIO): Could not find string 886434:513040 SQL CLI: SQLSetEnvAttr ( 1 , 10004 , 1 , 0 ) 886434:513352 SQL CLI: SQLAllocHandle ( 2 , 1 , SQLINTEGER* )
You can filter out everything except the CLI trace records. Run the following SQL statement using either Interactive SQL or any query tool SELECT * FROM QTEMP/QAP0ZDMP WHERE SUBSTR(QAP0ZDMP,17,8) = 'SQL CLI:'
CLI tracing in z/OS and OS/390 environments: To set up the CLI environment and trace, follow these steps:
1. Bind ODBC locally and to any remote databases. Because the zSeries (OS/390) agent uses ODBC to communicate with DB2 UDB, you must bind your CLI plan to all of the local and remote databases that your agent will access. 2. When you complete binding the CLI packages, verify that the DSNAOINI environment variable in your .profile file points to the ODBC initialization file that uses the CLI plan that you just bound. For example, if the CLI plan is named DWC6CLI and the local system is named DWC6, the ODBC initialization file should contain the following information:
SUBSYSTEM stanza [DWC6 ]MVSATTACH=CAF PLANNAME=DWC6CLI
275
3. Set up your ODBC initialization file. A sample ODBC initialization file, INISAMP, is included in the usr/lpp/DWC/ directory. You can edit this file to work with your system, or you can create your own file. To be sure that the file works correctly, verify that it is properly configured: The DSNAOINI environment variable must point to the initialization file. The file must include CONNECTTYPE=2 in the common stanza. The file must include MVSATTACHTYPE=CAF in the subsystem stanza. Ensure that you have a data source stanza for your DB2 system. It should specify the location name of the local DB2 system. An example of a DSNAOINI file is shown in Example 6-3.
Example 6-3 DSNAOINI file
[COMMON ] MVSDEFAULTSSID=DWC6 CONNECTTYPE=2 ;APPLTRACE=1 ;APPLTRACEFILENAME=/usr/lpp/DWC/logs/application.CLITRACE ;TRACEFLUSH=1 ;Example SUBSYSTEM stanza for V71A subsystem [DWC6 ] MVSATTACHTYPE=CAF PLANNAME=DWC6CLI ;DATA SOURCE stanza for ISC710P1 data source [ISC710P1 ]|
To turn on ODBC tracing, remove the three commented lines in the COMMON section. For more information about binding ODBC and the DSNAOINI file, see the DB2 Universal Database for z/OS and OS/390 ODBC Guide and Reference, SC26-9005.
DB2 DRDA problems

DB2 Warehouse Manager accesses different data sources through Distributed Relational Database Architecture. A problem you encounter when running a distributed relational database application can exhibit two general symptoms: The process returns an error symptom. The process does not complete in the expected time. The diagrams and procedures below show how you can classify problems as application errors, performance related problems and system related errors, so you can use standard problem analysis methods to resolve the problem.
276
The process returns an error symptom:

To describe this problem, let us discuss a very common scenario (described in Figure 6-32) where a DB2 Warehouse Manager user sample contains a source or a target table and does get some kind of error. Generally, there could be two reasons for this problem. Either it is a problem with distributed relational data access, or there is a problem with the relation database itself on the warehouse agent/warehouse target platform. If you receive an error message, use the error message, SQLCODE, or SQLSTATE to determine the cause of the problem. The message description indicates what the problem is and provides corrective actions. If you do not receive an error message, you must determine whether distributed relational database is causing the failure. To do this, run the failing statement locally on the warehouse agent platform or use the SQL interface to run the program locally. If you can create the problem locally, the problem is not with distributed relational database support. Use the respective platform specific problem analysis methods. You can generate CLI traces on the relational database to see if a CLI call is failing.
Error Messages
No (Bad data) Locate failing statement and run on application server locally or with SQL interface
Yes
Use error message, SQLCODE or SQLSTATE to determine cause of error and any possible actions
Yes Problem in relational database Run CLI traces
Still Failing
Noto Problem in distributed relational database
Figure 6-32 Resolving error symptoms from the process
277
Handling iSeries messages: The iSeries system sends a variety of system messages that indicate conditions ranging from simple typing errors to problems with system devices or programs. The system sends informational or inquiry messages for certain system events. Informational messages give you a status on what the system is doing. Inquiry messages give you information about the system, but also request a reply. In most of the message displays, a message is accompanied by a letter and a number code, such as CPF0083.
The first two or three letters indicate the message category. Some message categories for distributed database are shown in Table 6-8.
Table 6-8 Message categories
Category Description Library
CPA through CPZ MCH SQ and SQL TCP
Message from the operating system Licensed Internal code messages SQL messages TCP/IP messages
QSYS/QCPFMSG QSYS/QCPFMSG QSYS/QSQLMSG QTCP/QTCPMSGF
The remaining four digits (five digits if the prefix is SQ) indicate the sequence number of the message. The example message ID shown indicates this is a message from the operating system, number 0083. To obtain more information about a message on the message line of a display or in a message queue, do the following: 1. Move the cursor to the same line as the message. 2. Press the F1 (Help) key. The Additional Message Information display is shown in Figure 6-33.
278
Figure 6-33 Message type
You can get more information about a message that is not showing on your display if you know the message identifier and the library in which it is located. To get this information, enter the Display Message Description (DSPMSGD) command: DSPMSGD RANGE(SQL0204)MSGF(QSYS/QSQLMSG) This command produces a display that allows you to select the following information about a message: Message text Field data Message attributes All of the above
279
The text is the same message and message help text that you see on the Additional Message Information display. The field data is a list of all the substitution variables defined for the message and their attributes. The message attributes are the values (when defined) for severity, logging, level of message, alert, default program, default reply, and dump parameters. You can use this information to help you determine what the user was doing when the message appeared.
Message types: On the Additional Message Information display, you see the message type and severity code for the message. Table 6-9 shows the different message types for AS/400 messages and their associated severity codes.
Table 6-9 iSeries message types
Message type Informational messages. For informational purposes only; no reply is needed. The message can indicate that a function is in progress or that a function has completed successfully. Warning. A potential error condition exists. The program may have taken a default, such as supplying missing data. The results of the operation are assumed to be successful. Error. An error has been found, but it is one for which automatic recovery procedures probably were applied; processing has continued. A default may have been taken to replace the wrong data. The results of the operation may not be correct. The function may not have completed; for example, some items in a list ran correctly, while other items did not. Severe error. The error found is too severe for automatic recovery procedures and no defaults are possible. If the error was in the source data, the entire data record was skipped. If the error occurred during a program, it leads to an abnormal end of program (severity 40). The results of the operation are not correct. Severe error: abnormal end of program or function. The operation has ended, possibly because the program was not able to handle data that was not correct or because the user canceled it. Abnormal end of job or program. The job was not started or failed to start, a job-level function may not have been done as required, or the job may have been canceled. System status. Issued only to the system operator message queue. It gives either the status of or a warning about a device, a subsystem, or the system. Severity code
00
10
20
30
40
50
60
280
Message type Device integrity. Issued only to the system operator message queue, indicating that a device is not working correctly or is in some way no longer operational. System alert and user messages. A condition exists that, although not severe enough to stop the system now, could become more severe unless preventive measures are taken. System integrity. Issued only to the system operator message queue. Describes a condition where either a subsystem or system cannot operate. Action. Some manual action is required, such as entering a reply or changing printer forms.
Severity code
70
80
90 99
The message description indicates what the problem is and provides corrective actions. If you do not receive an error message, you must determine whether distributed relational database is causing the failure. To do this, run the failing statement locally on the warehouse agent platform or use the SQL interface to run the program locally. If you can create the problem locally, the problem is not with distributed relational database support. Use the respective platform specific problem analysis methods. You can generate CLI traces on the relational database to see if a CLI call is failing.
The process does not complete in the expected time:

If the request takes longer than expected to complete, you should check the Application Requester. Check the logs to see if the connect to a relational database is complete. If the system is not in a communication wait state, use problem analysis procedures to show whether there is a performance problem or a wait state somewhere else. There could be several reasons for slow query performance, such as these: An accessed table does not have an index. The first time you connect to a relational database (for example, DB2 UDB for iSeries) from a PC using a product like DB2 Connect, if you have not already created the SQL packages for the product in DB2 UDB for iSeries, the package will be created automatically, and the NULLID schema may need to be created automatically as well. This can take a long time and give the appearance of a performance problem. However it should be just a one-time occurrence.
281
A long delay will occur if the system to which you are trying to connect over TCP/IP is not available. A several minute time-out delay will precede the message. A remote host did not respond within the time-out period. An incorrect IP address in the relational database directory (on iSeries) will cause this behavior as well.
Warehouse control database problems

The following SQL statements may be helpful to determine inconsistencies in the warehouse control database.
Inconsistency error when importing a Data Warehouse Center tag file: An error may occur when importing a Data Warehouse Center tag file, and you may get the error code DWC03807E:
DWC03807E Step <step_name> being created or updated is associated with source resource, but the source tables are not associated with the steps source database
1. First use the SQL statement shown in Example 6-4 to see the steps/entries that are inconsistent and that will be changed.
Example 6-4 Find the corrupted relationships
select * from iwh.relationship rel where rel.relation_name='BV_TO_SourceIR' and rel.target_iwhid not in (select source_iwhid from iwh.relationship where relation_name='IR_TO_DR_Rel' and target_iwhid in (select target_iwhid from iwh.relationship where relation_name='BV_TO_Input_DR' and source_iwhid=rel.source_iwhid))and source_iwhid in (select source_iwhid from iwh.relationship where relation_name='BV_TO_Input_DR')
2. Now change them with the SQL statement shown in Example 6-5.
Example 6-5 Correct the inconsistency
update iwh.relationship rel set rel.target_iwhid = (select min(source_iwhid) from iwh.relationship where relation_name='IR_TO_DR_Rel' and target_iwhid in (select target_iwhid from iwh.relationship where relation_name='BV_TO_Input_DR' and source_iwhid=rel.source_iwhid)) where rel.relation_name='BV_TO_SourceIR' and source_iwhid in (select source_iwhid from iwh.relationship where relation_name='BV_TO_Input_DR' and source_iwhid=rel.source_iwhid)
Objects not found on import: This problem is caused on import, where you will receive an error DWC03142E with a message similar to the following:
DWC03142E Dataresource object 0000023047 was not found in the Data Warehouse Center control database.
282
1. Identify the steps that have dangling correlation objects, as shown in Example 6-6.
Example 6-6 Identify the steps that have dangling correlation objects
select name from iwh.businessview where iwhid in (select rel.source_iwhid from iwh.relationship rel where relation_name='BV_TO_Correlation' and target_iwhid in (select target_iwhid from iwh.relationship where relation_name='DR_TO_Correlation' and source_iwhid not in (select target_iwhid from iwh.relationship where relation_name='BV_TO_Input_DR' and source_iwhid =rel.source_iwhid) and target_iwhid=rel.target_iwhid))
2. Delete the dangling correlation objects for all the steps identified by seeing first (SELECT SQL statement) what we are deleting (Example 6-7).
Example 6-7 Delete the dangling correlation objects
select iwhid from iwh.correlation where iwhid in (select rel.target_iwhid from iwh.relationship rel where rel.relation_name='BV_TO_Correlation' and rel.target_iwhid in (select target_iwhid from iwh.relationship where relation_name='DR_TO_Correlation' and source_iwhid not in (select target_iwhid from iwh.relationship where relation_name='BV_TO_Input_DR' and source_iwhid =rel.source_iwhid) and target_iwhid=rel.target_iwhid)) delete from iwh.correlation where iwhid in (select rel.target_iwhid from iwh.relationship rel where rel.relation_name='BV_TO_Correlation' and rel.target_iwhid in (select target_iwhid from iwh.relationship where relation_name='DR_TO_Correlation' and source_iwhid not in (select target_iwhid from iwh.relationship where relation_name='BV_TO_Input_DR' and source_iwhid =rel.source_iwhid) and target_iwhid=rel.target_iwhid))
3. Delete the relationships to the dangling correlation objects by first seeing (with the SELECT statement) what we are deleting (Example 6-8).
Example 6-8 Delete the relationships to the dangling objects
select * from iwh.relationship where relation_name='BV_TO_Correlation' and target_iwhid not in (select iwhid from iwh.correlation) delete from iwh.relationship where relation_name='BV_TO_Correlation' and target_iwhid not in (select iwhid from iwh.correlation)
select * from iwh.relationship where relation_name='DR_TO_Correlation' and target_iwhid not in (select iwhid from iwh.correlation) delete from iwh.relationship where relation_name='DR_TO_Correlation' and target_iwhid not in (select iwhid from iwh.correlation)
283
Network problems
Communication errors may be encountered when the warehouse server is unable to communicate with agent sites, locally or remotely or when getting any TCP/IP connectivity error with the DB2 Warehouse Manager client/server components.
Warehouse server is unable to communicate with agent site

To communicate with each other, warehouse server and the agent site must use the same TCP/IP port entries. If you change a port number in the Windows NT/2000 services file, but do not change it on the agent system (or vice-versa), communication between warehouse server and warehouse agent system will fail. DB2 Warehouse Manager refers to ports by name rather than by number. During installation, DB2 Warehouse Manager assigns names to port numbers. On Windows NT/2000, DB2 Warehouse Manager writes the port assignments to the services file. On the agent system (for example, an iSeries system), DB2 Warehouse Manager uses the WRKSRVTBLE command to store the port assignments. By default DB2 Warehouse Manager uses these ports: 11000 (vwkernel) - warehouse server port 11001 (vwd) - warehouse agent daemon port 11002 (vwlogger) - warehouse logger Port You should only change a port number if that port number is already in use by another application. If you change a port number on either the Windows NT/2000 system or the agent system, ensure that you change the corresponding number on the other operating system.
TCP/IP connectivity errors

While using a remote agent, you might get the error message shown in Example 6-9.
Example 6-9 Error message example
Return Code = 7183 (Method = VWRemoteAgent::Initialize; Secondary Code =9117) Message: The warehouse server tried to spawn an agent but did not receive a valid start up acknowledgement from either the agent or the daemon.
284
The most common cause of RC7183 is improper configuration of TCP/IP connectivity between the warehouse server and the remote agent system. Communication between the warehouse server and the warehouse agent is bidirectional; the warehouse server sends messages to the warehouse agent, and the warehouse agent sends messages back to the warehouse server. Ensure that the warehouse server is connected to the remote agent system and vice versa. To test the communication between the warehouse server and the warehouse agent: 1. Ping the TCP/IP host name. Your host name is specified on the Parameters page of the remote agent site definition page of the Data Warehouse Center, as shown in Figure 6-34. If the ping fails, check if: The remote agent system is registered with your domain name server or that there is an entry for the system in the TCP/IP HOSTS file in the \winnt\system32 \drivers\etc directory. The agent system is running. The network is active.
Figure 6-34 Hostname defined on agent site definition
285
2. Ping the fully qualified TCP/IP host name of the warehouse server from the remote agent system. You must use the fully qualified name (hostname.domain), for example yourmachine.yourcompany.com. The fully qualified host name is the return address that the warehouse server gives the agent. If the ping fails, be sure that: The warehouse server is registered with your domain name server or has a host table entry on the remote agent system. The warehouse server is up and running. The network is active. If both of the ping attempts were successful, verify that the numeric IP address returned by the ping is actually the IP address of the workstation that you are trying to connect to.
6.5.4 What to do when the failure area is not identified

At that point, you will have to contact the IBM Support Center to raise a PMR and to receive assistance. Warehouse agent and warehouse server traces level 5, as described in Agent site location on page 248 for the agent trace and in Warehouse server location on page 246 for the warehouse server trace, might be required in order to recreate the problem and to provide IBM Support Center with both trace outputs.
286
Chapter 7.
DB2 Warehouse Manager hints and tips

Recovery issues in DB2 Warehouse Manager are also directly linked to the application design of the processes and steps involved in the data warehouse population and not only to data and application recovery issues, as performance issues should always have been taken into consideration when designing a data warehouse solution. This chapter discusses these topics: Warehouse application issues: Building a restartable application Application design hints and tips Process and steps failures Process and steps hints and tips Hints and tips to design specific processes
Performance issues: General performance advice Performance checklist to use when testing applications
287
7.1 Building a restartable application

The main question when designing an application on recovery perspectives is: How can a restartable application be built? This ranges from the design of the whole Extraction, Transformation, Move, and Load (ETML) application for building a data warehouse to the design of a single step itself. This section covers the ETML strategies for building a data warehouse or datamarts building processes and steps that use the functions of the DB2 Warehouse Manager. The data to feed the warehouse will have to be brought from the sources with specific ETML processes. It is important to define a good ETML strategy to ensure that the execution of these tasks will support the appropriate data volume and to be able to apply the proper business rules during the transformation (see Figure 7-1).
Operational Environment
flat files
Data Warehouse/Datamart Environment
Flat file READ process
Web
SAP R/3
Connectors for Web or SAP R/3 DB2 SELECT or LOAD process DB2 CAPTURE process Other RDBMS SELECT process Other RDBMS CAPTURE (triggers) process Transaction Program Queues Changes DB2 Apply Process Replicated Changes Transform Process Staging Area
Change Table
DB2 DB
Transform Process
DB2 DB DB2 LOGS
OTHER RDBMS
Data warehouse or datamart
Informix Oracle SQL Server

OTHER RDBMS
Change Table
Data Joiner
VSAM files
Transform Process Queue (MQ Series) Queue(MQ Series)
Extraction Phase
Transformation Phase
Move and Load Phase
Figure 7-1 ETML phases and processes
288
The DB2 Warehouse Manager would aid in defining these processes, making it easier to restart each step in the process if a failure should occur. Another point to consider in your application design is the importance of understanding what architectural elements are involved in the ETML and are essential to the ETML process flow. The ETML function is there to obtain the data, move it, transform it, and then store it.
7.1.1 Extraction
Consider the following issues in designing processes in the extraction phase: What are the operational sources of your data? Number of sources, and whether this would require creation of temporary tables or files to consolidate the data Purchased and external data, and what would be required to include this data with your other sources Homogeneous data from the DB2 UDB family Heterogeneous data from non IBM RDBMS that may require additional software to give transparent access to data How complex the data is Whether data complexity is aggravated, because this could mean pulling together many unrelated data sources. When will the operational data be available? Hourly Daily Nightly Weekly Monthly
What else can you determine about the data? What is the frequency of the data extraction? Are operational data sources connected to the data warehouse system? What is the amount of data to be moved to the data warehouse? Is the data selected from data sources inserted into a partitioned table? What automation of data extraction is required?
To extract data from the sources, many features may be used: SQL steps to select the source data and insert it into a staging area or in the target table IBM Data Replication to extract changes from data sources
Chapter 7. DB2 Warehouse Manager hints and tips
289
Use of replication warehouse programs with a replication source to extract inserts, updates, and deletes performed on relational data sources (rather than simply moving the whole source) DB2 massive loads
7.1.2 Transformation
This phase is the most complex and may involve several processes. You may need to store intermediate results in a staging area. Some issues are: What is the number of changes needed to be performed on the operational data? What level of integrity is required? Will there be some testing involved to ensure the integrity of data? What is the amount of programming required? Business rules can be defined in various ways: Clean transformer capability Unique/ check/ referential/ table constraints, implemented as triggers/user defined functions or stored procedures Adding programmable logic can be accomplished in various ways: Stored procedures using the SQL Procedures Language, C/C++, Java, or COBOL User defined functions User developed programs can be registered with and managed by DB2 Warehouse Manager, including the capability to return statistical metadata to DB2 Warehouse Manager Some specific transformers built on Java stored procedures and user defined functions are provided by DB2 Warehouse Manager to transform data according to statistical and warehouse requirements. There are three transformer groups: 1. Warehouse Transformers 2. Statistical Transformers 3. User-Defined Function These transformers can be used on OS/400 and z-OS and OS/390 as well.The following examples explain how you can use these transformers to solve some of your business problems.
290
Converting dates, times, and timestamps

A company has a set of data that includes the date, time, or timestamp in each row of a table. The company wants to convert the date, time, or timestamp data into a different date/time format. The solution is to use the FORMAT DATE AND TIME transformer to reformat the date and time data.
Note: FORMATDATE runs as a function against the input date column for each row in the table, and the formatted output date is placed in the output date column (which must exit).
The format of the input and output dates are chosen from menu lists of date and time formats. Optionally, a user can specify their own input or output date format using the pattern characters. If FORMATDATE cannot recognize an input date based on the input pattern string, then NULL is returned for the output date.
Getting statistics on business data collected

A company has collected data during the course of its day to day business operations. Decision makers within the company want to understand the characteristics of this data in order to determine whether or not operational changes are necessary. The solution is to use the CALCULATE STATISTICS transformer to perform basic statistical functions that numerically describe the companys data. This transformer writes summary information into relational tables which can be queried directly or used as input for further statistical analysis. There are ten statistical functions that can be achieved with this transformer. 1. COUNT number of items 2. SUM total when numbers are added together 3. MINIMUM smallest number 4. MAXIMUM largest number 5. RANGE difference between largest and smallest number 6. AVERAGE sum divided by the number of items (i.e mean) 7. VARIANCE average squared deviation from the mean 8. STANDARD DEVIATION square root of the variance 9. COEFFICIENT OF VARIATION standard deviation as a percentage of the mean 10.STANDARD ERROR standard deviation divided by the square root of the number of items
291
Following are some examples of how the functions can be used: Given a table associating sales personnel with a region, what is the size of the sales force in each region? (COUNT) Given quarterly revenues by region, what is the yearly revenue for each region? (SUM) Given daily sales for each month: What is the smallest and largest sale? (MINIMUM,MAXIMUM) What is the difference between the maximum and minimum? (RANGE) What is the average monthly sale? (AVERAGE) By how much do the sales amounts vary from the average? (VARIANCE,STANDARD DEVIATION, COEFFICIENT OF VARIATION)
Estimate the potential degree of discrepancy between a sample mean (average) and the population mean. (STANDARD ERROR)
7.1.3 Moving and loading data with DB2 Warehouse Manager

The Data Warehouse Center provides several different ways to move data, depending on the requirements:
To move a small amount of data: You can use SQL steps to select the source data and insert it into a target table. This process will be discussed later when we describe the SQL transformation step. To move changes to data: Rather than moving the whole source, use the replication warehouse programs with a replication source To move large amounts of data: You can use the export and load warehouse programs to export data from a table into a file, and then load the data into another table.
7.2 Application design hints and tips

This section provides a checklist summarizing the most important guidelines and recommendations to remember when building a warehouse population solution:
Never underestimate data modeling: Even if it seems easy to use the graphical user interface (Data Warehouse Center) directly to start building and designing the warehouse population solution, it might be much easier to do the data modelling and physical database design first before defining the processes. Although DB2 Warehouse Manager allows you to define target tables on the fly during process design, it is more thorough and better documented if you do the data definition as part of a proper data modelling exercise. For example, Erwin data can be imported into DB2 Warehouse Manager.
292
Perform one to one mapping: When building a system which accesses data from a number of physically different data sources, it is probably best first to map each source file or table to be extracted to a separate DB2 table and perform a one to one mapping. Joining the data can probably be done later more efficiently. The exception is in situations where two or more large data sources come from the same database. In this case, investigate whether there are efficiencies to be gained by using the source database to do the manipulation or joining. Use DB2 UDB load: The DB2 UDB load utility is significantly faster at loading data into DB2 UDB than any other method. You should experiment early on in the project to determine which sources should be input via the load utility and which should user other methods. You may also have to scale your configuration and use parallel/partitioned loads. Always minimize the number of steps: Once your design has loaded the source table into a DB2 intermediate table, you should try to devise SQL steps which will do the required processing to get to the target in as few SQL steps as possible, ideally one. If both the intermediate and target tables are in the same database, this will be the most efficient since DB2 Warehouse Manager will generate an INSERT from SELECT.
The number of steps per process should be kept reasonable. Perhaps a good guideline is to keep to what you can view on a single screen. By doing this, it is easier to see and understand what a process is designed to do.
Define some naming convention: You should try to maintain some naming convention that assists in understanding:
a. What the object is b. What its purpose is For example, you could prefix every SQL step with S_, UDP with U_.
Use UDP scripts: If you want to delete all the rows in a table, it may be much faster at execution time to use an installation written UDP script which will invoke the load utility to replace load with an empty file than issue the SQL delete statement. The load utility will run in a few seconds; the SQL DELETE might take minutes due to the number of deletions it is logging. Plan to test a subset of data: If possible, develop a subset of the real data for testing purposes: You do not want to wait several hours just to unit-test a simple change of a process. This way, basic testing of a complete system can take only tens of minutes rather than many hours.
293
7.3 Step and process failures

This section describes the steps and processes involved in any DB2 Warehouse Manager application design. Based on Operations and Work in Progress examples, we explain what are retry and interval parameters and what to do when a failure occurs.
7.3.1 Step and process definition

A step is a logical entity in the Data Warehouse Center that includes: The structure of the output table or file The mechanism (either SQL or a program) used for populating the output table or file The schedule by which the output table or file is populated; steps can be scheduled as follows: On demand, or scheduled to run at a set time Scheduled to run only once, or repeatedly for example every Sunday Scheduled to run in sequence, so that when one step finishes running, the next step begins running Scheduled to run upon completion, either successful or not, of another step. If you schedule a process, the first step in the process runs at the scheduled time In Figure 7-2 you can see all types of steps available in the Data Warehouse Center. These types of steps can be used with the process modeler to define all the necessary warehouse activities for that specific process.
294
Figure 7-2 The various steps in the Data Warehouse Center
Steps move data and transform data by using SQL statements or by calling programs. When you run a step, the transfer of data between the warehouse source and the warehouse target, and any transformation of that data takes place.
A process contains a series of steps that populates a warehouse target in a warehouse database by extracting data from one or more warehouse sources, which can be database tables or files. However, you can also define a process for launching programs that does not specify any warehouse sources or targets. For example, you can write a program to send a notification to the operator stating the number of records updated after the successful completion of the process. Generally, the first step in a series of interdependent steps is invoked by either a schedule, or an external trigger or manually, then the whole series will process automatically as designed by the warehouse administrator.
295
These are the various types of steps that can be performed:
SQL steps: An SQL step uses an SQL SELECT statement to extract data from a warehouse source and generates an INSERT statement to insert the data into the warehouse target table. Program steps: A process step can work with warehouse programs. There are several types of program steps. These steps run predefined programs and utilities. Transformer steps: Transformer steps are stored procedures and a user-defined function that specify statistical or warehouse transformations that you can use to transform data. The transformers can be used to clean, invert and pivot data, generate primary keys and period tables, and calculate various statistics. When the process runs, the transformer step writes data to one or more warehouse targets. The target table and the sources have to be in the same underlying database. Currently these transformer steps run on DB2 UDB database only. User-defined program steps: A user-defined program step is a logical entity within the Data Warehouse Center that represents an application that you want it to start. These can be locally written programs, or externally purchased programs. These programs can be used in place of an SQL step. The programs can be written in any language that produces an executable, or can be batch files, DLLs, or DB2 Stored Procedures.
7.3.2 What happens when a step/process fails

What happens if a warehouse step completely fails for some reason? Does the whole step have to start over? The capability to restart is really dependent on the how the step is implemented. In the case of an SQL transform, there are processing options that tell WHM what to do, for example, if a situation of no rows is returned. Is this acceptable, or is this an error, or should a warning be issued? In addition, a commit count can be specified; for example, commit after some number of rows. If, for some reason, the SQL statement returns an error, the DB2 engine returns the program back to DB2 Warehouse Manager. Any uncommitted rows will be rolled back and the step stops. The step processing options are shown in Figure 7-3.
296
Figure 7-3 Step processing options
What happens then depends on how the workflow is defined: Is it based on successful or unsuccessful completion of the step? If the step is designed to notify the operator upon completion via an e-mail, then one is sent so that the appropriate action can be taken. Where is all this taking place? How are the steps monitored? The Data Warehouse Center provides the monitoring capability via the Work In Progress (WIP) graphical user interface. The following screens will walk you through the process: The WIP is launched from the Warehouse drop-down menu on the main Data Warehouse Center window, as shown in Figure 7-4.
297
Figure 7-4 Work In Progress Start Screen
The WIP screen indicating the type of status messages available (for example, if a step was scheduled to run) is shown Figure 7-5.
Figure 7-5 WIP detailed step status
298
A step can be run manually by clicking on the Work in Progress drop-down menu then selecting the Run New Step as shown Figure 7-6. It can also be run by reclicking on the object in the WIP panel and by choosing Run Now from the popup menu.
Figure 7-6 Run step manually screen
299
As this step (the one that was manually submitted) replaces the data, the previous successful run step of this step is purged, and the new step will populate the table, as shown in Figure 7-7.
Figure 7-7 Manual submission step progress
300
To see more information on a step, right-click that step to bring up the context menu and select Show Statistics. The Statistics screen will appear. Right-click the step to show additional detail information, as shown in Figure 7-8. The statistics on a step varies by step type. With SELECT and INSERT we can get all the information; with INSERT FROM SELECT we can get all except for the number of bytes; for programs we get very little information; and for UDPs we have no statistics.
Figure 7-8 Step statistics screen example
More information on the execution history of the step, including the number of rows, and successes or failures of the step, is shown in Figure 7-9.
301
Figure 7-9 Information on the execution history of step
7.4 Steps and process hints and tips

This section provides a checklist summarizing the most important guidelines and recommendations to keep in mind to improve operability and restartability of processes and to design steps.
7.4.1 Process general design checklist

These are the most important guidelines and recommendations to improve process design: Do not implement incremental commit unless absolutely necessary. It is better to somehow decrease the size of the unit of work and if necessary run multiple steps.
302
During design and testing, satisfy yourself that the steps are truly restartable. This should be a given for SQL steps, but check them out, and also review any UDPs, utilities, or other steps. Keep the unit of work to a reasonable size for the system you are running on (for example, hundreds of thousands of rows on Windows, but not millions; a few millions of rows on UNIX, but not billions)
7.4.2 SQL steps

An SQL step will generate select from the source(s) and insert into a target table:
Scenario where source and target tables are on different databases: When the source and target tables are in different databases (data will flow through the agent), the agent will execute the select (via a cursor fetch) at the source database. Then the agent will execute insert statements at the target database. It is best to have the agent collocated with the target database to take advantage of DB2 block fetches and local inserts Scenario where source and target tables are in the same database: When the source and target tables are in the same database, an INSERT INTO SELECT FROM statement is generated. The agent sends the statement to the database, and all data movement will be within the database engine and will not flow through the agent. Commit count cannot be used.
For any SQL steps which are likely to be extracting from a set of large DB2 tables, it is useful to list the SQL generated, and copy and paste each set of SQL to a special directory (collection) of flat files. Later, these can be run against the SQL Index Advisor and also SQL Explain to help ensure good performance of the system. Additionally, special measures have to be taken if you want to have a target table which exploits DB2's NOT NULL WITH DEFAULTS capability.
7.4.3 UDPs
A user-defined program represents an application that you want the Data Warehouse Center to start. For example, you can write a user-defined program that will perform the following process: 1. Export data from a table. 2. Manipulate that data. 3. Write the data to an interim output resource or a warehouse target.
303
If you write a UDP, you may want to ensure that any errors get signaled back to the DB2 Warehouse Manager error log, displayable through the Work In Progress menus. How to do this is documented in the DB2 UDB Data Warehouse Center Application Integration Guide, SC26-9994. An example is also provided in Appendix D, UDP to invoke SQL statements and get SQL codes back on page 345.
7.4.4 Transformers
Transformer class files are installed with DB2 UDB Server under sqllib/function/com/ibm/data/xf. If DB2 Warehouse Manager has been set up by a custom or standalone installation, transformer class files are not created. Before using transformers and enabling target for transformers, it is important to follow this users checklist: 1. Install a warehouse agent. 2. Install JDK 1.1.8. 3. Update the environment variables as specified in the DB2 Warehouse Manager Installation Guide, GC26-9998. 4. Update the database manager configuration for the target DB2 instance. 5. Update the database configuration for the target database. 6. Enable target for transformers and install transformers. 7. After performing the above steps, if you are still having problems: a. Check that VWS_PATH has been set: By default, this is set to \sqllib on Window NT. By default, this is set to $instance/sqllib on UNIX.
b. Look for transformer class files in:
sqllib/function/com/ibm/data/xf on Windows NT. $instance/sqllib/function.com on UNIX.
c. Look for files DropXfSQL and CreateXfSQL in sqllib/bin. d. Look for the messages file Xf.Properties in sqllib/function.
To use the SQL transformer within Data Warehouse Center:

1. Drop one or more source table objects on the palette. 2. Drop an SQL Transform object on the palette. 3. Draw a Data Link object between the Source objects and the SQL Transform object. 4. Double-click the SQL Transform object. 5. Alternately, right-click the SQL Transform object and select Properties.
304
7.5 Specific application hints and tips

In this section we provide hints and tips on various issues regarding: Data exception Summary records update
7.5.1 How to set up data exception with DB2 Warehouse Manager

Data exception processing capability is provided by the DB2 database engine and the load utility, and is used within DB2 Warehouse Manager. There is some built-in capability inherent in the Load Utility that creates a file for records that cannot be loaded, and there are exception tables for rows that violate either a referential constraint or a check constraint. If these functions are unable to handle the logic required for the application, then it is necessary to use triggers or write a stored procedure to do the exception analysis. Stored Procedures can be written in C/C++,COBOL, or REXX. Here are some examples of various implementations: 1. Data validation: The data is valid from the viewpoint of data types for example numeric values contain only valid numerical values, 0 to 9, + or (as a sign), and decimal point. When using the DB2 build data loader and it tries to load a record in which there is a basic problem with the validity of the data type or for some other basic reason it cannot be loaded then the loader can put the bad rows into a reject file. It will then continue processing. 2. Check constraints: The data is appropriate based on its value, according to some type of value constraint; for example, a number has to be positive or within a range, or a value has to be one of a set of values. When a table is created or altered, the check constraints on each column can be defined. A check constraint can basically be thought of as a WHERE clause. And, since this is done within the database engine, it is enforced for all insert and update activity. 3. Referential constraint: A value in a table must exist as a key in another table. For example, a cable operator must exist in a table operator table. Just as with the check constraints, referential constraints are defined on the table and enforced by the database engine. Violations will be put into an exception table as with check constraints.
Attention: It is sometime preferable to validate Referential Integrity during the transformation steps. This will allow faster loading, using the DB2 Load utility, since Referential Integrity will not be done by the database engine, which can have a negative impact on the performance for a large amount of data.
305
4. No duplicate values in certain fields: Just as with check constraints and referential constraints, a unique constraint can be defined on a column and enforced by the database engine. Again, violations will be put into the exception table. If all of these constraints cannot handle the application design scenario, a stored procedure can be used to do the validation of the data and log the results to the exception table, or do whatever logic needs to done.
7.5.2 How to update summary records (incremental aggregation)

Updating of summary records can be accomplished with Automatic Summary Tables which can be updated from the base table via a refresh command or automatically as changes are made to the base table. Replication can also be used to apply delta changes to aggregate tables. Triggers can be written to update summaries or aggregates and stored procedures can be written to provide special logic for incremental aggregation.
7.6 General performance advice

Performance can be measured in two ways:
Response time: Performance is the measure of time spent on an individual operation or a job. Capacity planning or throughput: Performance can be measured in terms of transactions per hour. It can be measured in terms of data transfer rates or resource utilizations, such as CPU utilization or disk storage utilization.
There are many factors which can impact performance in the DB2 Warehouse Manager environment, such as: High resource utilization: CPU, memory, and disk Application design considerations when using stored procedures, UDPs, etc. Object contention Speed of communication links Congestion of network resources Processor speed of the client system In the next sections we focus on: High resource utilization: CPU, memory, and disk Stored procedure performance considerations Load scalability considerations Running application performance checklist
306
7.6.1 High resource utilization: CPU, memory, and disk

When the CPU utilization rate is high (above 70% or 80%), or when paging begins to rise, or when some disks give a bad average response, the server may experience slowdowns. Several tools are available on different operating system platforms to determine the CPU utilization rate, the disk response time, or the paging activity. For example on iSeries system these tools include Control Language commands as: Work with System Status (WRKSYSSTS) command Work with Active Jobs (WRKACTJOB) command Work with Disk Status (WRKDSKSTS) command Figure 7-1 shows the similar commands and utilities on the pSeries (AIX) and the SUN Solaris systems.
Table 7-1 UNIX commands
Command pSeries (AIX) Solaris
System Activity Reporter Virtual Memory Statistics Disk Information
sar vmstat bootinfo -s hdisk#
sar vmstat format -d c#t#d# format>current format>inquiry vxprint -dl vxprint -vl prtconf sbin/prtdiag
List Physical Volume List logical Volume Physical RAM Diagnostics
lspv lslv bootinfo -r diag
7.6.2 Overall system-wide performance

You should observe and balance the overall system-wide performance before focusing on a specific performance problem. The specific component performance may only be a relatively small part of the overall performance in a distributed environment. For example, If the entire system is functioning poorly, even if no warehouse server is started, there is no reason to focus on DB2 Warehouse Manager performance the first time.
307
Installing warehouse components on systems which are already used to run a line of business applications should be done with a special care. If the average CPU utilization rate is already high (for example over 50%) or disk usage is high (for example 40%), the additional load created by these components may lower the performance of existing applications.
Jobs consuming CPU

If the warehouse server uses a lot of CPU time at certain times, it may be because of some agents or server tasks.The NT/2000 administrator should be able to tell what tasks were running at a given time. If you are experiencing performance problems, list the warehouse manager tasks that were running at those times to help with your analysis. A performance problem can be caused by one task that requires a lot of resources (for example CPU) during a long time. Poor performance can also be caused by an excessive number of tasks running at the time, competing for the same resources. In both cases, changing the run attributes of a job can be a solution. These job attributes are handled differently on different operating systems.
Program malfunctioning
A performance problem may also be the consequence of a malfunction (program loop, for example). A job may use a lot of resources without any sensible reason, and you will need to look for the cause of this problem.
7.6.3 Stored procedures performance considerations

To improve the performance of stored procedures, consider implementing one or more of the following techniques: Using VARCHAR parameters instead of CHAR parameters Forcing DB2 to look up stored procedures in the system catalogs Catalog the stored procedure as a NOT FENCED stored procedure
Using VARCHAR parameters instead of CHAR parameters

You can improve the performance of your stored procedures by using VARCHAR parameters instead of CHAR parameters. Using VARCHAR data types instead of CHAR data types prevents DB2 from padding parameters with spaces before passing the parameter, and decreases the amount of time required to transmit the parameter across a network.
308
For example, if your client application passes the string A SHORT STRING as a CHAR(200) parameter to a stored procedure, DB2 has to pad the parameter with 186 spaces, null-terminate the string, then send the entire 200 character string and null-terminator across the network to the stored procedure. In comparison, passing the string A SHORT STRING as a VARCHAR(200) parameter to a stored procedure results in DB2 simply passing the 14 character string across the network.
Forcing DB2 to look up stored procedures in system catalogs

When you call a stored procedure, the default behavior for DB2 is to search the sqllib/function and sqllib/function/unfenced directories for a shared library with the same name as the stored procedure before looking up the name of the shared library for the stored procedure in the system catalog. Only stored procedures of PARAMETER TYPE DB2DARI can have the same name as their shared library, so only DB2DARI stored procedures benefit from the default behavior of DB2. If you use stored procedures cataloged with a different PARAMETER TYPE, the time that DB2 spends searching the above directories degrades the performance of those stored procedures. To enhance the performance of stored procedures that are not cataloged as PARAMETER TYPE DB2DARI, set the value of the DB2_STPROC_LOOKUP_FIRST registry variable to ON. This registry variable forces DB2 to look up the name of the shared library for the stored procedure in the system catalog before searching the above directories. To set the value of the DB2_STPROC_LOOKUP_FIRST registry variable to ON, issue the following command from the CLP: db2set DB2_STPROC_LOOKUP_FIRST=ON
NOT FENCED stored procedures

Your stored procedure can run as either a FENCED or a NOT FENCED stored procedure, depending on whether you register the stored procedure as FENCED or NOT FENCED in the CREATE PROCEDURE statement. A NOT FENCED stored procedure runs in the same address space as the database manager (the DB2 Agent's address space). Running your stored procedure as NOT FENCED results in increased performance when compared with running it as FENCED because FENCED stored procedures, by default, run in a special DB2 process. The address space of this process is distinct from the DB2 System Controller.
309
Note 1: While you can expect performance improvements from running NOT FENCED stored procedures, user code can accidentally or maliciously damage the database control structures. You should only use NOT FENCED stored procedures when you need to maximize the performance benefits. Test all your stored procedures thoroughly prior to running them as NOT FENCED.
Note 2: If a severe error does occur while you are running a NOT FENCED stored procedure, the database manager determines whether the error occurred in the stored procedure code or the database code, and attempts an appropriate recovery.
7.6.4 Dynamically scale load processing

With the agent architecture, a step is assigned to an agent which is assigned to a machine and you can have multiple agent definitions pointing to one actual agent. The machine to which an agent definition is pointing can be changed by the administrator dynamically. With this capability to dynamically scale load processing to the data warehouse based on input volumes, you can define some arbitrary number of agents (for example 10) and spread the step assignments across all 10. When you want to add another machine to help process your steps, for whatever reason, you can change the definition of some of these agents (for example for half of them) to point to the physical agent on the new machine. Then all of the steps assigned to these 5 agents will automatically start executing on the new machine. You can choose between design alternatives in which technology you implement your steps. For example: when loading a flat file into your warehouse, you have choices of what kind of technology to use. The ODBC text driver is simpler to implement but is a poor performer compared to the DB2 load utility which also gives you some additional capability. If you design your workflow in smaller, discrete steps then you have the option of taking care of any bottlenecks without impacting or redoing your entire workflow. You also have to be concerned with the optimization of the performance of the database engine on which your warehouse is based. Typical database tuning techniques apply.
310
7.7 Performance checklist when testing applications

Here we provide a performance checklist to keep in mind when testing your applications: If you have a database with many tables, why not consider creating multiple warehouse sources/targets on that same physical database and then importing a subset of tables into each warehouse definition? By doing this, you will limit the number of metadata objects the GUI needs to retrieve from the warehouse control database each time a user want to add objects to a process or list objects in a source or target. For example, if you have a source database that has 5000 tables, you may want to consider creating 5 or more warehouse sources and then logically grouping 1000 tables into each warehouse. This way, when the user wants to use one of these source tables, and using the add table function of the process modeler, the user will see 5 warehouse sources and choose one to display all the tables in that source, rather than having to retrieve and look through 5000 tables each time. If you ever did any tracing of ODBC, CLI or similar, are any of those trace settings still enabled? On one occasion this improved performance by a factor of 15. If data is being refreshed from outside, would using the DB2 load utility in your process improve the speed? The load utility has been seen in some instance to be 10 times the speed of other methods. Is there any unnecessary creation of intermediate tables or use of multiple SQL steps where you might be able to use one SQL step? What tuning (if any) of the target database has been conducted which would help the performance of the warehouse runs? Are you performing a complete refresh of a data table when it would be far more efficient performance-wise to just update the changed data? What indexes (if any) have been applied to any source tables? Would indexes aid the performance? What indexes (if any) have been applied to the target tables? Are these having a very detrimental effect on the performance of loading the data into that table? What parallelism is there in the hardware and what does the software configuration allow you to exploit that you are not currently exploiting? Have you conducted an examination of your processes to discover how much elapsed time and resource the top 10 processes take, for example?
311
The 80:20 rule often applies, meaning that 80 percent of the resource is consumed by 20% of the processes. Understanding this can save a lot of work by allowing you to concentrate on those processes which are the most likely to be improved upon. Do you have any administration tasks scheduled in your warehouse processes to check what tables might need REORGing? Do you have any processes which issue RUNSTATS to update the table statistics in the catalog?
312
Appendix A.
Using DB2 Warehouse Manager Connector for the Web

This appendix describes how to use the DB2 Warehouse Manager Connector for the Web, and how to integrate it through Data Warehouse Center, based on a presentation given at the IDUG 2001 conference in Florence. It also provides recommendations when starting to implement the Web Connector.
A.1 Using Web Connector: step-by-step procedure

When starting to use the Web Connector, you need to perform the following steps using both the WebSphere Site Analyzer (WSA) user interface and the Data Warehouse Center user interface.
A.1.1 Using WebSphere Site Analyzer administration interface

In Figure A-1 we show how the WSA administrator user interface looks with: List of projects List of data imports (3 types) with associated current status. Log file data imports (for example, HTTP access log)
313
Database data imports that are specific traffic tables Web Tracker data imports
Figure A-1 Using WSA user interface
Before using the Web Connector in Data Warehouse Center, some preliminary steps need to be done in WSA: 1. Define a project. 2. Define at least one data import. 3. Run the extraction of at least one data import successfully. This step is necessary to create the Webmart tables. The detailed steps are illustrated in the following sections.
314
1. Define a WSA project (see Figure A-2): Project parameters are described in detail in the WSA Infocenter documentation. Website host names are the names of the servers that make up the Website (usually multiple machines). Webmart database can be stored in DB2 UDB or Oracle. The Webmart has to be a UNICODE (UTF-8) database and must be preconfigured in a certain way. This configuration procedure is documented in the WSA Infocenter as well.
Figure A-2 Defining a WSA project
Appendix A. Using DB2 Warehouse Manager Connector for the Web
315
2. Define a data import As mentioned previously, there are three types of data imports: Log file data import (see Figure A-3) Several industry standard log types are supported and the option also exists to define customizable ones.
Figure A-3 Defining a WSA log file data import
Database table data import (see Figure A-4)

At the moment WebSphere Application Server (WAS) and WebSphere Commerce Suite (WCS) traffic table are supported.
316
Figure A-4 Defining a WSA database table
Web Tracker data import (see Figure A-5) For near real-time analysis, the WSA Web Tracker feature allows you to collect traffic information or visitor behavior on-the-fly. There is one Web Tracker data import per project created automatically at project definition time. This data import cannot be removed.
Figure A-5 Defining a Web Tracker data import
317
Usually you would have only the Web Tracker data import and maybe some WebSphere Commerce Suite /WebSphere Application Server tables in the same project but no Web server log files to avoid redundancy. 3. Define scheduling and display notifications and status information for data imports in WSA a. Scheduling For log file and database table data imports, a schedule has to be set up from within WSA (see Figure A-6).
Figure A-6 Scheduling a log file data import
The Web Tracker data import cannot be scheduled. The feature can only be turned on or off, as shown in Figure A-5 on page 317.
318
b. Notifications A notification about the outcome of the data extraction in WSA can be set up for all data imports: see Figure A-7 for the log file and database notification example.
Figure A-7 Notification setup for log file data import
See Figure A-5 on page 317 for the Web Tracker data import notification example.
319
c. Extraction status The current status for all data imports can be obtained by clicking on the status icon in the WSA main window (see Figure A-8).
Figure A-8 Status for the log file data import extraction
If the Web Tracker feature is enabled, the status for the Web Tracker data import shows the timestamp of the last Webmart update made by the Web Tracker (see Figure A-9).
Figure A-9 Web tracker status monitor window
Note: The extraction should be run at least once.
320
A.1.2 Using the Data Warehouse Center user interface

The main steps to follow in Data Warehouse Center are: 1. Define a new WebSphere Site Analyzer warehouse source 2. Define a Web Traffic Polling step to check whether WSA Webmart is populated and up to date 3. Define a SQL step to schedule extracts from the Webmart in the warehouse target 4. Run steps using the Data Warehouse Center Work In Progress panel The detailed steps are illustrated below: 1. Define a new WebSphere Site Analyzer warehouse source: a. Use the new Data Warehouse Center source type (see Figure A-10).
Figure A-10 Introducing a new Data Warehouse Center source type
321
b. Define the WSA interface with this new source type (see Figure A-11).
WSA Webmart DB connection parameters
Connection/ Communication interface with WSA Server
Figure A-11 Defining the WSA interface
322
c. Select the data to use (see Figure A-12).
Webmart Views (user defined)
Data Imports (defined via WSA Web client)
Webmart Tables (generated by WSA)
slosh over items
Figure A-12 Selecting the data to use
d. Store the data selected (see Figure A-13).
Click OK to save
Figure A-13 Storing the data selected
323
2. Define a Web Traffic Polling step to check whether the WSA Webmart is populated and/or up-to-date (see Figure A-14).
Figure A-14 Introducing the Data Warehouse Center polling step
These are the steps to follow: a. Define the polling step parameters (see Figure A-15). The unit of polling is the data import. The warehouse source name menu only shows WSA source types. If no Data Imports were selected at source definition time, none will be shown. The step will always terminate successfully in this case.
324
Figure A-15 Defining the polling step parameters
b. Define the polling step processing options (see Figure A-16).
Figure A-16 Define the polling step processing options
325
3. Define an SQL step to schedule extracts from the Webmart (see Figure A-17). The Data Warehouse Center Process Model is used to model the data flow in Data Warehouse Center. The polling step tells the SQL step to run the extraction once the Webmart is populated or in other terms when the WSA extraction has finished. The step is successful when the clickstream assimilation process for the data imports selected as step input is completed otherwise a retry will be done and the step will fail if the duration is exceeded. A step will also terminate successfully, if the data imports are not scheduled from within WSA (that is inactive).
Source icon
'data' link
Polling step icon
SQL step copies data from source DB to target DB
'on success' link
Figure A-17 Using new source and step within the Process Model
326
4. Run steps using the DWC Work In Progress panel This is the step (see Figure A-18) to schedule and start running a step or process.
Figure A-18 Running the process from the Work In Progress window
A.2 Hints and tips for using the Web Connector

Implementation of the Web Connector requires 2 parts in both WSA and DB2 Warehouse Manager package: WSA API through a JSP interface that is the hook between WSA and DB2 Warehouse Manager Connector for the Web built on Java. The JSP interface: Retrieves the WSA metadata Project names Data import name Status of clickstream assimilation Overcomes the Web client or Java client mismatch DB2 Warehouse Manager new agent commands to handle the Import WSA source and the Polling step execution
327
The flow to import Webmart tables in the warehouse target is described in Figure A-19.
5 Web Client
Kernel
Agent
C O N
2
JSP Engine (WAS)
DWC - Admin client UI
Agent daemon
Agent
C O N
WebMart Admin DB
WSA
DB2 WM
1 - Collect WSA interface input via GUI 2 - Connector calls JSP (HTTP request) 3 - JSP retrieves WSA metadata from the Admin DB and encodes it 4 - Agent retrieves WebMart table/view definitions using data collected in 3 5 - Result of Connector program call is displayed on GUI
Figure A-19 Importing Webmart tables flowchart
Following is a summary of hints and tips you should know about when using the DB2 Warehouse Manager Connector for the Web:
Installation hints: Note: DB2 Warehouse Manager product needs to be installed before the DB2 Warehouse Manager Connector.
The installation prerequisites described in the IBM DB2 Warehouse Manager Installation Guide Version 7, GC26-9998-02 have changed. For Windows NT and 2000: After installing the DB2 Warehouse Manager Connector, you need only to restart Windows NT or 2000 system.
328
DB2 Warehouse Manager product installation automatically installs the Java Runtime environment software and sets CLASSPATH and PATH system environment variables. For AIX and SUN Solaris: Install the Java Developers Kit (JDK Version1.1.8 in the /usr/jdk_base directory When installing the connector feature, update the IWH.environment file Check the CLASSPATH environment variable as shown below:
CLASSPATH=/usr/jdk_base/ib/classes.zip:$(DB2DIR?)/bin/db2_vw_w eb.jar:$(DB2DIR?)/bin/ibmjsse.jar:$(CLASSPATH)
Optionally, to ensure that the agent will read the right executable set, change the path variable to:
PATH =/usr/jdk_base/bin:$(PATH) export PATH
Before defining warehouse source using Data Warehouse Center:
Check that the Webmart tables have been created. You create the tables by running the WSA data extraction process to import data for your project. After you run the WSA data extraction process, bring up the WSA Status Monitor by clicking on the status icon in the WSA user interface main window. The status monitor indicates whether the data extraction process ran successfully. If your data extraction process failed, take the actions indicated by the WSA error message. Check that values supplied in the WSA Project Information window match the corresponding values in the Data Warehouse Center Warehouse Source definition notebook. In the WebSphere Site Analyzer's Project Information window (see Figure A-2), ensure that: The Project name (www.idug.org in Figure A-2 on page 315) is identical to the Site name defined in the WSA Properties window (see Figure A-11 on page 322). The Database URL (SASITE in Figure A-2 on page 315) is the same as the Database name in the Data Warehouse Center warehouse source definition notebook (see Figure A-11 on page 322).
329
When running the Data Warehouse Center Web traffic polling step:
If an unexpected failure occurs when running the Data Warehouse Center Web traffic polling step, check that you have selected a warehouse source name in Data Warehouse Center when the Web traffic polling has been defined (see Figure A-15).
How to use WSA security and secure connections:
Before you can use DB2 Warehouse Manager Connector for the Web with WSA security and Secure Socket Layer (SSL), you must complete the following prerequisites:
a. Install latest Fixpack for DB2 Universal Database Version 7 b. Configure WebSphere Application Server to use security and SSL. Refer to the WebSphere Application Server documentation, available online through the Information Center, for instructions to set up security and SSL. c. Create a Web server keystore file called Serverkey.kdb that contains one or more client or server certificates. To do this, you can use the Java keytool or IBM iKeyman (which is part of the WebSphere Application Server package). d. Configure the Web server used by WebSphere Application Server for SSL using the keystore that you created in Step c. The default port number is 443.
To use Warehouse Manager Connector for the Web with WSA security and Secure Socket Layer (SSL), follow these steps:
a. Use the Java keytool to export the client and server certificates from the server keystore file Serverkey.kdb. b. Use the Java keytool to create a new keystore file called jssecacerts. Import all the certificates that you exported in the previous step into the jssecacerts keystore file. c. Copy the newly created keystore file jssecacerts into the following directory:
<warehouse manager agent dir> \java\java12\jdk\jre\lib\security

where warehouse manager agent dir is the directory where your warehouse manager agent is installed. d. After you enable WebSphere security, restart the WSA application servers. When you define your warehouse source in Data Warehouse Center, specify 443, (the default), as the port number in the WebSphere Site Analyzer Source Database page and enter the correct user ID and password to connect to the WSA server, as shown in Figure A-11 on page 322.
330
Appendix B.
Using DB2 Warehouse Manager Connector for SAP

This appendix describes how to use DB2 Warehouse Manager Connector for SAP R/3 and how to integrate it through Data Warehouse Center. It also provides recommendations when starting to implement the SAP R/3 connector.
B.1 Using SAP R/3 connector: step-by-step procedure

When starting to use the SAP R/3 Connector, you need to perform the following steps using the Data Warehouse Center user interface: The main implementation steps are: 1. Define SAP R/3 as a new warehouse source After SAP R/3 connector installation, Data Warehouse Center proposes also SAP as a possible warehouse source (see Figure B-1).
331
Figure B-1 Defining SAP R/3 as a new warehouse source
332
The steps to define the new SAP R/3 warehouse source are: a. Specify the right type of connection: The connection type can be Application server (see Figure B-2).
Figure B-2 Defining application server connection type
Appendix B. Using DB2 Warehouse Manager Connector for SAP
333
The connection type can also be Group of servers (see Figure B-3).
Figure B-3 Defining group of server connection type
When the connection fails, the following error message may appear:
DWC07356E An agents processing of a command of type importSAPBONames failed for edition0of step?. RC=7356 RC2 =8455
334
b. Select SAP R/3 Business Objects (see Figure B-4).
Figure B-4 Selecting SAP R/3 Business Objects
335
c. Define SAP R/3 as a warehouse source (see Figure B-5).
Figure B-5 Defining SAP R/3 as a warehouse source
336
d. Define properties of Business Objects: By selecting metadata (parameters) (see Figure B-6)
Figure B-6 Defining the parameters of the Business Object selected
337
By mapping the parameters (see Figure B-7)
Figure B-7 Mapping the Business Object parameters
338
2. Define the SAP R/3 process using DWC process model A new process is defined using Data Warehouse Center Process Modeler (see SalesOrder process in Figure B-8).
Figure B-8 Defining a SAP R/3 process
339
SAP R/3 Data Extract step properties need to be defined following these steps: a. Select Business Object input parameters (see Figure B-9).
Figure B-9 Selecting Business Object input parameters
340
b. Select output parameters to be mapped to columns (see Figure B-10).
Figure B-10 Selecting SAP R/3 output parameters to be mapped to columns
Column mapping and processing options will also be defined when defining the properties of the SAP R/3 step. 3. Run the step using the DWC Work In Progress panel The new process previously defined SalesOrder Process will be launched using the Data Warehouse Center Work In Progress panel and results will be checked in the target warehouse named SalesOrder_All.
341
B.2 Hints and tips for using the SAP R/3 Connector
The following is a summary of hints and tips you should know about the DB2 Warehouse Manager Connector for SAP R/3:
Performance issues with Windows 2000:
When you are using Windows 2000 and import a business object from SAP, the RSVP.EXE process is automatically run. This process uses all of the processor resources. To avoid this: a. Click Start --> Settings --> Control Panel. b. Double-click the System icon. Click the Advanced tab. c. Click the Environment Variables tab. In the System variable field, click the New button. In the Variable Name field, type: QOSDISABLE In the Variable Value field, type: 1 Click OK
Now run your import process.

Performance issue with SAP R/3:
To avoid getting the error Time Limit Exceeded during SAP R/3 Data Extract step processing, that means that the step has exceeded the maximum permitted run time on SAP R/3, ask your SAP System Administrator to change the default value for the system profile parameter rdisp/max_wprun_time from 300 seconds (default value) to 600 seconds or more.
Getting problems to access certain Business Object:
This problem may occur when the key field for the business object selected does not have a corresponding GetList export parameter (Refer to Figure B-7 on page 338). Without this export parameter, parameter mapping cannot be completed. Export parameters, which provide the details about their corresponding key fields, are required to run GetDetail. When parameter mapping cannot be completed, you can run GetList for your business object, but you cannot run GetDetail, which provides specific details about the key field that you have selected. On the other hand, incorrect results in your GetDetail report are the result of mapping a key field to the wrong export parameter.
342
Appendix C.
Recovering the Information Catalog Manager database

To avoid losing your data in case of a hardware or software failure, establish a routine for backing up your Information Catalog Manager database, configuration information, and supporting software. How frequently you back up these components depends on how frequently you make changes to your information catalog, and on your organizations policies for backups.
C.1 Information Catalog Manager database

Backing up Information Catalog Manager (ICM) databases is crucial to ensure that you can recover your descriptive data if your databases become inconsistent or corrupt.
343
You can use either of two methods to back up the ICM database:
Use the DB2 UDB BACKUP utility: To back up the ICM database, you can use the standard procedures for DB2 backup and recovery. Export all of the data into tag language files: You can export the ICM objects and object types in the form of tag language files from your ICM database. You can also import the Information Catalog Manager tag language files into your ICM database.
C.2 ICM configuration information

ICM creates some configuration information on administrators and users workstations. The stored information consists of user-specific data, such as collections and saved searches for a particular information catalog. The Information Catalog Manager for Windows stores this information in the system registry. Enter REGEDIT at an MS-DOS prompt to view the Windows system registry in the Windows Registry Editor. To locate the working directory for the Information Catalog Manager for Windows, check the DGWPATH environment variable. (On Windows NT, you can find the environment variable on the Control Panel under Environment Variables. On Windows 95, you can find the environment variable in your AUTOEXEC.BAT file.) Then, search for the working directory name in the Registry Editor. One branch, or folder, in the registry has the suffix INI for each information catalog that you access. One folder contains wildcard settings, the last user ID that logged on, and the last information catalog that was used. Your administrator folder also includes default export settings. The online help for the Registry Editor explains how to import and export selected branches or how to restore the entire registry. Whenever there are any major changes or additions to your information catalog, you should back up your, and all your users, Information Catalog Manager configuration information.
344
Appendix D.
UDP to invoke SQL statements and get SQL codes back

When running warehouse processes, it is often useful to run steps which check the status of the warehouse or compare the content or structure of the warehouse data with data that is held in warehouse control tables. If things do not check out correctly, then often you will want to stop the warehouse loading or manipulation process and take corrective action. The chief requirements are: To be able to run some statement which will check out the content or structure of the data and return a zero return code if the check is satisfactory To be able to return a non-zero return code if unsatisfactory, while at the same time delivering a textual explanation in the warehouse server log In addition to this, there is often the requirement to issue direct update or delete statements via DB2 Warehouse Manager. To this end, we wrote a sample C program (which currently has only been tested on Windows) which, as one of its parameters, uses an SQL statement. This statement can be a SELECT, INSERT, UPDATE or DELETE (although INSERT is handled by warehouse server SQL Builder, so probably it would not need to be used).
345
The program runs as a User Defined Program (UDP), and if everything works satisfactorily, it returns a zero return code. However, if an error of any kind is raised, it will obtain the SQL error message and ensure that this is signalled back to the warehouse server. In turn this is available to be displayed in the error log (through the Work In Progress menus).
D.1 The UDP source code

The source draft code is provided in Example D-1.
Example: D-1 Source program
/*****************************************************************************/ /* */ /* Name: execsql.c - Command line program to execute an SQL statement */ /* against DB2. */ /* */ /* It returns the SQL return code as the program exit code, and */ /* writes additional error info into the log file specified in the */ /* VWP_LOG environment variable. This program is intended to be */ /* used in conjunction with DB2 Warehouse Manager. */ /* */ /* Syntax is: execsql <database> <userid> <password> <SQL> */ /* */ /* */ /* (C) Copyright IBM Corp, 2001 */ /* */ /* Change log: */ /* Defect Date Who Description */ /* 14/05/01 SEM Created. */ /* */ /*****************************************************************************/ /*---------------------------------------------------------------------------*/ /* Includes. */ /*---------------------------------------------------------------------------*/ #include <stdio.h> #include <sqlcli.h>
/*---------------------------------------------------------------------------*/ /* Function declarations. */ /*---------------------------------------------------------------------------*/ void processErrorInfo(FILE * pLogFileName, SQLRETURN clirc, SQLSMALLINT handleType, SQLHANDLE handle, SQLINTEGER * pSQLCode);
346
/*---------------------------------------------------------------------------*/ /* Main function. */ /*---------------------------------------------------------------------------*/ int main(int argc, char *argv[ ], char *envp[ ]) { char * pDatabase; /* Ptr to DB name string. */ char * pUserid; /* Ptr to User ID string. */ char * pPassword; /* Ptr to password string. */ char * pSQLString; /* Ptr to the SQL to execute. */ char * pLogFileName; /* Ptr to the name of log file. */ FILE * pLogFile = NULL; /* Ptr to the log file. */ SQLHANDLE envHandle; /* CLI environment handle. */ SQLHANDLE connection; /* Handle to the connection. */ SQLHANDLE stmt; /* Handle to the SQL statement. */ int exitcode = 0; /* Final program exit code. */ SQLRETURN clirc; /* Return code form CLI calls. */ SQLINTEGER sqlCode; /* SQL status return code. */ SQLSMALLINT colCount; /* Count of columns in a result set. */
/*-------------------------------------------------------------------------*/ /* Try to get the name of the file to log messages to from the environment */ /* variable, and open the log file. */ /*-------------------------------------------------------------------------*/ pLogFileName = getenv("VWP_LOG"); if (pLogFileName != NULL) pLogFile = fopen(pLogFileName, "w"); /*-------------------------------------------------------------------------*/ /* Validate the command line arguments. */ /*-------------------------------------------------------------------------*/ if ((argc != 5) || (argv[1] == NULL) || (argv[2] == NULL) || (argv[3] == NULL) || (argv[4] == NULL)) { if (pLogFile != NULL) fprintf(pLogFile, "Invalid number of command line arguments.\n Usage: execsql <database> <userid> <password> <SQL>\n"); exitcode = -1; return exitcode; } else { pDatabase = argv[1]; pUserid = argv[2]; pPassword = argv[3]; pSQLString = argv[4]; }
Appendix D. UDP to invoke SQL statements and get SQL codes back
347
/*-------------------------------------------------------------------------*/ /* Init the DB2 CLI environment. */ /*-------------------------------------------------------------------------*/ clirc = SQLAllocHandle(SQL_HANDLE_ENV, SQL_NULL_HANDLE, &envHandle); if (clirc == SQL_SUCCESS) { /*-----------------------------------------------------------------------*/ /* Allocate a connection handle. */ /*-----------------------------------------------------------------------*/ clirc = SQLAllocHandle(SQL_HANDLE_DBC, envHandle, &connection); if (clirc == SQL_SUCCESS) { /*---------------------------------------------------------------------*/ /* Connect to the database. */ /*---------------------------------------------------------------------*/ clirc = SQLConnect(connection, pDatabase, (SQLSMALLINT)strlen(pDatabase), pUserid, (SQLSMALLINT)strlen(pUserid), pPassword, (SQLSMALLINT)strlen(pPassword)); if (clirc == SQL_SUCCESS) { /*-------------------------------------------------------------------*/ /* Allocate a statement handle. */ /*-------------------------------------------------------------------*/ clirc = SQLAllocHandle(SQL_HANDLE_STMT, connection, &stmt); if (clirc == SQL_SUCCESS) { /*-----------------------------------------------------------------*/ /* Execute the SQL. */ /*-----------------------------------------------------------------*/ clirc = SQLExecDirect(stmt, pSQLString, SQL_NTS); if (clirc == SQL_SUCCESS) { /*---------------------------------------------------------------*/ /* If the SQL produced a result set, then try to fetch the first */ /* row. */ /*---------------------------------------------------------------*/ clirc = SQLNumResultCols(stmt, &colCount); if (clirc == SQL_SUCCESS) { if (colCount > 0) { clirc = SQLFetch(stmt); if (clirc != SQL_SUCCESS) { processErrorInfo(pLogFile, clirc, SQL_HANDLE_STMT, stmt, &sqlCode); exitcode = sqlCode; }
348
} } else { processErrorInfo(pLogFile, clirc, SQL_HANDLE_STMT, stmt, &sqlCode); exitcode = sqlCode; } } else { processErrorInfo(pLogFile, clirc, SQL_HANDLE_STMT, stmt, &sqlCode); exitcode = sqlCode; } /*-----------------------------------------------------------------*/ /* Free the statement. */ /*-----------------------------------------------------------------*/ clirc = SQLFreeHandle(SQL_HANDLE_STMT, stmt); if (clirc != SQL_SUCCESS) { processErrorInfo(pLogFile, clirc, SQL_HANDLE_STMT, stmt, &sqlCode); exitcode = sqlCode; } } else { processErrorInfo(pLogFile, clirc, SQL_HANDLE_DBC, connection, &sqlCode); exitcode = sqlCode; } /*-------------------------------------------------------------------*/ /* Disconnect from the database. */ /*-------------------------------------------------------------------*/ clirc = SQLDisconnect(connection); if (clirc != SQL_SUCCESS) { processErrorInfo(pLogFile, clirc, SQL_HANDLE_DBC, connection, &sqlCode); exitcode = sqlCode; } } else { processErrorInfo(pLogFile, clirc, SQL_HANDLE_DBC, connection, &sqlCode); exitcode = sqlCode; }
349
/*---------------------------------------------------------------------*/ /* Free the connection handle. */ /*---------------------------------------------------------------------*/ clirc = SQLFreeHandle(SQL_HANDLE_DBC, connection); if (clirc != SQL_SUCCESS) { processErrorInfo(pLogFile, clirc, SQL_HANDLE_DBC, connection, &sqlCode); exitcode = sqlCode; } } else { processErrorInfo(pLogFile, clirc, SQL_HANDLE_ENV, envHandle, &sqlCode); exitcode = sqlCode; } /*-----------------------------------------------------------------------*/ /* Free the environment handle */ /*-----------------------------------------------------------------------*/ clirc = SQLFreeHandle(SQL_HANDLE_ENV, envHandle); if (clirc != SQL_SUCCESS) { processErrorInfo(pLogFile, clirc, SQL_HANDLE_ENV, envHandle, &sqlCode); exitcode = sqlCode; } } else { if (pLogFile != NULL) fprintf(pLogFile, "Couldn't allocate DB2 CLI environment handle.\n"); exitcode = clirc; }
/*--------------------------------------------------------------------------*/ /* Close the log file, if there is one open. */ /*--------------------------------------------------------------------------*/ if (pLogFile != NULL) fclose(pLogFile); return exitcode; }
350
/*---------------------------------------------------------------------------*/ /* Function to get detailed error information, and print it to the log file. */ /*---------------------------------------------------------------------------*/ static void processErrorInfo( FILE * pLogFile, /* Ptr to file to write log info to. */ SQLRETURN clirc, /* The return code from CLI. */ SQLSMALLINT handleType, /* Type of handle. */ SQLHANDLE handle, /* Handle to get error info from. */ SQLINTEGER * pSQLCode) /* Ptr to var for returning SQL code. */ { char message[SQL_MAX_MESSAGE_LENGTH]; char sqlstate[5]; SQLINTEGER sqlcode; SQLSMALLINT length; SQLSMALLINT i = 1; int warning = 0; /* Set the returned SQL code to success, by default. */ *pSQLCode = 0; /* Special processing for SQL_NO_DATA_FOUND, as this does not put any */ /* diagnostic records on the handle. */ if (clirc == SQL_NO_DATA_FOUND) { /* Set the SQL code to 100 and the SQLState to 02000. */ sqlcode = 100; strcpy(sqlstate, "02000"); /* Return the SQL code. */ *pSQLCode = sqlcode; /* If there's a log file available, write to it. */ if (pLogFile != NULL) { fprintf(pLogFile, "<RC>%ld</RC>\n", sqlcode); fprintf(pLogFile, "<MSG>SQL0100W No row was found for FETCH, UPDATE or DELETE; or the result of a query is an empty table.</MSG>\n"); fprintf(pLogFile, "<WARNING>1</WARNING>\n"); fprintf(pLogFile, "<SQLSTATE>%s</SQLSTATE>\n", sqlstate); } } else { /* If CLI returned SQL_SUCCESS_WITH_INFO then treat it as a warning. */ if ((clirc == SQL_SUCCESS_WITH_INFO) || (clirc == SQL_NO_DATA_FOUND)) warning = 1;
351
/* Loop through all the messages for the handle. */ while (SQLGetDiagRec(handleType, handle, i, sqlstate, &sqlcode, message, SQL_MAX_MESSAGE_LENGTH + 1, &length) == SQL_SUCCESS) { /* Return first SQL return code to caller, unless we're processing a */ /* warning, in which case it's just left as success. */ if ((i == 1) && !warning) *pSQLCode = sqlcode; /* If there's a log file available, write to it. */ if (pLogFile != NULL) { fprintf(pLogFile, "<RC>%ld</RC>\n", sqlcode); fprintf(pLogFile, "<MSG>%s</MSG>\n", message); if (warning) fprintf(pLogFile, "<WARNING>1</WARNING>\n"); fprintf(pLogFile, "<SQLSTATE>%s</SQLSTATE>\n", sqlstate); } i++; } } }
Remember that this code is currently Windows only so it will require to run via a Windows agent. If you have to run from agent in UNIX then you will need to re-compile and make the file under UNIX.
D.2 Procedure to use the UDP under Windows NT/2000

To illustrate the use of this sample EXECSQL code, we show below a very simple example which will check that there are 9 rows in the sample table ORG. This is one of the tables loaded in the DB2 UDB SAMPLE database. This is not highly realistic but it does show the flexibility of the solution and how you can provide your own error messages back into the warehouse server.
352
Other uses of the EXECSQL code which come to mind are: To check that timestamps in one or more warehouse or mart tables conform to timestamps held in a control table To check that the number of rows in a recently updated table are equal to the number of rows before the process started plus the number of rows in the original input table To check that all the values of a particular column in a fact table are contained in the relevant dimension table To use this EXECSQL code: Register it as a User Defined Program with parameters of database, userid, password and SQL string, all of which are character strings (see Figure D-1).
Figure D-1 Define the UDP
Define parameters of database, userid, password and SQL string, all of which are character strings. The character strings have to be enclosed in double quotes so SAMPLE should be SAMPLE (see Figure D-2).
353
Figure D-2 Define UDP parameters
Define the process to use this UDP (see Figure D-3)
Figure D-3 Define the process
354
Supply the values for database, userid, password and the SQL string to be executed. Figure D-4 shows the use of RAISE_ERROR to help get your own messages into the DB2 Warehouse Manager log. This example will check the number of rows in a table ORG and give an error if there are not 9 rows.
Figure D-4 Process values
355
Promote the step and test it using the Work in Progress mechanism. If the SQL string executes with a return code of 0 then the step will be successful. If for some reason the SQL fails (this can be intentional for example you can use the RAISE_ERROR function within an SQL statement) then the EXECSQL code will return a none zero return code equal to the SQL code. Additional information will be supplied in the DB2 Warehouse Manager log, viewable from the Work in progress window (see Figure D-5).
Figure D-5 Run the UDP process
356
Appendix E.
Moving from test to production environment

This appendix describes the procedures to get a DB2 Warehouse Manager test and production environment running simultaneously. In this appendix, we describe: The various options for test to production initial migration, including the steps to create a test environment Test to production incremental migration
E.1 Test to production initial migration

Here we discuss the various options to migrate from DB2 Warehouse Manager Test environment to the DB2 Warehouse Manager Production Environment: Demoting and copying the steps Copying steps and moving to a different subject area Changing the database definition Creating the test environment
357
E.1.1 Demoting and copying the steps

To migrate from test to production, follow these steps: 1. Demote the DB2 Warehouse Manager step to development and remove the link connecting the step to the Test database target/source. 2. In the SQL steps Column Mapping section, define the target table in the Production database with the same schema, table name and tablespace as the target table in the Test database. 3. Copy the SQL create table statement from the Test database table and paste the SQL in the new Production database target. DB2 Warehouse Manager will not allow Test database target to be removed because steps are still using these tables.
E.1.2 Copying steps and moving to a different subject area

This process will yield the same results as above. The copied process will have to be migrated to the Test or Production database. Follow these steps: 1. Expand the warehouse Subject Area in the Data Warehouse Center. 2. Expand and select a process. 3. Right-click the steps you want to copy. 4. Copy the step to a new process. 5. Now, right-click the process and click Move. 6. Move the process into a new subject area.
E.1.3 Changing the database definition

This option is recommended over the two options discussed above. Changing the database definition of the Test database to point to production database is the best solution. In order to avoid demoting all the steps to development, tools like db2move and ERWin should be used to create the tables in the production environment before migration.Tables with a name of less than length of 18 can be reverse engineered and created via ERwin. Data should be exported and imported for those tables with data that the programmers wish to keep in production. For tables with a name of greater than length of 18, db2move should be used: 1. To determine which tables have names greater than length of 18, use the following command on DB2 command line window:
db2 "select tabname from syscat.tables where tabschema in ('SCHEMA1','SCHEMA2') and length(tabname) > 18 order by tabname"
358
2. Run the command (db2move DBTEST export tn) listing a maximum of 10 tables after tn separated by commas:
db2move DBTEST export -tn BNFT_GP_DESC_LOOKUP, CLAIM_TYPE_DISTINCT, GRP_BAND_DESC_LOOKUP, HLTH_CTR_STAGING_SRC, NDC_FREQ_INCR_STAGING, NDC_SRC_INCR_STAGING, PLN_PROD_TYP_UPDATE_SRC, PROVIDER_INCR_STAGING
3. Issue the command db2move DBPROD import to replace_create the tables. This command must be issued from the same directory that the export files were created in. 4. After all SCHEMA1 and SCHEMA2 tables have been created in the production database, log in to Data Warehouse Center and locate warehouse targets. 5. Right-click Test Database Target and select Properties. 6. Change the Name to Production Database Target under the Warehouse Target tab. 7. Under the Database tab, change the Database name to DBPROD and click OK. All processes will now run in the production database. In addition to the warehouse target switch, the following must also be updated: The DATABASE parameter in all autoloader configuration scripts should be updated to DBPROD CONNECT TO DBPROD should be added to the top of all SQL scripts and TERMINATE should be added to the bottom Because the default connection database for the Warehouse ID is DBTEST, you should remember to CONNECT TO DBPROD when needing to query or change tables in production. When in doubt, issue the following command on a DB2 command window:
GET CONNECTION STATE
E.1.4 Creating the test environment

It is your database administrators responsibility to create a second warehouse control database on the warehouse server. This database, DWTESTDB, should be used for development and testing. To initially synchronize up the production and test warehouse control databases, DWCTRLDB and DWTESTDB respectively, the database administrators will restore a copy of the backup of DWCTRLDB to the test DB name, DWTESTDB.
Appendix E. Moving from test to production environment
359
The production warehouse control database, DWCTRLDB, must be the active warehouse control database on the warehouse server so that the production programs can run on its schedule. Since a server or workstation can have only one active warehouse control database at a time, the test warehouse control database has to run on its own server or workstation. The DWTESTDB warehouse control database will be initialized on the workstation by running Control Database Management. This program sets the active warehouse control database on the workstation to DWTESTDB allowing DB2 Warehouse Manager services to run. When you wish to connect to the test or production databases from your workstation, you proceed to logging in to the Data Warehouse Center. At the log in screen, you press Advanced button and enter the warehouse control database you wish to connect to: When connecting to the production warehouse control database, enter DWCTRLDB as the Control Database and production server name as the server host name. When connecting to the test warehouse control database, enter DWTESTDB as the Control Database and Test server name as the server host name. The server service name, vwkernel, does not change when connecting to either warehouse control database. Since both databases reside on the same server, there is no need to change the Server host name. Since the test warehouse control database is originally a restoration of the production warehouse control database, all of the programs point to the production database. To point the DWTESTDB database to the test database, follow the procedure outlined in Appendix E.1, Test to production initial migration on page 357.
E.2 Test to production incremental migration

Now that both warehouse control databases are created, there will be two methods to migrating processes on an ongoing basis. If there are a significant number of processes that need to be promoted to production or the processes are complex involving multiple steps, then the preferred method is to perform the same procedure as in the initial migration. Since only new tables will have to be added to the production database, db2move should be used to create these new tables and any tables whose data structure has changed.
360
Once the tables are created in the production database, the database administrators will restore a copy of DTESTDB to the production warehouse control database name, DWCTRLDB which will restore on top of the old production warehouse control database name, DWCTRLDB. Log in to the production warehouse control database, DWCTRLDB, and change the warehouse target to point to production database. Due to necessity to keep both databases in synchronization, the production jobs in both databases should be secured and have limited access.
E.2.1 Partial migration

DB2 WM provides a method to migrate a process, in case you do not wish to migrate the entire test warehouse control database and instead wish to do partial updates to the production warehouse control database. If you are using the import utility to move a warehouse source from a test environment to a production environment, make sure that the production environment does not already have a warehouse source with the same warehouse source name unless you want to over-write the definition of the warehouse source. If you import a step into a system that contains a step with the same name, then you must either delete the step that you want to overwrite, or change the step to development mode. Otherwise, the step cannot be updated and an error will occur. To migrate a process from the Data Warehouse Center: 1. Open Data Warehouse Center. 2. Highlight Warehouse (on left panel) --> and click Selected (top menu) --> Export Metadata --> Interchange File. 3. Filename box - click "..." to select path and enter filename for export file in filename box. 4. Select from left panel subject areas, sources, targets, and whatever you wish to migrate. 5. Click the ">" to move selection to the right panel. 6. Check the export dependent source properties and /or include process schedules if desired. 7. Click OK. 8. Start Data Warehouse Center, connecting to the production warehouse control database and repeat the steps above clicking on Import Metadata.
Appendix E. Moving from test to production environment
361
362
Related publications
The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook.
IBM Redbooks
For information on ordering these publications, see How to get IBM Redbooks on page 365.
Getting Started with Data Warehouse and Business Intelligence, SG24-5415 Business Intelligence Certification Guide, SG24-5747 Building the Operational Data Store on DB2 UDB, SG24-6513 Migrating to DB2 UDB Version 7.1 in a Visual Warehouse Environment, SG24-6107 IBM ESS and IBM DB2 UDB Working Together, SG24-6262 Implementing ESS Copy Services ON UNIX and Windows NT/2000, SG24-5757 Backing Up DB2 Using Tivoli Storage Manager, SG24-6247 WOW! DRDA Supports TCP/IP: DB2 Server for OS/390 and DB2 Universal Database, SG24-2212
Other resources
These publications are also relevant as further information sources:
DB2 UDB Data Recovery and High Availability Guide and Reference, DATA-RCVR DB2 UDB Troubleshooting Guide, GC09-2850 DB2 For Windows Quick Beginnings, GC209-2971 DB2 Warehouse Manager Installation Guide, GC26-9998 DB2 UDB Administration Guide: Planning, SC09-2946 DB2 UDB Administration Guide: Performance, SC09-2945 DB2 UDB Administration Guide: Implementation, SC09-2944 DB2 UDB Command Reference, SC09-2951
363
DB2 UDB Data Movement Utilities Guide and Reference, SC09-2955 DB2 UDB Data Warehouse Center Administration Guide, SC26-9993 DB2 UDB Data Warehouse Center Application Integration Guide, SC26-9994 DB2 Warehouse Manager Managing ETI Extract Conversion Programs with DB2 Warehouse Manager, SC27-1268 DB2 Query Patroller Installation Guide, GC09-2959 DB2 Query Patroller Users Guide, SC09-2960 DB2 Query Patroller Administration Guide, SC09-2958 Information Catalog Manager Users Guide, SC26-9996 Information Catalog Manager Administration Guide, SC26-9995 OLAP Setup and Users Guide, SC27-0702 OLAP Integration Server Model Users Guide, SC27-0783 OLAP Integration Server Metaoutline Users Guide, SC27-0784 OLAP Integration Server Administration Guide, SC27-0787 Data Mining with IBMs Intelligent Miner, GC27-1236 Intelligent Miner Scoring V7R1 Administration and Programming for DB2, SH12-6719 DB2 UDB Installation and Configuration Supplement, GC09-2957 DB2 UDB Message Reference, Volume 1, GC09-2978 DB2 UDB Message Reference, Volume 2, GC09-2979 Replication Guide and Reference, SC26-9920 SQL Reference, Volume 1, SC09-2974 SQL Reference, Volume 2, SC09-2975 AIX Version 4.3 Differences Guide, SG24-2014 DB2 UDB for z/OS and OS/390 ODBC Guide and Reference, SC26-9005 DB2 UDB for z/OS and OS/390 Installation Guide, GC26-9008 DB2 Enterprise Extended Edition for UNIX Quick Beginnings, GC09-2964 Program Directory for IBM DB2 UDB Server for z/OS and OS/390 DB2 Warehouse Manager, GI10-8221 Program Directory for IBM DB2 Warehouse Manager Sourcing Agent, GI10-8244
364
Referenced Web sites

These Web sites are also relevant as further information sources: IBM Business Intelligence Solutions Homepage
http://www-4.ibm.com/software/data/bi/
IBM Software Homepage

http://www.software.ibm.com/
IBM Database Management Homepage

http://www.software.ibm.com/data/
IBM DB2 Warehouse Manager Homepage

http://www-4.ibm.com/software/data/db2/datawarehouse/
IBM DB2 OLAP Server Homepage

http://www-4.ibm.com/software/data/db2/db2olap/
IBM DProp Homepage

http://www.software.ibm.com/data/dprop/
IBM DataJoiner Homepage

http://www.software.ibm.com/data/datajoiner/
IBM Data Management Performance

http://www.software.ibm.com/data/db2/performance
How to get IBM Redbooks

You can order hardcopy Redbooks, as well as view, download, or search for Redbooks at the following Web site:
ibm.com/redbooks
You can also download additional materials (code samples or diskette/CD-ROM images) from that site.
IBM Redbooks collections

Redbooks are also available on CD-ROMs. Click the CD-ROMs button on the Redbooks Web site for information about all the CD-ROMs offered, as well as updates and formats.
Related publications
365
366
Index
A
ABAP 26 actions unpredictable 37 adaptability 3 administration 170 administrative client 148 agent 4, 1617, 21 AIX 22 AS/400 23 daemon 17, 240 default 8, 17 distributed 89 iSeries 23, 250 local 17, 240 logs 35, 244, 248 messages 240 multiple agents 16, 310 OS/2 23 OS/390 23 remote 17, 240 SUN Solaris 22 Windows NT/2000 22 agents 5 AIX 5, 8, 23 API 72 application 231 design 287, 292, 306 restartable 288 architecture 20 layers 3 archived logs 40 AS/400 39 Ascential 7 assistance 286 automation 2, 13, 16, 140 AUTORESTART 45, 106 availability 3, 113, 289 continuous 121 average 11 AIX splitcopy 136 associated files 43 automation 173 DB2 UDB 174 Windows 173 buffer size 68 control files 48 cost 41 cumulative 52 database 63 delta 52 full database 40, 48, 52 image 63 incremental 51 information 60 management 140 methods 41 notebook 65 offline 41, 48, 52 online 41, 50 parameter files 48 partitioned database 87 pending state 85, 197, 203 performance 68 planning 41 procedures 39 restrictions 69 schedule 40, 47 speed 41 system catalog tables 48 tablespace 40, 50 user tables 48 using Data Warehouse Center 224 using different operating system 43 verification 142, 144 window 41, 119 wizard 65 bandwidth 52 BAPI 26 GetDetail 26 GetList 26 block fetching 19 BO 26 BOR 26
B
backup 36
367
browser 24 business data 26 events 26 rules 2, 26 transactions 26 Business Application Programming Interface 26 Business Object Repository 26 Business Objects 2627, 335
location 60 point-in-time 123 time 60 correlation 11 crash 33 recovery 44 CURRENT TIMEZONE 50 CUT 50, 200, 202 CWM 15
C
Call Level Interface 21, 272 capacity planning 306 cascading 105 catalog partition 88 chi-square 11 cleansing 2 name and address 29 CLI 21, 272 clickstream data 23 client administrative 5 Client Access 251 CLP 65, 71, 79, 154 cluster 100, 104, 122 collocated 303 Commerce data 23 commit 44 Common Warehouse Metamodel 15 communication 239, 306 concurrency 15 configuration 1920, 31 connectivity 21 Connector 9 connector 6 for SAP R/3 23, 26, 259, 331 tips 342 for the Web 23, 313 tips 327 consistency 42 bits 111 constraints 26, 290, 305 container 72, 119 deletion 181 error 207 resize 207 Coordinated Universal Time 50, 200 copy
D
data adding data from external sources 2 calculating 2 changes 52 cleansing 2, 5 consistency 26, 42 converting codes 2 corruption 112 damaged 42 exception 305 flow 20 geo-coding 2 loading 13 lost 42 movement 11, 13, 15, 88, 292 persistent 26 reconciled 2 relationships 4243, 51 transforming 13 data exception 305 data import 2425, 314 notification 319 data sources 5, 21 multiple 2 register 14 data stripe mirroring 114 data warehouse 2, 5 accessing 4 building 4 designing 34 governing 4 maintaining 4 multi-tiered 13 population 2 Data Warehouse Center 5, 8, 13, 16, 147, 151, 313, 321, 331 error message 259
368
error token 233 Infocenter 258 logon 263 shut down 152 database backup 47 cloned 131 configuration file 46, 56, 75, 169 corrupted 63 damaged 44, 63 frozen state 133 local 1819 non-recoverable 75 partition 69 prior state 82 remote 20 restart 45 restore 70 performance 76 to a new database 75 to an existing database 74 restore command 71 rollforward 131 rollforward recovery 49 shut down 48, 63 standby 93 database administrator 30, 359 database partition 105 DataJoiner 19 datamart 2, 4 designing 3 population 2 DataStage 7 data-striping 114 dates converting 291 DB2 Call Level Interface 272 DB2 Command Line Processor 65, 71, 79, 154 DB2 Control Center 13 DB2 Information Center 232, 234 DB2 Journal 158, 163, 168, 174, 178 db2 list history 200 DB2 OLAP Server 13, 15 DB2 private protocols 1920 DB2 Scheduler 174 DB2 Script Center 174 DB2 Suspended I/O feature 121, 126 DB2 UDB 6, 1314, 21, 24, 147 Control Center 8
DB2 UDB Enterprise - Extended Edition 105 DB2 UDB Enterprise Edition 8 DB2 UDB for OS/400 6 DB2 UDB for z/OS and OS/390 6 DB2 Warehouse Manager 5, 8, 23, 26 administration 147, 149 components 1, 4 configuration 148 DB2 XML Extender 29 DB2_PARALLEL_IO 119 db2adutl 140, 144, 171 db2ckbkp 66, 142, 170 db2diag.log 181 db2inidb 99, 126 db2level 268 db2look 90 db2move 44, 89 db2mscs 108 db2set 90, 119 development 147148, 151, 224 devices 112 diagnostic 251, 254 disaster recovery 39, 69, 92 standby via replication 94 transport of database backup 93 transport of logs 93 disk arrays 113 copy 120 corrupted 35 error 207 mirroring 96, 112, 116 parity 117 raid arrays 36 swapping 36 DRDA 1920 dropped table recovery 86, 197, 216 restrictions 219 duplexing 116 DWC 16
E
emergency 63 error application 53 code 233 communication 284 coverage 104
Index
369
disk 33, 81 GUI 244 log 234, 346 message 232233, 238 operating system 37 stored procedure 310 TCP/IP connectivity 284 token 234, 258 Error Correction Code 117 Erwin 15 ESS Advanced Copy Services 119, 121, 123 ETI ETI*Extract 5, 7, 15 ETML 288 architecture 1 logical layers 3 physical components 3 export 13, 44, 90, 92, 151 eXtended Markup Language 15 Extract Transform Move Load 1 extraction 15, 19, 289 automation 289 export 15 frequency 289 import 15 load 15 scheduling 326 SQL SELECT 15 status 320 user-written 15
scenarios 37 software 34 source 32 step 294 storage 147 warehouse server 33, 270 warehouse target 33 failure scenarios 40 failures consequences 31 feedback log 254 File Transfer Program 22 fixpacks 231, 268, 270 check 268 FlashCopy 120, 122123, 133 flat files 6, 11, 1314, 21 LAN 22 z-OS and OS/390 22 FormatDate 291 framework 3 FTP 22 message log 250 program 250 trace file 250 servers 24
G
granularity 82 graphical process modeler 13 graphical user interface 5, 8 GUI 5, 231
F
failover 97, 100, 103104 hot standby 106, 110 mutual takeover 107, 110 failure 3031, 234 agent 36 application 147 communication 32, 36 DB2 Warehouse Manager components 229 disk 111 hardware 31, 229, 271 media 72, 112, 147 network 33 ping 286 point of failure 113 power 40, 45, 53, 147 process 294
H
HACMP 99, 103104 configuration 105 HAGEO 95 hardware error 207 heartbeat 104 heartbeat monitoring 100 high availability 36 techniques 97, 229 High Availability Cluster Multi-Processing 99 high service level 36 Highly Available Geographic Cluster 95 history file 170, 221 HTTP servers 24 Hyperion Essbase Server 13
370
I
IBM Enterprise Storage Server 119 IBM ESS 130, 133 IBM Support 239, 248, 259, 268, 270, 286 idle 102 import 44, 90, 92, 151, 361 ordering 226 IMS 6, 22 Data Propagator 22 inconsistencies 197 incremental commit 32, 302 InfoCenter 24 Information Catalog 5, 20 Information Catalog Manager 5, 9, 15, 20, 63, 231 backup 343 configuration 344 Informix 5, 14, 21 integrity 39, 66, 142, 290 Intel 39 interval parameter 294 iSeries 5, 9, 32, 39 load programs 250251 iwh2exp2 227 iwh2imp2 228
J
Java 28 jobs 178, 271 Journaled File System 98, 134
mirrored 135 logretain 48, 50, 5253, 58, 149 logs 49, 75, 112, 131132, 281 active 53, 113 deletion 112 archive 53, 63, 149 buffer 57 capture 52, 55 circular 36, 52, 149 integrity 111 list history 143 mirrored 136 mirroring 113 offline archived 54 online archived 54 primary 269 primary logs 57 prune 58 secondary 53, 269 secondary log 57 shipping 93, 99 size 57, 269 space 150 logsecond 57 lost data 41 Lotus 1.2.3 6 Lotus Notes 6
M
malfunctions 35 manager 5 mapping 293, 338 message iSeries 278 type 280 message queues 22 messages queue 29 metadata 5, 9, 13, 1516, 2627, 29, 39, 151, 219, 224, 269, 337 business 20 errors 246 export 225 performance 227 import 226 performance 227 interchange 15 migration 226 publishing 20
L
LAN 66 LIST HISTORY 60 listening 241 load 44, 90, 292293 program trace file 251 LOB 219 Local Area Network 66 locking 83 log forwarding 93 primary 53, 57 sequence number 60 logbufsz 57 logfilsiz 57 logging 52 dual 112 logical volume
Index
371
Microsoft Cluster Server 99 Microsoft SQL Server 5, 14, 21 migration 151, 224, 357 mirroring 111112 asynchronous 122 dynamic synchronous 122 logical volumes 134 Mission Critical Linux 99 monitoring 16, 104, 297 monitoring facility 13 MQ-Assist wizard 28 MQSeries 6, 22, 28 MQSeries messages 28 MSCS 99, 103, 108 configuration 109 Multi-Computer/ServiceGuard 99 mutual takeover 101, 103, 105
outage 98
P
parallelism 68, 76, 87 partition 88 partitioned database 103, 289 patches 231, 268 Peer-to-Peer Remote Copy 122 pending state 49 performance 16, 19, 113114, 117118, 231 advice 306 checklist 311 issues 287 problem 281 SAP R/3 connector 342 stored procedure 308309 system 307 VARCHAR 308 physical volume split off 135 planning 92 point of control 5 point-in-time 122 polling step 25 PPRC 95, 122 prerequisites 23, 26 problem analysis procedure 281 cause 230 CLI 273 communication 281 database access 272 DB2 CLI 272 determination 230, 232, 241 DRDA 276 fixing 230 network 284 process 281 source identification 230, 241 warehouse control database 282 process 11, 287 definition 295 in flight 35 monitoring 13 operability 302 SAP R/3 339 SQL 18 transformation 16
N
named pipes 67 naming convention 226, 293 near real time 28 network 231 bandwidth 15 Network File System 22 NFS 22 nodes 100, 104 logical 105 notification 319
O
objects models 26 ODBC 5, 21, 272 generic ODBC driver 6 sources 13 OLAP Server program 250, 254 program trace 254 OLE-DB 6, 1314, 28 operating system error 267, 271 Operational Data Stores 4 operations 231 Oracle 5, 14, 21, 24 orthogonal RAID-5 118 OS/2 5, 8 OS/390 5, 39 OS/400 5
372
production 147148, 151, 224, 357 program malfunction 308 project 25 pSeries 5
Q
QMF for Windows 6, 9, 29 queries 30 govern 30 long running 6 workload 30 query and reporting tool 5 Query Patroller 9, 30 queues 6, 1314 quiesce 51, 83 exclusive mode 83 in share 83 intent 83 reset 83
R
RAID 111, 113 DB2 UDB 119 RAID-0 114115, 117 RAID-1 114, 116, 118 RAID-5 114, 117118 real time 24 recommendations 31, 37, 147, 229, 292, 331 recovery 35 AIX splitcopy 137 crash 53, 129 critical tables 85 disaster 62 dropped table 85 end of the logs 50 history file 59 prune 62 retention period 61 minimum time 197 partitioned database 69 point-in-time 42, 4950, 69, 82 procedures 39, 41 remote site 95 restore database command 47 rollforward 48 speed 41 strategy 40
time 60 unit of recovery 42, 51 using Data Warehouse Center 224 version recovery 47 WITHOUT ROLLING FORWARD option 47 Redbooks Web site 365 Contact us xxiv redirect 73 redundancy 111, 114 Redundant Arrays of Inexpensive Disks 113 referential integrity 83 regression 11 relational database 13, 21 reliability 3 remote mirroring 95 site 122 replication 13, 15, 52, 55, 292, 306 reports 24 requirements build-time 3 run-time 3 resources utilization 30, 307 response time 306 restart 34, 41 restartability 32 restore 99 buffers 76 size 76 method 48 partitioned database 88 restart 222 restrictions 77 without rolling forward 223 wizard 71 retry parameter 294 return code 233, 250251, 254 RC1 239 reusability 3 RFC 26, 259 rollback 4445 rollforward 77 examples 80 invoking 79 partitioned database 88 pending state 49, 81 restart 222 restrictions 86
Index
373
tablespace 81 rolling periods 15 rotating 105
S
SAN 123 SAP Connector 259 SAP R/3 1314, 26 SAPGUI 26 scalability 4 scheduling 13 security 3 service level 2, 41 SET files 248 sockets 239 software corrupted 35 Solaris 8 source 17, 21 error 36 local 20 recovery 32 remote 19 split mirror 97, 99, 119, 129130 backup image 132 to clone a database 129 splitcopy AIX 134 SQL error 32 SQL step 25 SQLOGDIR 58 sqlrestore 72 standby 99, 102 hot 104 idle 104 standby machine 99 star schema 24 statistics 16, 291, 301 Steeleye 99 step 11, 287 checking 345 definition 294 design 302 execution 302 failure 296, 301 in flight 35 number of rows 301 number of steps 293
on demand 12 processing option 296 program 296 SQL 11, 296, 303 success 301 transformer 11, 296, 304 UDP 303 user-defined program 296 Web Traffic Polling 321 steps 35 storage 66, 98 devices 67 management 138 off-site 63 problems 40 space 40 stored procedures 11, 2829, 40, 290 subtotals 11 summary records 306 summary tables 11 Sun Cluster 99 SUN Solaris 5, 23 Sybase 5, 14, 21 synchronization 359 system catalog 49, 81, 197 system catalogs 309
T
table frozen 51 function 28 histories 15 tablespace critical 69 drop 49 list 82 quiesce 83 redirected restore 72 rollforward recovery 49 status 62 tag file 151, 225226, 282 move 226 target 17 error 36 table 1820 TCP/IP 22, 284 test 293, 311, 357 testability 3
374
threads 239 throughput 306 time-out 3233, 37 Tivoli Storage Manager 64, 138, 144, 171 API 140 Backup-Archive client 139 tools data administrator tools 6 trace 242 CLI 244, 246, 274 Connector 244 Control Center 244 level 1 247 level 2 247 level 3 247 level 4 247 level 5 248 message queue 244, 249 ODBC connectivity 244 transformer 244, 254 UDP 244, 249 warehouse server 244, 247 traffic 314 transactions 44, 77, 83, 129 transformations 11, 147, 239, 290 built-in 15 cube load and precalculation 15 integrating products 5 SQL 15 statistical 5 user-defined 5 transformer User-Defined Function 290 transformers 11, 15 clean 11, 290 generate period tables 11 generate primary keys 11 pivot 11 scheduled 12 statistical 11, 290 to invert 11 warehouse 290 triggers 197, 290 Trillium Batch System 7 Trillium Software System 29 troubleshooting 26, 234 TSM 64, 76, 171
U
UDF 29 UDFs 40 UDP 12, 29, 248, 301 program example 346 scripts 293 using 352 UNICODE 315 unit of work 44, 53, 302 incomplete 45 UNIX 8, 32, 39, 64, 67 UOW 44 user checklist 265, 271, 292, 302, 304 corruption 35 error 231, 262, 271 User Defined Function 11, 40 User Defined Program 17, 240, 248 user-defined program 12 userexit 48, 50, 58, 149 users 5, 20
V
Vality INTEGRITY 15 Integrity 5, 7 VERITAS Cluster Server 99 view 28 VSAM 6, 22
W
warehouse development 11 log 32 management 13 infrastructure 4 population 15 process 16 warehouse agent 11, 16, 23, 26, 231, 240, 272 distributed 4 local 5 monitoring 149 remote 5 warehouse control database 16, 20, 39, 63, 147, 231, 264, 268 backup 147 backup verification 169171 corrupted 219, 246
Index
375
delta backup 168 incremental backup 168 migration 246 offline backup 154155 online backup 159 point-in-time recovery 197198, 204 recovery 151 reinstate 35 restore recovery 181, 183 rollforward recovery 36, 149, 185 standby 219 tablespace backup 164 tablespace redirected recovery 207208, 210 tablespace rollforward 187, 191 warehouse server 5, 8, 16, 18, 32, 148, 230231, 239, 286, 346 crash 34 warehouse source 11, 22, 25, 2729, 231, 240, 321, 331 corruption 32 warehouse target 11, 29, 39, 231, 240, 295, 321 Web 1314 site health 23 traffic measures 23 visitor behavior 23 Web channel 23 Web server log 318 Web Tracker 24, 317 Webmart 2425, 314, 321 WebSphere Application Server 2324, 316 WebSphere Commerce Server 24 WebSphere Commerce Suite 316 WebSphere Edge Server 24 WebSphere MQ 6, 1314, 22, 28 WebSphere Personalization Server 24 WebSphere Portal Server 24 WebSphere Site Analyzer 24, 258, 313 Windows 2000 5, 1617, 23, 26 NT 5, 1617, 23, 26 Windows 2000 8, 63, 239 Windows NT 8, 63, 239 Wolfpack 103 workflow 16, 297 workload 30 write suspend 130, 132 WSA 24, 258 Infocenter 315
project 315
X
XML 15, 29 XRC 122
Z
z/OS 5, 39 zSeries 5, 9, 32
376
(0.5 spine) 0.475<->0.875 250 <-> 459 pages
Back cover
DB2 Warehouse Management:

High Availability and Problem Determination Guide
DB2 Warehouse Manager components and failure consequences High Availability techniques for data, servers, and media Problem determination methodology
Using IBM DB2 Universal Database and DB2 Warehouse Manager provides a central point of control for managing data sources, including processes for building and populating data warehouses (or datamarts). The objective of this IBM Redbook is to deliver guidelines for building a High Availability DB2 warehouse environment and to assist with problem determination when using DB2 Warehouse Manager. This book will help you to understand the DB2 Warehouse Manager architecture and components, and to cope with system failures and their consequences. It also provides a set of documentation on the different High Availability techniques you can use when facing database, server, or disk failures. The book discusses a problem determination methodology that takes a customer through various unplanned outages. We explain how to proceed when a problem occurs, how to do problem determination and problem source identification, and how to fix the problem. Included are many helpful hints and tips on warehouse application and performance issues.
INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION
BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment.
For more information: ibm.com/redbooks

SG24-6544-00 ISBN 0738424625

Warehouse Management System

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Warehouse Management System

Transféré par

Droits d'auteur :

Formats disponibles

Front cover

DB2 Warehouse Management:

Copyright IBM Corp. 2002

DB2 Warehouse Management: High Availability and Problem Determination Guide

DB2 Warehouse Management: High Availability and Problem Determination Guide

DB2 Warehouse Management: High Availability and Problem Determination Guide

Copyright IBM Corp. 2002

DB2 Warehouse Management: High Availability and Problem Determination Guide

DB2 Warehouse Management: High Availability and Problem Determination Guide

DB2 Warehouse Management: High Availability and Problem Determination Guide

Copyright IBM Corp. 2002

DB2 Warehouse Management: High Availability and Problem Determination Guide

Copyright IBM Corp. 2002

DB2 Warehouse Management: High Availability and Problem Determination Guide

Copyright IBM Corp. 2002

Other company trademarks

DB2 Warehouse Management: High Availability and Problem Determination Guide

The team that wrote this redbook

Copyright IBM Corp. 2002

DB2 Warehouse Management: High Availability and Problem Determination Guide

Send your comments in an Internet note to:

Mail your comments to the address on page ii.

DB2 Warehouse Management: High Availability and Problem Determination Guide

DB2 warehouse management architecture

Copyright IBM Corp. 2002

1.1 Generic ETML subsystem architecture

DB2 Warehouse Management: High Availability and Problem Determination Guide

Chapter 1. DB2 warehouse management architecture

Transform and Build Component

1.2 DB2 Warehouse Manager components

DB2 Warehouse Management: High Availability and Problem Determination Guide

Chapter 1. DB2 warehouse management architecture

DB2 Warehouse Management: High Availability and Problem Determination Guide

DB2 Warehouse Manager Architecture: Agent Based

Data Warehouse Center

Message xx Windows 95/98, NT, 2000, AIX, Solaris

Data Metadata Message Data

Metadata yy Control Database Log Editions Configuration Data

NT, 2000, OS/2, AIX, iSeries Sun, zSeries

Included with DB2 UDB

Flat Files, Web or SAP R/3

Figure 1-2 DB2 Warehouse Manager architectural view

Chapter 1. DB2 warehouse management architecture

DB2 Warehouse Management: High Availability and Problem Determination Guide

Chapter 1. DB2 warehouse management architecture

Table 1-1 Platforms and components

DB2 Warehouse Management: High Availability and Problem Determination Guide

1.2.1 Warehousing tasks and objects

Chapter 1. DB2 warehouse management architecture

Fast Deployment with Pre-built Transformations

Figure 1-3 Pre-built transformers

DB2 Warehouse Management: High Availability and Problem Determination Guide

seit reporp ro f kci l c t h g i r

1.2.2 The Data Warehouse Center

The Data Warehouse Center graphical interface

Chapter 1. DB2 warehouse management architecture

Figure 1-4 Data Warehouse Center GUI Interface

DB2 Warehouse Management: High Availability and Problem Determination Guide

Chapter 1. DB2 warehouse management architecture

1.2.3 The warehouse server

1.2.4 The warehouse control database

1.2.5 The warehouse agent

DB2 Warehouse Management: High Availability and Problem Determination Guide

Figure 1-5 Agent-based architecture