English

Sign in (or register)

Technical topics

Evaluation software

Community

Events

Search developerWorks

developerWorks

Information Management

Technical library

IBM Data Movement Tool

Move data from source databases to DB2 in an easy way
Vikram S. Khatri, Certified Consulting I/T Specialist, IBM Summary: This article presents a very simple and powerful tool to move data from various sources easily. The tool allows enabling applications from Oracle and Sybase to run on IBM DB2, Version 9.7 for Linux, UNIX, and Windows as is, with very little or no changes. The tool can also be used to move data from various other database management systems to DB2 for Linux, UNIX, and Windows and DB2 for z/OS. The tool also supports moving data from a source database to the DB2 in a pureScale environment. Tags for this article: application_and_information_integration, application_development, data, databases, databases_and_data_management, db2, db2_data_movement_tool, db2_for_linux_and_unix_and_windows, from, ibm... more tags Tag this! Update My dW interests (Log in | What's this?) Date: 22 Nov 2010 (Published 19 Jun 2009) Level: Intermediate PDF: A4 and Letter (508KB | 28 pages) Get Adobe Reader Also available in: Russian Portuguese Spanish Activity: 34801 views Comments: 14 ( View | Add comment - Sign in) Average rating (37 votes) Rate this article

Introduction
This tool can be used to move data from various data sources to the DB2 in a pureScale environment. Beginning with DB2 V9.7 for Linux, UNIX, and Windows, the Migration Toolkit (MTK) is not required in order to use applications from Oracle and Sybase (After Fixpack 3) on DB2 products. This tool replaces the MTK functionality with a greatly simplified workflow. For all other scenarios, for example, moving data from a database to DB2 for z/OS, this tool supports the MTK particularly in the area of the high speed data movement. Using this tool, as much as 4TB of data have been moved in just three days. AGUI provides an easy to use interface for the novice while the command line API is often preferred by the advanced user.

Table of contents Introduction Preparation Extracting objects and data Deploying objects and loading data Plan for very large data movement Support for the tool Frequently asked questions Acknowledgements Disclaimer Download Resources About the author Comments

Preparation
Download
First, download the tool from the Download section to your target DB2 server. Additional steps are required to move data to DB2 for z/OS. (Check for the latest available version of the tool.)

Installation
Once you have downloaded the IBMDataMovementTool.zip file, extract the files into a directory called IBMDataMovementTool on your target DB2 server. Aserver side install (on DB2) is strongly recommended to achieve the best data movement performance.

Next steps from IBM

Prerequisites
DB2 V9.7 should be installed on your target server if you are enabling an Oracle application to be run on DB2 for Linux, UNIX, and Windows. Java version 1.5 or higher must be installed on your target server. To verify your current Java version, run java version command. By default, Java is installed as part of DB2 for Linux, UNIX, and Windows in <install_dir>\SQLLIB\java\jdk (Windows) or /opt/ibm/db2/V9.7/java/jdk (Linux). Table 1. Location of JDBC drivers for your source database and DB2
Database Oracle SQL Server Sybase MySQL PostgreSQL DB2 for Linux, UNIX, and Windows DB2 for z DB2 for i Teradatra MS Access JDBC drivers ojdbc5.jar or ojdbc6.jar or ojdbc14.jar, xdb.jar, xmlparserv2.jar or classes12.jar or classes111.jar for Oracle 7 or 8i sqljdbc5.jar or sqljdbc.jar jconn3.jar and antsjconn2.jar for DB2 SKIN feature mysql-connector-java-5.0.8-bin.jar or latest driver postgresql-8.1-405.jdbc3.jar or latest driver db2jcc.jar, db2jcc_license_cu.jar or db2jcc4.jar, db2jcc4_license_cu.jar db2jcc.jar, db2jcc_license_cisuz.jar or db2jcc4.jar, db2jcc4_license_cisuz.jar jt400.jar terajdbc4.jar and tdgssconfig.jar Optional Access_JDBC30.jar

Try: DB2 Express-C Demo: Use Ajax with DB2 Community: DB2

Tags Search all tags

Popular article tags | My article tags

More Less

about_the_pr... access administration apis application_... application_... architecture

architecture_... architecture_-... availability business_inte... business_perf... certifications cognos cognos_8_busin... cognos_busines... componentbas... configuratio... content_manage... crossproduct... customization data data_access data_integrity data_studio data_warehousi... database databases_an... db2 db2_for_i5/os

Environment setup

db2_for_linu...
converted by Web2PDFConvert.com

UNIX: Login to your server as DB2 instance owner. Windows: Launch a DB2 Command Window. Change to the IBMDataMovementTool directory. The tool is a JAR file with two driver scripts to run the tool.
IBMDataMovementTool.cmd - Command script to run the tool on Windows. IBMDataMovementTool.sh - Command script to run the tool on UNIX. IBMDataMovementTool.jar - JAR file of the tool. Pipe.dll - A DLL required on Windows if pipe option is used.

db2_for_linu...
db2_for_linux_... db2_for_z/os db2_luw db2luw debugging/test... deploying_an_... eclipse extensibility filenet ibm ids informix informix_tools informix_ult... infosphere infosphere_dat... infosphere_dat... infosphere_wa... infrastructur...

Create the DB2 target database

Since a database connection to the target is required to run the tool, the DB2 database must be created first. On DB2 V9.7, we recommended that you use the default automatic storage and choose a 32KB page size. When enabling applications to be run on DB2 V9.7, the instance and the database must be operating in compatibility mode. It is also recommended to adjust the rounding behavior to match that of Oracle. You can deploy objects out of dependency order by setting the revalidation semantics to deferred_force. On UNIX systems
$ db2set DB2_COMPATIBILITY_VECTOR=ORA $ db2set DB2_DEFERRED_PREPARE_SEMANTICS=YES $ db2stop force $ db2start $ db2 "create db testdb automatic storage yes on /db2data1, /db2data2,/db2data3 DBPATH ON /db2system PAGESIZE 32 K" $ db2 update db cfg for testdb using auto_reval deferred_force $ db2 update db cfg for testdb using decflt_rounding round_half_up

installation...
integrating_p... java java_technolo... legacy/backend... linux mdm migration open_source optim optim_develop... optimization performance scalability security soa_(service-o... sql sql_-_dynamic sql_-_static sql_(structur... stored_procedu... tools_and_util... tuning tutorial unix web_applicatio... web_services websphere windows xml

On Windows systems
C:\> db2set DB2_COMPATIBILITY_VECTOR=ORA C:\> db2set DB2_DEFERRED_PREPARE_SEMANTICS=YES C:\> db2stop force C:\> db2start C:\> db2 "create db testdb automatic storage yes on C:,D: DBPATH ON E: PAGESIZE 32 K" C:\> db2 update db cfg for testdb using auto_reval deferred_force C:\> db2 update db cfg for testdb using decflt_rounding round_half_up

View as cloud | list Dig deeper into Information Management on developerWorks Overview

Extracting objects and data

Before you run the tool, have the following information for your source and DB2 server ready: IP Address or Host Name of the source and DB2 servers Port numbers to connect Name of the databases, SID, sub-system name etc. as required AUser ID with DBAprivileges on the source database Password for that user Location of your source database and DB2 JDBC drivers Enough space or volume/mount point information where data will be stored Run IBMDataMovementTool.cmd on Windows or ./IBMDataMovementTool.sh on UNIX. The tool will start a GUI if the server is capable of displaying graphics. Otherwise it will switch to the interactive command line mode to gather input.
On Windows: IBMDataMovementTool.cmd On UNIX: chmod +x IBMDataMovementTool.sh ./IBMDataMovementTool.sh

New to Information Management Products Downloads Technical library (articles, tutorials, and more) Training Support Forums and community Events Newsletter

Seeking nominations

You will now see a GUI window. Some messages should also appear in the shell window. Please look through these messages to ensure no errors were logged before you start using the GUI. If you have not set DB2_COMPATIBILITY_VECTOR, the tool will report a warning. Please follow the steps to set the compatibility vector if you have not done so.

What is the DB2_COMPATIBILITY_VECTOR? The DB2_COMPATIBILITY_VECTOR is used to place both the DB2 V9.7 instance and database into an Oracle compatible mode. for details see the DB2 V9.7 Information Center.
Nominate IBM Champions: experts who help you build a smarter planet with IBM Special offers

[2010-01-10 17.08.58.578] INPUT Directory = . [2010-01-10 17.08.58.578] Configuration file loaded: 'jdbcdriver.properties' [2010-01-10 17.08.58.593] Configuration file loaded: 'IBMExtract.properties' [2010-01-10 17.08.58.593] appJar : 'C:\IBMDataMovementTool\IBMDataMovementTool.jar' [2010-01-10 17.08.59.531] DB2 PATH is C:\Program Files\IBM\SQLLIB [2010-01-10 17.35.30.015] *** WARNING ***. The DB2_COMPATIBILITY_VECTOR is not set. [2010-01-10 17.35.30.015] To set compatibility mode, discontinue this program and run the following commands [2010-01-10 17.35.30.015] db2set DB2_COMPATIBILITY_VECTOR=FFF [2010-01-10 17.35.30.015] db2stop force [2010-01-10 17.35.30.015] db2start

Using the Graphical User Interface

The GUI screen as shown in Figure 1 has fields for specifying the source and DB2 database connection information. The sequence of events in this screen are: 1. Specify source and DB2 connection information. 2. Click on Connect to Oracle to test the connection. 3. Click on Connect to DB2 to test the connection.

Trial software offers

converted by Web2PDFConvert.com

4. Specify the working directory where DDL and data are to be extracted to. 5. Choose if you want DDL and/or DATA. If you only select DDL, an additional genddl script will be generated. 6. Click on the Extract DDL/Data button. You can monitor progress in the console window. 7. After the data extraction is completed successfully, go through the result output files for the status of the data movement, warnings, errors and other potential issues. 8. Optionally, you can click on the View Script/Output button to check the generated scripts, DDL, data or the output log file. 9. Click on the Deploy DDL/Data button to create tables, indexes in DB2 and load data that was extracted from the source database. 10. You can use Execute DB2 Script to run the generated DB2 scripts instead of running it from the command line. The data movement is an interative exercise. If you need to drop all tables before you start fresh, you can select the drop table script and execute it. You can also use this button to execute the scripts in the order you want them to be executed. Figure 1. Input parameters for source and DB2 database

After clicking on the Extract DDL/Data button, you will notice tool's messages in the View File tab, as shown in Figure 2: Figure 2. Extract DDL and Data

After completing the extraction of DDL and DATA, you will notice several new files created in the working directory. These files can be used at the command line to run in DB2.

Configuration files
The following command scripts are regenerated each time you run the tool in GUI mode. However, you can use these scripts to perform all data movement steps without the GUI. This is helpful when you want to embed this tool as part of a batch processes to accomplish an automated data movement. Table 2. Command scripts
File name Description

IBMExtract.properties This file contains all input parameters that you specified through your GUI or command line input values. You can edit this file manually to modify or correct parameters. Note: This file is overwritten each time you run the GUI.

converted by Web2PDFConvert.com

unload

This script is created by the tool. It unloads data from the source database server to flat files, if you check DDL and Data options. The same script moves data from source database to DB2 using pipes, if you check the pipe option in the GUI to eliminate intermediate flat files. The pipe option is controlled through usePipe option in the IBMExtract.properties file. This script is created by the tool, and you can run it after deploying data to verify rowcounts in source and DB2 database.

rowcount

Figure 3. Files created after data extraction

Using command line mode

You can run the tool using command line mode particularly when the GUI capability is not available. The tool switches modes automatically if it is not able to start GUI. If you want to force to run the tool in command line interactive mode, you can specify the -console option to the IBMDataMovementTool command.
On Windows: IBMDataMovementTool -console On UNIX: ./IBMDataMovementTool.sh -console

You will be presented with interactive options to specify source and DB2 database connection parameters in step-by-step process. Asample output from the console window is shown as below:
[2010-01-10 20.08.05.390] INPUT Directory = . [2010-01-10 20.08.05.390] Configuration file loaded: 'jdbcdriver.properties' [2010-01-10 20.08.05.390] Configuration file loaded: 'IBMExtract.properties' [2010-01-10 20.08.05.390] appJar : 'C:\IBMDataMovementTool\IBMDataMovementTool.jar' Debug (Yes) :1 Debug (No) :2 Enter a number (Default=2) : IS TARGET DB2 LOCAL (YES) : 1 IS TARGET DB2 REMOTE (NO) : 2 Enter a number (Default=1) : Extract DDL (Yes) :1 Extract DDL (No) :2 Enter a number (Default=1) : Extract Data (Yes) :1 Extract Data (No) :2 Enter a number (Default=1) : Enter # of rows limit to extract. (Default=ALL) : Enter # of rows limit to load data in DB2. (Default=ALL) : Compress Table in DB2 (No) : 1 Compress Table in DB2 (YES) : 2

converted by Web2PDFConvert.com

Enter a number (Default=1) : Compress Index in DB2 (No) : 1 Compress Index in DB2 (YES) : 2 Enter a number (Default=1) : ******* Source database information: ***** Oracle :1 MS SQL Server :2 Sybase :3 MS Access Database : 4 MySQL :5 PostgreSQL :6 DB2 z/OS :7 DB2 LUW :8 Enter a number (Default 1) : DB2 Compatibility Feature (DB2 V9.7 or later) : 1 No Compatibility feature :2 Enter compatibility feature (Default=1) :

Deploying objects and loading data

Create database target objects
After extraction of the DDL and DATA, you have three different ways of deploying the extracted objects in DB2. Click the Deploy DDL/DATA button from the GUI screen Go to the Interactive Deploy tab and deploy objects in step-by-step process Deploy DDL/DATAusing command line script db2gen Which options to choose to deploy data are based upon the data and objects movement requirements. If you are migrating only non PL/SQL DDL objects and DATA, using the db2gen script or clicking the Deploy DDL/DATAbutton from the GUI will suffice. The interactive deploy option is likely your better choice when you are also deploying PL/SQL objects such as triggers, functions, procedures, and PL/SQL packages. The GUI screen, as shown in Figure 4, is used for interactive deployment of DDL and other database objects. The sequence of events in this screen is: 1. Ensure you are connected to DB2 using the Extract/Deploy tab. 2. Click on the Interactive Deploy tab. 3. Use the Open Directory button to select the working directory containing the previously extracted objects. The objects are read and listed in a tree view. 4. You can deploy all objects by pressing Deploy All Objects button on the toolbar. Most objects will deploy successfully while others may fail. 5. When you click on an object which failed to deploy in the tree view, you can see the source of the object in the editor window. The reason for the failure is listed in the deployment log below. 6. The Oracle compatibility mode generally allows deployment of objects as is. However, there may still be unsupported features that prevent successful deployment of some objects out of the box. Using the editor you can adjust the source code of these objects to work around any issues. When you deploy the changed object, the new source is saved with a backup of the old source. 7. You can select one or more objects using the CTRL key and click Deploy Selected Objects button on the toolbar to deploy objects after they have been edited. Often deployment failures occur in a cascade which means that once one object is successfully deployed others which depend on it will also deploy. 8. Repeat steps 5 through 7 until all objects have been successfully deployed. Figure 4. Interactive deploy of the objects

Compare row counts

Go to the root directory of the data movement and run the rowcount script. You should see a report generated in the "<source database name>.tables.rowcount" file. The report contains row counts from both source and target databases.

converted by Web2PDFConvert.com

oracle : db2 "TESTCASE"."CALL_STACKS" : 123 "TESTCASE"."CALL_STACKS" : 123 "TESTCASE"."CLASSES" : 401 "TESTCASE"."CLASSES" : 401 "TESTCASE"."DESTINATION" : 513 "TESTCASE"."DESTINATION" : 513

Use pipes to move data

When the source database size is too large and there is not enough space to hold intermediate data files, using pipe is the recommended way to move the data. On Windows systems The tool uses Pipe.dll to create Windows pipes and makes sure that this dll is placed in the same directory where IBMDataMovementTool.jar file is placed. On UNIX systems The tool creates UNIX pipes using the mkfifo command for use to move data from source to DB2. Before you can use pipe between source and DB2 database, it is necessary to have table definition created. Follow this procedure: 1. Specify # Extract Rows=1 in the GUI or set LimitExtractRows=1 in the IBMExtract.properties, if you're using the command line window. 2. Click on the Extract DDL/Data button to unload the data, or run the unload script from the command line window. 3. Click on the Deploy DDL/Data button, or run the db2gen script from the command line window. 4. Select Use Pipe, or set usepipe=true in the IBMExtract.properties, if you're using the command line window. 5. Click on the Extract / Deploy through Pipe Load button, or run the unload script from the command line window. Figure 5. Use of pipe to move data

Additional steps for DB2 on z/OS

You can use this tool from z/OS to do the data movement from a source database to DB2 for z/OS. However, the following additional steps are required. 1. Download and install JZOS. 2. This zip file contains a file named jzos.pax. FTP this file using Unix System Services in binary mode to the directory where you would like JZOS installed. 3. Change to the directory where you saved the .pax file. 4. Run the command: pax -rvf. This will create a subdirectory called jzos in your current working directory. This subdirectory will be referred to as <JZOS_HOME> 5. In the user's home directory, create a file named .profile based upon the template given below by making changes as per your z/OS DB2 installation. UNLOAD process on z/OS This tool requires USS to run but DB2 LOAD on z/OS can not use HFS files to load the data. That is why, you need to use JZOS toolkit to create PS datasets on z/OS from Unix System Services. However, DB2 LOAD can use USS (or HFS) files to LOAD CLOBS/BLOBS in DB2. That is why, we create PS datasets on z/OS to move the data from source database to z/OS and we use Unix System Services HFS files to keep all the BLOBS/CLOBS. The LOAD statement can not be run from USS. Use SYSPROC.DSNUTILS stored procedures to run the LOAD, CHECK DATAand RUN STATS. Creating PS datasets is a challenge since you need to allocate it for each table. Afixed size cannot be allocated ahead of time because the size of the table is unknown. It might waste lots of space on z/OS. To avoid a space problem, use an algorithm to allocate the size.

export JZOS_HOME=$HOME/jzos export JAVA_PATH=/usr/lpp/java/J1.5 export PATH=$JAVA_HOME/bin:$PATH export CLPHOME=/usr/lpp/db2/db2910/db2910_base/lib/IBM export CLASSPATH=$CLASSPATH:/usr/lpp/db2/db2910/db2910_base/lib/clp.jar export CLPPROPERTIESFILE=$HOME/clp.properties export LIBPATH=$LIBPATH:<JZOS_HOME<
converted by Web2PDFConvert.com

alias db2="java com.ibm.db2.clp.db2"

6. CLPHOME and CLASSPATH may have to be modified depending on your environment. Replace <JZOS_HOME> with the appropriate directory. 7. In the user's home directory, create a file name clp.properties based upon template given below:
#Specify the value as ON/OFF or leave them blank DisplaySQLCA=ON AutoCommit=ON InputFilename= OutputFilename= DisplayOutput= StopOnError= TerminationChar= Echo= StripHeaders= MaxLinesFromSelect= MaxColumnWidth=20 IsolationLevel= <SUBSYSTEM_NAME>=<IP address>:<port number>/<location name>,USER,PASSWD Replace items on the last line as appropriate.

8. Run the command chmod 777 <JZOS_HOME>/*.so 9. Run IBMDataMovementTool.sh -console command and specify values of the parameters through interactive user response. 10. IBMExtract.properties, geninput and unload scripts are created for you. 11. zdb2tableseries parameter in IBMExtract.properties is for specifying the name of the series for PS datasets. For example, if your TSO ID is DNET770, and this param is set to R, the name of the PS dataset created for first table will be DNET777.TBLDATA.R0000001 12. The parameter znocopypend is used to add NOCOPYPEND parameter in LOAD statement. With this parameter, the z/OS DB2 DBAcan perform the backup because the table will not be put in COPY pending mode. 13. The parameter zoveralloc specifies by how much you want to oversize your file allocation requests. Avalue of 1 means that you are nor oversizing at all. In an environment with sufficient free storage, this might work. In a realistic environment, 15/11 (1.3636) will be a good estimate. It is recommended that you start at 1.3636 (15/11) and lower the value gradually until you get file write errors, and then increase it a little. If you know the value of SMS parameter REDUCE SPACE UP TO, you should be able to calculate the perfect value of overAlloc by setting it to 1 / (1 - (X/100)), where X is the value of REDUCE SPACE UP TO given as an integer between 0 - 100. Note that REDUCE SPACE UP TO represents a percentage. 14. The parameter zsecondary is used to allocate fixed secondary extents. Start with a value of 0 and increase it slowly until file errors occur and then bring it back down 15. Run geninput script to create an input file for the unload process. 16. Run unload script to generate DDL and DATA. 17. Run generated script to create the DDL and load data on z/OS DB2. 18. The DSNUTILS will fail if you do not delete those datasets. The following java program can delete those intermediate datasets.
java -cp /u/dnet770/migr/IBMDataMovementTool.jar:$JZOS_HOME/ibmjzos.jar \ -Djava.ext.dirs=${JZOS_HOME}:${JAVA_HOME}/lib/ext ibm.Cleanup

19. After data loading is completed into DB2 tables on z/OS, you may find the datasets that you need to delete. Use the the following java program to delete those datasets as a part of cleanup. Create a script jd as shown below:
JZOS_HOME=$HOME/jzos JAVA_HOME=/usr/lpp/java/J1.5 CLASSPATH=$HOME/migr/IBMDataMovementTool.jar:$JZOS_HOME/ibmjzos.jar LIBPATH=$LIBPATH:$JZOS_HOME $JAVA_HOME/bin/java -cp $CLASSPATH \ -Djava.ext.dirs=${JZOS_HOME}:${JAVA_HOME}/lib/ext ibm.Jd $1

Change file permission to 755 and run it and then you will get an output shown below:
DNET770:/u/dnet770/migr: >./jd USAGE: ibm.Jd <filter_key> USAGE: ibm.Jd "DNET770.TBLDATA.**" USAGE: ibm.Jd "DNET770.TBLDATA.**.CERR" USAGE: ibm.Jd "DNET770.TBLDATA.**.LERR" USAGE: ibm.Jd "DNET770.TBLDATA.**.DISC"

So, if you want to delete all datasets under "DNET770.TBLDATA", use following command.
DNET770:/u/dnet770/migr: >./jd "DNET770.TBLDATA.**"

Plan for very large data movement

The strength of this tool is for large scale data movement. This tool has been used to move 4TB of Oracle data in just three days with good planning and procedures. Here are the tips and techniques that will help you to achieve large scale data movement in the time window constraint that you might have.

Hardware requirement and capacity planning

converted by Web2PDFConvert.com

It is out of the scope of this article to discuss hardware requirements and database capacity planning but it is important to keep in mind following considerations for estimating time to complete large scale data movement. You need a good network connection between source and DB2 server, preferably of 1GBPS or higher. You will be limited by the network bandwidth for the time frame to complete the data movement. The number of CPUs on the source server will allow you to unload multiple tables in parallel. For database size greater than 1TB, you should have minimum 4 CPU on source server. The number of CPUs on the DB2 server will determine the speed of the LOAD process. As a rule of thumb, you will require 1/4 to 1/3 of the time to load data in data and rests will be consumed by the unload process. Plan ahead the DB2 database layout. Please consult IBM's best practice paperss for DB2

Tips and techniques

Gain understanding of the tool in the command line mode. Use GUI to generate data movement scripts (geninput and unload) and practice data unload by running unload script from the command line. Extract only DDL from source by setting GENDDL=true and UNLOAD=false in the unload script. Use the generated DDL to plan for the table space and table mapping. Use a separate output directory to store generated DDL and data by specifying the target directory using -DOUTPUT_DIR parameter in the unload script. The generation of the DDL should be done ahead of the final data movement. Use geninput script to generate a list of tables to be moved from source to DB2. Use SRCSCHEMA=ALL and DSTSCHEMA=ALL parameter in the geninput script to generate a list of all tables. Edit the file to remove unwanted tables and split it into several input files to do a staggered movement approach where you perform unload from source and load to target in parallel. After breaking the table input file (generated from geninput script) into several files, copy the unload script into equivalent different files, change the name of the input file, and specify a different directory for each unload process. For example, you could create 10 unload scripts to unload 500 tables from each unload script, totalling 5000 tables. Make sure that you do DDL and DATAin separate steps. Do not mix these 2 into a single step for such large movement of data. The tool unloads data from the source tables in parallel controlled by NUM_THREADS parameter in the unload script. The default value is 5, and you can increase it to a level where CPU utilization on your source server is around 90%. Pay attention to the tables listed in the input tables file. The script geninput does not have intelligence to put the tables in a particular order, but you need to order the tables in such a way as to minimize unload time. The tables listed in the input files are fed to a pool of threads in a round robin fashion. It may so happen that all the threads have finished the unload process but one is still running. In order to keep all threads busy, organize the input file for the tables in the increasing numbers of rows. It may still so happen that all tables have unloaded and a few threads are still holding up unloading very large tables. You can unload the same table in multiple threads if you can specify the WHERE clause properly in the input file. For example:
"ACCOUNT"."T1":SELECT * FROM "ACCOUNT"."T1" WHERE id between 1 and 1000000 "ACCOUNT"."T1":SELECT * FROM "ACCOUNT"."T1" WHERE id between 1000001 and 2000000 "ACCOUNT"."T1":SELECT * FROM "ACCOUNT"."T1" WHERE id between 2000001 and 3000000 "ACCOUNT"."T1":SELECT * FROM "ACCOUNT"."T1" WHERE id between 3000001 and 4000000

Make sure that you use the right keys in the WHERE clause, which should preferrably be either the primary key or a unique index. The tool takes care of making proper DB2 LOAD scripts to load data from multiple files generated by the tool. There is no other setup required to unload the same table in multiple threads, except to add different WHERE clause as explained. After breaking your unload process in several steps, you can start putting data in DB2 simultaneously when a batch has finished unloading the data. The key here is the seperate output directory for each unload batch. All necessary files are generated to put data in DB2 in the output directory. For DDL, you will use generated db2ddl script to create table definitions. For data, you will use db2load script to load the data in DB2. If you combine DDL and data in a single step, the name of the script will be db2gen. Automate the whole process in your shell scripts so that the unload and load processes are synchronised. Each and every large data movement from Oracle or other databases to DB2 is unique. You will have your skills tested determining how to automate all of these jobs. Save the output of the jobs in a file by using the tee command, so that you can keep watching the progress, and the output is saved in a log file.

Run mock tests

It is a bad idea to fail to do the mock movement to test your automation and validate the way you planned staggered unload from source and load in DB2. The level of customization is only in creating the shell scripts to run these tasks in the right order. Follow these steps to run the mock tests: 1. Copy your data movement scripts and automation shell scripts to a mock directory. 2. Estimate your time by unloading a few large tables in a few threads, and accordingly stagger the movement of the data. 3. Add a WHERE clause to limit the number of rows to test the movement of data. For example, you can add a ROWNUM clause to limit the number of rows in Oracle or use the TOP clause for SQL Server.
"ACCOUNT"."T1":SELECT * FROM "ACCOUNT"."T1" WHERE rownum < 100 "ACCOUNT"."T2":SELECT * FROM "ACCOUNT"."T2" WHERE rownum < 100 "ACCOUNT"."T3":SELECT * FROM "ACCOUNT"."T3" WHERE rownum < 100 "ACCOUNT"."T4":SELECT * FROM "ACCOUNT"."T4" WHERE rownum < 100

4. Practice your scripts and make changes as necessary, and prepare for the final run.

Final run
1. You have already extracted DDL and made the required manual changes for the mapping between tables and tablespaces if required. 2. Take a downtime for the movement of the data. 3. Make sure your have around 10000 open cursors setting for the Oracle database if that is the source. 4. Watch the output from the log file. For large movement of data, it is much more about planning, discipline and the ability to automate jobs. The tool provides all the capability that you require for such movement. This little tool has moved very large databases from source to DB2.

converted by Web2PDFConvert.com

Support for the tool

This tool is not supported by the IBM support organization. However, you can report bugs, issues, suggestions, enhancement requests in the support forum.

Frequently asked questions

Table 3. Frequently asked questions
Question/Issue Do I need to install anything on my source database server in order for this tool to work? What are the supported platforms for this tool? I am running this tool from a secure shell window on my Linux/Unix platform and I see few messages in the command line shell but I do not see GUI and it seems that tool has hung. Answer/Solution You do not need to install anything on your source database for this tool.

Windows, z/OS, AIX, Linux, UNIX, HP-UX, Solaris, Mac and any other platform that has a JVM on it. Depending upon your DISPLAY settings, the GUI window has opened on your display capable server. You need to properly export your DISPLAY settings. Consult your Unix system adminstrator.

I am trying to move data from There is no JDBC drivers provided with the tool due to licensing considerations. You should get PostgreSQL and I do not see your database JDBC driver from your licensed software. PostgreSQL JDBC driver attached with the tool. It is not possible to grant DBA to the user extracting data from Oracle database. How can I use the tool? What are the databases to which this tool can connect? What version of Java do I need to run this tool? You will at least need SELECT_CATALOG_ROLE granted to the user and SELECT privileges on tables used for migration. Any database that has a type-IV JDBC driver. So, you can connect to MySQL, PostgreSQL, Ingres, SQL Server, Sybase, Oracle, DB2 and others. It can also connect to a database that has a ODBCJDBC connector so you can also move fromAccess database. You need minimum Java 1.5 to run the tool. The dependency for Java 1.5 is basically due to the GUI portion of the tool. If you really need support for Java 1.4.2, send me a note and I will compile the tool for Java 1.4.2 but the GUI will not run to create the data movement driver scripts. You can determine the version of Java by running this command. $ java -version C:\>java -version How do I check the version of the tool? Run IBMDataMovementTool -version on Windows or ./IBMDataMovementTool.sh -version on Linux/UNIX

I am get the error "Unsupported You are using a version of Java less than 1.5. Install Java higher than version 1.4.2 to overcome major.minor version 49.0" or "(.:15077): this problem. We prefer that you install IBM Java. Gtk-WARNING **: cannot open display: " when I run the tool. What does it mean? What information do I need for a source and DB2 database servers in order to run this tool? I am running this tool from my Windows workstation and it is running extremely slow. What can I do? You need to know IP address, port number, database name, user id and password for the source and DB2 database. The user id for the source database should have DBA priviliges and SYSADM privilege for the DB2 database. The default memory allocated to this tool from IBMDataMovementTool.cmd or IBMDataMovementTool.sh command script is 990MB by using -Xmx switch for the JVM. Try reducing this memory as you might be having less memory on your workstation.

I am doing a data movement from SQL Specify mssqltexttoclob=true in IBMExtract.properties file. Server to DB2. How do I get my TEXT field to go to VARCHAR in DB2. I am doing a data movement from Sybase to DB2 and it did not move my T-SQL procedures to DB2. I am doing a DDL movement from Sybase to DB2 and I have my Sybase objects in a file. I do not see a way to specify DDL file as a data source. I am doing a data movement from MS Access to DB2 and I do not see all indexes etc in the DDL generated. I am doing a data movement from Sybase to DB2 using this tool and I am getting tons of error. I am doing a data movement from MySQL to DB2 and I am running out of memory. I am doing a data movement from The purpose of this tool is only DDL and DATA movement. You will have to use MTK for the purpose of procedure / triggers movement. The purpose of this tool is the high speed data movement and that is why there is no capability to transform a DDL file from a database to DB2. You can however use IBM InfoSphere Data Architect to trasnform a DDL from a source database to a target. We use basic ODBC-JDBC connector to connect to MS Access database. You will need a different commercial JDBC driver to obtain complete set of DDLs. You can try HXTT JDBC driver for MS Access. If you use HXTT driver, you will have to specify DBVENDOR=hxtt in generated unload script instead of access. It is quite possible that your Sybase database is not enabled for required JDBC support. Please consult your Sybase DBA to ensure that correct JDBC stored procedures are installed in your Sybase database. Try different values with FETCHSIZE=nnn in the generated unload script and run the data movement from command line. If you use GUI tool, it will overwrite unload script. The additional JAR files are mainly required for Oracle XML data types. You should get those files
converted by Web2PDFConvert.com

I am doing a data movement from Oracle to DB2 and I notice that there are 3 jars files required for the data movement. My understanding is the we only need a JDBC driver for data movement. Why additional jar files? I want Oracle data type of CLOB to go as DBCLOB in DB2.

The additional JAR files are mainly required for Oracle XML data types. You should get those files from your Oracle installation directory.

Go to IBMExtract.properties file and set DBCLOB=true.

I am using this tool to move data from The user ID connecting to Oracle should have SELECT_CATALOG_ROLE granted to it and SELECT Oracle to DB2 and I am getting many privileges on the tables. Oracle SQL error that a table was not found. I do not want NCHAR and NVARCHAR2 to go as GRAPHIC or VARGRAPHIC in DB2. I want them to go as CHAR and VARCHAR2 since I created DB2 database as UTF-8. Can I do data movement from Oracle database to DB2 version less than V9.7/V9.5? Go to IBMExtract.properties file and set GRAPHIC=false.

Yes, go to IBMExtract.properties and set db2_compatibility=false

I noticed that your tool moved Oracle's Go to IBMExtract.properties and set roundDown_31=false. NUMBER(38) to NUMBER(31) and I understand that DB2 supports only up to 31. I do not want to round down and I want to convert this to DOUBLE. I am getting lots of data rejected. How do I get that rejected data in a file so that I can analyze the reason of rejection. I am trying to load data from a workstation to a DB2 server and I am getting erros. Do I have to run the tool from server only? I can only login to my DB2 server through a SSH shell and we do not allow X-Windows to run on DB2 server. How do I run this GUI tool to move DDL and DATA? Why did you not create DB2 database through your script since you ask the name of the database. Why do I need xdb.jar and xmlparserv2.jar in addition to Oracle JDBC driver? Go to IBMExtract.properties and set dumpfile=true.

It is preferable to run this tool from the DB2 server to extract data from the source database and avoid an intermediate server. However if you want to run this tool from an intermediate server, you can specify REMOTELOAD=TRUE in the generated script unload. Please remember that DB2 LOAD utility requires for BLOBS/CLOBS/XML data to be available on server. You will need to mount those directories with same naming convention on the target DB2 server. Run IBMDataMovementTool.sh from your SSH and if there is no graphics support, the tool will switch to command line input automatically. If it does not switch for some reason, specify console option to the IBMDataMovementTool.sh command and it will force to run the tool in the interactive command line mode. The command line mode is just a way to gather the input and to generate necessary scripts for data movement. The use of GUI is just a way to generate the scripts and the actual works is done through the scripts only. DBAs normally like to create their database as per their storage paths information. We do however create necessary table spaces so that tables are put automatically in right table space by DB2. You should consider reading IBM's best practice papers to carefully plan for your database. It is recommended that you create DB2 database with 32K page size as default. The xdb.jar and xmlparserv2.jar will be required if your Oracle data contains XML data. You can locate xdb.jar in folder server/RDBMS/jlib and xmlparserv2.jar in lib folder. If you are unable to locate these, you can download Oracle XDK for Java.

I am getting This error comes if you are running the tool on Windows 64-bit platform using Java 32-bit JVM. java.lang.UnsatisfiedLinkError: Pipe Install Java 64-bit JVM on your Windows platform, and rerun the tool. Pipe.dll is not a valid Win32 application. How do I fix this?

Acknowledgements
Many IBMers from around the world provided valuable feedback to the tool and without their feedback, the tool in this shape would not have been possible. I acknowledge significant help, feedback, suggestions and guidance from following people. Jason AArnold Serge Rielau Marina Greenstein Maria N Schwenger Patrick Dantressangle Sam Lightstome Barry Faust Vince Lee Connie Tsui Raanon Reutlinger Antonio Maranhao Max Petrenko Kenneth Chen Masafumi Otsuki Neal Finkelstein

Disclaimer
This article contains a tool. IBM grants you ("Licensee") a non-exclusive, royalty free, license to use this tool. However, the tool is provided as-is and without any warranties, whether EXPRESS OR IMPLIED, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY FITNESS FOR APARTICULAR PURPOSE OR NON-INFRINGEMENT. IBMAND ITS LICENSORS SHALL NOT , BE LIABLE FOR ANY DAMAGES SUFFERED BY LICENSEE THAT RESULT FROM YOUR USE OF THE SOFTWARE. IN NO EVENT WILL IBM OR ITS LICENSORS BE LIABLE FOR ANY LOST REVENUE, PROFIT OR DATA, OR FOR DIRECT, INDIRECT,
converted by Web2PDFConvert.com

SPECIAL, CONSEQUENTIAL, INCIDENTAL OR PUNITIVE DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY ARISING OUT OF THE USE OF OR INABILITY TO USE SOFTWARE, EVEN IF IBM HAS BEEN ADVISED OF , THE POSSIBILITY OF SUCH DAMAGES.

Download
IBM Data Movement Tool1 Note 1. Anew build of the tool is uploaded very frequently, after bug fixes and new enhancements. Click on Help > Check New Version from the GUI or enter the command ./IBMDataMovementTool.sh -check to check if a new build is available for download. You can find the Tool's build number from the Help > About menu option or by entering the ./IBMDataMovementTool.sh -version command. This tool uses JGoodies Forms 1.2.1, JGoodies Look 2.2.2, and JSyntaxPane 0.9.4 packages for the GUI interface.

Resources
Learn "Migrate from MySQL or PostgreSQL to DB2 Express-C" (developerWorks, June 2006) was the first article written for this tool. "DB2 Viper 2 compatibility features" (developerWorks, July 2007) is the article that explains compatibility features. You can also use Migration Toolkit, for the migration of data and procedures. Get products and technologies Download DB2 Express-C 9.7, a no-charge version of DB2 Express database server for the community. Download a free trial version of DB2 9.7 for Linux, UNIX, and Windows.. Download IBM product evaluation versions and get your hands on application development tools and middleware products from DB2, Lotus, Rational, Tivoli, and WebSphere. Discuss Participate in the discussion forum. Check out developerWorks blogs and get involved in the developerWorks community.

About the author

Vikram S Khatri works for IBM in the Sales and Distribution Division and is a member of the DB2 Migration team. Vikram has 24 years of IT experience and specializes in enabling non-DB2 applications to DB2. Vikram supports the DB2 technical sales organization by assisting with complex database migration projects as well as with database performance benchmark testing.

Rate this article

Average rating (37 votes) 1 star 2 stars 3 stars 4 stars 5 stars Submit

Comments
Add comment: Sign in or register to leave a comment. Note: HTML elements are not supported within comments.

Notify me when a comment is added

1000 characters left

converted by Web2PDFConvert.com

Post

Total comments (14)

Show: Most recent comments

The answers to many of these questions are available on the DB2 Application Enablement forum at http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1803, which the author, Vikram, monitors regularly. GPT-Tom, as you know, the answer to your question is at http://www.ibm.com/developerworks/forums/thread.jspa?threadID=368808&tstart=0.
Posted by kelleydg on 22 April 2011 Report abuse

I am trying to use tool to migrate (change) 9.1 DB s to UTF-8 dbs. I have three tables that will not load back up. I am getting this error msg: Any ideas from anyone? com.ibm.db2.jcc.c.SqlException: DB2 SQL error: SQLCODE: -104, SQLSTATE: 42601, SQLERRMC: 65;<identifier>;ABORT at com.ibm.db2.jcc.c.fg.d(fg.java:1340) at com.ibm.db2.jcc.b.gb.k(gb.java:351) at com.ibm.db2.jcc.b.gb.e(gb.java:96) at com.ibm.db2.jcc.b.w.e(w.java:83) at com.ibm.db2.jcc.b.vb.i(vb.java:164) at com.ibm.db2.jcc.c.fg.q(fg.java:1311) at com.ibm.db2.jcc.c.gg.d(gg.java:2386) at com.ibm.db2.jcc.c.hg.X(hg.java:115) at com.ibm.db2.jcc.c.hg.execute(hg.java:98) at ibm.DeployObjects.CallSP(DeployObjects.java:445) at ibm.DeployObjects.DeployAllObjectsInDB2(DeployObjects.java:605) at ibm.DeployObjects.main(DeployObjects.java:692)
Posted by GPT-Tom on 21 April 2011 Report abuse

You can use standard JVM also. IBM JVM is optimized and seem to run better but I guess that many people will raise an eyebrow on my assertion.
Posted by Anabas on 24 February 2011 Report abuse

In the article, you said that We prefer that you install IBM Java. Where can I download IBM Java? Can you please provide a link for the download? Sheila
Posted by mq@user on 23 February 2011 Report abuse

Please use forum of this tool to post your questions. I do not this very often. The varchar problem is fixed in the latest build of the tool. So, download it from the site at http://bit.ly/hHXnx4
Posted by Anabas on 26 January 2011 Report abuse

Back to top Print this page Share this page Follow developerWorks

Technical topics
AIX and UNIX IBM i Information Management Lotus Rational Tivoli WebSphere Cloud computing Industries Integrated Service Management Java technology Linux Open source SOAand web services Web development XML

Evaluation software
By IBM product By evaluation method By industry

Community
Forums Groups Blogs Wikis Terms of use Report abuse IBM Champion program

About developerWorks
Site help and feedback Contacts Article submissions

IBM
Solutions Software Software services Support Product information Redbooks Privacy Accessibility

Events
Briefings Webcasts Find events

Related resources
Students and faculty Business Partners

More...

More...

converted by Web2PDFConvert.com