Best Practices

Incorporating Target's Standards

DataStage is a trademark of International Business Machines Corporation

Main Source

"DataStage Technical Design and Construction Procedures" This is a "living document"

\\nicsrv10\TTS\E\ETL\Best Practices\DataStageTechDoc\DataStageTech.doc

that is, a work in progress changes will be notified

Job Naming

Each DW "project" has three-letter code

for example GLB

Within Jobs branch create category with that name

keep all objects together in order to support MetaStage functions

Job Naming

Job name begins with database identifier

for example GTL

GTLJB0001 GTLJB0002 GTLJB0002TEST GTLJB0003

Followed by job identifier and sequence

Stage Names

First 3-4 characters: stage type

SEQL (Sequential File stage) LKFS (Lookup File Set stage)

Remainder should be meaningful and descriptive

first character to be capitalized

Link Names

Links prior to final active stage

shortdesc_InTo_stagedesc shortdesc_OutTo_stagedesc

Links after final active stage

Links from passive stage

In_linkdesc
Out_linkdesc_action

Links to passive stage

Links from Lookup stage

Lkup_linkdesc

Example
Images copyright claimed by Ascential Software Corporation

Reusable Components

Images copyright claimed by Ascential Software Corporation

Create reusable components where possible

shared containers flexible routines

Annotations

Annotations are to be used to explain processing Description annotation shows purpose of job

Annotations
Description Annotation
Images copyright claimed by Ascential Software Corporation

Job Descriptions
Images copyright claimed by Ascential Software Corporation

Become text of description annotation Short description visible in Detail view (Manager)

Stage/Link Naming

Stages are named after

the data they access (passive stages) the function they perform (active) for the data they carry
such as Sequential_File_0

Links are named

Do not leave default names

Developing Jobs
1.

Keep it simple
jobs with many stages are hard to debug and maintain documentation

2.

Start small and Build to final Solution

plan use view data, copy, and peek start from source and work out develop with a 1 node configuration file, small set of data

Developing Jobs (continued)

3.

Solve the business problem before the performance problem

dont worry too much about partitioning until the sequential flow works as expected

4.

If you have to write to disk use a persistent Data Set

Developing Jobs (continued)

Images copyright claimed by Ascential Software Corporation

Iterative Design

Use Copy or Peek stage as stub Test job in phases small first, then increasing in complexity Use Peek stage to examine records

Example Phase 1
Images copyright claimed by Ascential Software Corporation

Example Phase 2
Images copyright claimed by Ascential Software Corporation

Example Phase 3
Images copyright claimed by Ascential Software Corporation

Transformer Stage

Transformer stage generates code Always include reject link Always test for null value before using a column in a function Be aware of column and stage variable data types

often developer does not pay attention to Stage Variable data type try to maintain the data type as imported

Avoid data type conversions

Job Parameters

Provide insurance against

things that change over time (for example passwords, filter conditions) things that different in different environments (for example DSNs, pathnames, passwords)

Job Parameters

Created in Job Properties Each parameter has

name prompt text (mandatory) type default value (design time) help text

Defining Job Parameters

Images copyright claimed by Ascential Software Corporation

Click to add environment variables

Using Job Parameters

In fields in passive stages delimit with "#" characters

for example #SourceDir#

Names are case-sensitive In expressions choose from expression editor

not delimited

Useful Environment Variables

APT_DUMP_SCORE

report osh to message log

establishes name of configuration file and therefore degree of parallelism

APT_CONFIG_FILE

DUMP SCORE Output

Images copyright claimed by Ascential Software Corporation

Setting APT_DUMP_SCORE yields:

Partitioner And Collector

Two DataSets

Mapping Node --> partition

Configuration Files

Make a set for 1X, 2X,. Use different ones for test versus production Include as a parameter in each job Automatic scaling