Data Extraction

Data Extraction, Transformation, and Loading
Techniques
-->
Data Extraction, Transformation, and Loading Techniques

INTRODUCTION
During the ETL process, data is extracted from an OLTP database, transformed to match the data
warehouse schema, and loaded into the data warehouse database. Many data warehouses also
incorporate data from non-OLTP systems, such as text files, legacy systems, and spreadsheets; such
data also requires extraction, transformation, and loading.
In its simplest form, ETL is the process of copying data from one database to another. This simplicity
is rarely, if ever, found in data warehouse implementations; in reality, ETL is often a complex
combination of process and technology that consumes a significant portion of the data warehouse
development efforts and requires the skills of business analysts, database designers, and application
developers.
When defining ETL for a data warehouse, it is important to think of ETL as a process, not a physical
implementation. ETL systems vary from data warehouse to data warehouse and even between
department data marts within a data warehouse. A monolithic application, regardless of whether it is
implemented in Transact-SQL or a traditional programming language, does not provide the flexibility
for change necessary in ETL systems. A mixture of tools and technologies should be used to develop
applications that each perform a specific ETL task.
The ETL process is not a one-time event; new data is added to a data warehouse periodically. Typical
periodicity may be monthly, weekly, daily, or even hourly, depending on the purpose of the data
warehouse and the type of business it serves. Because ETL is an integral, ongoing, and recurring
part of a data warehouse, ETL processes must be automated and operational procedures
documented. ETL also changes and evolves as the data warehouse evolves, so ETL processes must
be designed for ease of modification. A solid, well-designed, and documented ETL system is
necessary for the success of a data warehouse project.
Data warehouses evolve to improve their service to the business and to adapt to changes in business
processes and requirements. Business rules change as the business reacts to market influences
the data warehouse must respond in order to maintain its value as a tool for decision makers. The
ETL implementation must adapt as the data warehouse evolves.
Microsoft SQL Server 2000 provides significant enhancements to existing performance and
capabilities, and introduces new features that make the development, deployment, and maintenance
of ETL processes easier and simpler, and its performance faster.
ETL FUNCTIONAL ELEMENTS
Regardless of how they are implemented, all ETL systems have a common purpose: they move data
from one database to another. Generally, ETL systems move data from OLTP systems to a data
warehouse, but they can also be used to move data from one data warehouse to another. An ETL
system consists of four distinct functional elements:
Extraction
Transformation
Loading
Meta data
Extraction
The ETL extraction element is responsible for extracting data from the source system. During
extraction, data may be removed from the source system or a copy made and the original data
retained in the source system. It is common to move historical data that accumulates in an
operational OLTP system to a data
warehouse to maintain OLTP performance and efficiency. Legacy systems may require too much
effort to implement such offload processes, so legacy data is often copied into the data warehouse,
leaving the original data in place. Extracted data is loaded into the data warehouse staging area (a
relational database usually separate from the data warehouse database), for manipulation by the
remaining ETL processes.
Data extraction is generally performed within the source system itself, especially if it is a relational
database to which extraction procedures can easily be added. It is also possible for the extraction
logic to exist in the data warehouse staging area and query the source system for data using ODBC,
OLE DB, or other APIs. For legacy systems, the most common method of data extraction is for the
legacy system to produce text files, although many newer systems offer direct query APIs or
accommodate access through ODBC or OLE DB.
Data extraction processes can be implemented using Transact-SQL stored procedures, Data
Transformation Services (DTS) tasks, or custom applications developed in programming or scripting
languages.
Transformation
The ETL transformation element is responsible for data validation, data accuracy, data type
conversion, and business rule application. It is the most complicated of the ETL elements. It may
appear to be more efficient to perform some transformations as the data is being extracted (inline
transformation); however, an ETL system that uses inline transformations during extraction is less
robust and flexible than one that confines transformations to the transformation element.
Transformations performed in the OLTP system impose a performance burden on the OLTP
database. They also split the transformation logic between two ETL elements and add maintenance
complexity when the ETL logic changes.
Tools used in the transformation element vary. Some data validation and data accuracy checking can
be accomplished with straightforward Transact-SQL code. More complicated transformations can be
implemented using DTS packages. The application of complex business rules often requires the
development of sophisticated custom applications in various programming languages. You can use
DTS packages to encapsulate multi-step transformations into a single task.
Listed below are some basic examples that illustrate the types of transformations performed by this
element:
Data Validation
Check that all rows in the fact table match rows in dimension tables to enforce data integrity.
Data Accuracy
Ensure that fields contain appropriate values, such as only "off" or "on" in a status field.
Data Type Conversion
Ensure that all values for a specified field are stored the same way in the data warehouse regardless
of how they were stored in the source system. For example, if one source system stores "off" or "on"
in its status field and another source system stores "0" or "1" in its status field, then a data type
conversion transformation converts the content of one or both of the fields to a specified common
value such as "off" or "on".
Business Rule Application
Ensure that the rules of the business are enforced on the data stored in the warehouse. For example,
check that all customer records contain values for both FirstName and LastName fields.
Loading
The ETL loading element is responsible for loading transformed data into the data warehouse
database. Data warehouses are usually updated periodically rather than continuously, and large
numbers of records are often loaded to multiple tables in a single data load. The data warehouse is
often taken offline during update operations so that data can be loaded faster and SQL Server 2000
Analysis Services can update OLAP cubes to incorporate the new data. BULK INSERT, bcp, and the
Bulk Copy API are the best tools for data loading operations. The design of the loading element
should focus on efficiency and performance to minimize the data warehouse offline time. For more
information and details about performance tuning, see Chapter 20, "RDBMS Performance Tuning
Guide for Data Warehousing."
Meta Data
The ETL meta data functional element is responsible for maintaining information (meta data) about
the movement and transformation of data, and the operation of the data warehouse. It also
documents the data mappings used during the transformations. Meta data logging provides
possibilities for automated administration, trend prediction, and code reuse.
Examples of data warehouse meta data that can be recorded and used to analyze the activity and
performance of a data warehouse include:
Data Lineage, such as the time that a particular set of records was loaded into the data
warehouse.
Schema Changes, such as changes to table definitions.
Data Type Usage, such as identifying all tables that use the "Birthdate" user-defined data type.
Transformation Statistics, such as the execution time of each stage of a transformation, the
number of rows processed by the transformation, the last time the transformation was executed,
and so on.
DTS Package Versioning, which can be used to view, branch, or retrieve any historical version of a
particular DTS package.
Data Warehouse Usage Statistics, such as query times for reports.
ETL DESIGN CONSIDERATIONS
Regardless of their implementation, a number of design considerations are common to all ETL
systems:
Modularity
ETL systems should contain modular elements that perform discrete tasks. This encourages reuse
and makes them easy to modify when implementing changes in response to business and data
warehouse changes. Monolithic systems should be avoided.
Consistency
ETL systems should guarantee consistency of data when it is loaded into the data warehouse. An
entire data load should be treated as a single logical transactioneither the entire data load is
successful or the entire load is rolled back. In some systems, the load is a single physical transaction,
whereas in others it is a series of transactions. Regardless of the physical implementation, the data
load should be treated as a single logical transaction.
Flexibility
ETL systems should be developed to meet the needs of the data warehouse and to accommodate the
source data environments. It may be appropriate to accomplish some transformations in text files and
some on the source data system; others may require the development of custom applications. A
variety of technologies and techniques can be applied, using the tool most appropriate to the
individual task of each ETL functional element.
Speed
ETL systems should be as fast as possible. Ultimately, the time window available for ETL processing
is governed by data warehouse and source system schedules. Some data warehouse elements may
have a huge processing window (days), while others may have a very limited processing window
(hours). Regardless of the time available, it is important that the ETL system execute as rapidly as
possible.
Heterogeneity
ETL systems should be able to work with a wide variety of data in different formats. An ETL system
that only works with a single type of source data is useless.
Meta Data Management
ETL systems are arguably the single most important source of meta data about both the data in the
data warehouse and data in the source system. Finally, the ETL process itself generates useful meta
data that should be retained and analyzed regularly. Meta data is discussed in greater detail later in
this chapter.
ETL ARCHITECTURES
Before discussing the physical implementation of ETL systems, it is important to understand the
different ETL architectures and how they relate to each other. Essentially, ETL systems can be
classified in two architectures: the homogenous architecture and the heterogeneous architecture.
Homogenous Architecture
A homogenous architecture for an ETL system is one that involves only a single source system and a
single target system. Data flows from the single source of data through the ETL processes and is
loaded into the data warehouse, as shown in the following diagram.
Most homogenous ETL architectures have the following characteristics:

Single data source: Data is extracted from a single source system, such as an OLTP system.
Rapid development: The development effort required to extract the data is straightforward because
there is only one data format for each record type.
Light data transformation: No data transformations are required to achieve consistency among
disparate data formats, and the incoming data is often in a format usable in the data warehouse.
Transformations in this architecture typically involve replacing NULLs and other formatting
transformations.
Light structural transformation: Because the data comes from a single source, the amount of
structural changes such as table alteration is also very light. The structural changes typically involve
denormalization efforts to meet data warehouse schema requirements.
Simple research requirements: The research efforts to locate data are generally simple: if the data
is in the source system, it can be used. If it is not, it cannot.
The homogeneous ETL architecture is generally applicable to data marts, especially those focused on
a single subject matter.
Heterogeneous Architecture
A heterogeneous architecture for an ETL system is one that extracts data from multiple sources, as
shown in the following diagram. The complexity of this architecture arises from the fact that data from
more than one source must be merged, rather than from the fact that data may be formatted
differently in the different sources. However, significantly different storage formats and database
schemas do provide additional complications.
Most heterogeneous ETL architectures have the following characteristics:

Multiple data sources.
More complex development: The development effort required to extract the data is increased
because there are multiple source data formats for each record type.
Significant data transformation: Data transformations are required to achieve consistency among
disparate data formats, and the incoming data is often not in a format usable in the data warehouse.
Transformations in this architecture typically involve replacing NULLs, additional data formatting,
data conversions, lookups, computations, and referential integrity verification. Precomputed
calculations may require combining data from multiple sources, or data that has multiple degrees of
granularity, such as allocating shipping costs to individual line items.
Significant structural transformation: Because the data comes from multiple sources, the amount of
structural changes, such as table alteration, is significant.
Substantial research requirements to identify and match data elements.
Heterogeneous ETL architectures are found more often in data warehouses than in data marts.
ETL DEVELOPMENT
ETL development consists of two general phases: identifying and mapping data, and developing
functional element implementations. Both phases should be carefully documented and stored in a
central, easily accessible location, preferably in electronic form.
Identify and Map Data
This phase of the development process identifies sources of data elements, the targets for those data
elements in the data warehouse, and the transformations that must be applied to each data element
as it is migrated from its source to its destination. High level data maps should be developed during
the requirements gathering and data modeling phases of the data warehouse project. During the ETL
system design and development process, these high level data maps are extended to thoroughly
specify system details.
Identify Source Data
For some systems, identifying the source data may be as simple as identifying the server where the
data is stored in an OLTP database and the storage type (SQL Server database, Microsoft Excel
spreadsheet, or text file, among others). In other systems, identifying the source may mean preparing
a detailed definition of the meaning of the data, such as a business rule, a definition of the data itself,
such as decoding rules (O = On, for example), or even detailed documentation of a source system for
which the system documentation has been lost or is not current.
Identify Target Data
Each data element is destined for a target in the data warehouse. A target for a data element may be
an attribute in a dimension table, a numeric measure in a fact table, or a summarized total in an
aggregation table. There may not be a one-to-one correspondence between a source data element
and a data element in the data warehouse because the destination system may not contain the data
at the same granularity as the source system. For example, a retail client may decide to roll data up
to the SKU level by day rather than track individual line item data. The level of item detail that is
stored in the fact table of the data warehouse is called the grain of the data. If the grain of the target
does not match the grain of the source, the data must be summarized as it moves from the source to
the target.
Map Source Data to Target Data

A data map defines the source fields of the data, the destination fields in the data warehouse and any
data modifications that need to be accomplished to transform the data into the desired format for the
data warehouse. Some transformations require aggregating the source data to a coarser granularity,
such as summarizing individual item sales into daily sales by SKU. Other transformations involve
altering the source data itself as it moves from the source to the target. Some transformations decode
data into human readable form, such as replacing "1" with "on" and "0" with "off" in a status field. If
two source systems encode data destined for the same target differently (for example, a second
source system uses Yes and No for status), a separate transformation for each source system must
be defined. Transformations must be documented and maintained in the data maps. The relationship
between the source and target systems is maintained in a map that is referenced to execute the
transformation of the data before it is loaded in the data warehouse.
Develop Functional Elements
Design and implementation of the four ETL functional elements, Extraction, Transformation, Loading,
and meta data logging, vary from system to system. There will often be multiple versions of each
functional element.
Each functional element contains steps that perform individual tasks, which may execute on one of
several systems, such as the OLTP or legacy systems that contain the source data, the staging area
database, or the data warehouse database. Various tools and techniques may be used to implement
the steps in a single functional area, such as Transact-SQL, DTS packages, or custom applications
developed in a programming language such as Microsoft Visual Basic. Steps that are discrete in
one functional element may be combined in another.
Extraction
The extraction element may have one version to extract data from one OLTP data source, a different
version for a different OLTP data source, and multiple versions for legacy systems and other sources
of data. This element may include tasks that execute SELECT queries from the ETL staging database
against a source OLTP system, or it may execute some tasks on the source system directly and
others in the staging database, as in the case of generating a flat file from a legacy system and then
importing it into tables in the ETL database. Regardless of methods or number of steps, the extraction
element is responsible for extracting the required data from the source system and making it available
for processing by the next element.
Transformation
Frequently a number of different transformations, implemented with various tools or techniques, are
required to prepare data for loading into the data warehouse. Some transformations may be
performed as data is extracted, such as an application on a legacy system that collects data from
various internal files as it produces a text file of data to be further transformed. However,
transformations are best accomplished in the ETL staging database, where data from several data
sources may require varying transformations specific to the incoming data organization and format.
Data from a single data source usually requires different transformations for different portions of the
incoming data. Fact table data transformations may include summarization, and will always require
surrogate dimension keys to be added to the fact records. Data destined for dimension tables in the
data warehouse may require one process to accomplish one type of update to a changing dimension
and a different process for another type of update.
Transformations may be implemented using Transact-SQL, as is demonstrated in the code examples
later in this chapter, DTS packages, or custom applications.
Regardless of the number and variety of transformations and their implementations, the
transformation element is responsible for preparing data for loading into the data warehouse.
Loading
The loading element typically has the least variety of task implementations. After the data from the
various data sources has been extracted, transformed, and combined, the loading operation consists
of inserting records into the various data warehouse database dimension and fact tables.
Implementation may vary in the loading tasks, such as using BULK INSERT, bcp, or the Bulk Copy
API. The loading element is responsible for loading data into the data warehouse database tables.
Meta Data Logging
Meta data is collected from a number of the ETL operations. The meta data logging implementation
for a particular ETL task will depend on how the task is implemented. For a task implemented by
using a custom application, the application code may produce the meta data. For tasks implemented
by using Transact-SQL, meta data can be captured with Transact-SQL statements in the task
processes. The meta data logging element is responsible for capturing and recording meta data that
documents the operation of the ETL functional areas and tasks, which includes identification of data
that moves through the ETL system as well as the efficiency of ETL tasks.
Common Tasks
Each ETL functional element should contain tasks that perform the following functions, in addition to
tasks specific to the functional area itself:
Confirm Success or Failure. A confirmation should be generated on the success or failure of the
execution of the ETL processes. Ideally, this mechanism should exist for each task so that rollback
mechanisms can be implemented to allow for incremental responses to errors.
Scheduling. ETL tasks should include the ability to be scheduled for execution. Scheduling
mechanisms reduce repetitive manual operations and allow for maximum use of system resources
during recurring periods of low activity.
SQL SERVER 2000 ETL COMPONENTS
SQL Server 2000 includes several components that aid in the development and maintenance of ETL
systems:
Data Transformation Services (DTS): SQL Server 2000 DTS is a set of graphical tools and
programmable objects that lets you extract, transform, and consolidate data from disparate sources
into single or multiple destinations.
SQL Server Agent: SQL Server Agent provides features that support the scheduling of periodic
activities on SQL Server 2000, or the notification to system administrators of problems that have
occurred with the server.
Stored Procedures and Views: Stored procedures assist in achieving a consistent implementation
of logic across applications. The Transact-SQL statements and logic needed to perform a
commonly performed task can be designed, coded, and tested once in a stored procedure. A view
can be thought of as either a virtual table or a stored query. The data accessible through a view is
not stored in the database as a distinct object; only the SELECT statement for the view is stored in
the database.
Transact SQL: Transact-SQL is a superset of the SQL standard that provides powerful
programming capabilities that include loops, variables, and other programming constructs.
OLE DB: OLE DB is a low-level interface to data. It is an open specification designed to build on the
success of ODBC by providing an open standard for accessing all kinds of data.
Meta Data Services: SQL Server 2000 Meta Data Services provides a way to store and manage
meta data about information systems and applications. This technology serves as a hub for data
and component definitions, development and deployment models, reusable software components,
and data warehousing descriptions.
THE ETL STAGING DATABASE
In general, ETL operations should be performed on a relational database server separate from the
source databases and the data warehouse database. A separate staging area database server
creates a logical and physical separation between the source systems and the data warehouse, and
minimizes the impact of the intense periodic ETL activity on source and data warehouse databases. If
a separate database server is not available, a separate database on the data warehouse database
server can be used for the ETL staging area. However, in this case it is essential to schedule periods
of high ETL activity during times of low data warehouse user activity.
For small data warehouses with available excess performance and low user activity, it is possible to
incorporate the ETL system into the data warehouse database. The advantage of this approach is
that separate copies of data warehouse tables are not needed in the staging area. However, there is
always some risk associated with performing transformations on live data, and ETL activities must be
very carefully coordinated with data warehouse periods of minimum activity. When ETL is integrated
into the data warehouse database, it is recommended that the data warehouse be taken offline when
performing ETL transformations and loading.
Most systems can effectively stage data in a SQL Server 2000 database, as we describe in this
chapter. An ETL system that needs to process extremely large volumes of data will need to use
specialized tools and custom applications that operate on files rather than database tables. With
extremely large volumes of data, it is not practical to load data into a staging database until it has
been cleaned, aggregated, and stripped of meaningless information. Because it is much easier to
build an ETL system using the standard tools and techniques that are described in this chapter, most
experienced system designers will attempt to use a staging database, and move to custom tools only
if data cannot be processed during the load window.
What does "extremely large" mean and when does it become infeasible to use standard DTS tasks
and Transact-SQL scripts to process data from a staging database? The answer depends on the load
window, the complexity of transformations, and the degree of data aggregation necessary to create
the rows that are permanently stored in the data warehouse. As a conservative rule of thumb, if the
transformation application needs to process more than 1 gigabyte of data in less than an hour, it may
be necessary to consider specialized high performance techniques, which are outside the scope of
this chapter.
This section provides general information about configuring the SQL Server 2000 database server
and the database to support an ETL system staging area database with effective performance. ETL
systems can vary greatly in their database server requirements; server configurations and
performance option settings may differ significantly from one ETL system to another.
ETL data manipulation activities are similar in design and functionality to those of OLTP systems
although ETL systems do not experience the constant activity associated with OLTP systems. Instead
of constant activity, ETL systems have periods of high write activity followed by periods of little or no
activity. Configuring a server and database to meet the needs of an ETL system is not as
straightforward as configuring a server and database for an OLTP system.
For a detailed discussion of RAID and SQL Server 2000 performance tuning, see Chapter 20,
"RDBMS Performance Tuning Guide for Data Warehousing."
Server Configuration
Disk storage system performance is one of the most critical factors in the performance of database
systems. Server configuration options offer additional methods for adjusting server performance.
RAID
As with any OLTP system, the RAID level for the disk drives on the server can make a considerable
performance difference. For maximum performance of an ETL database, the disk drives for the server
computer should be configured with RAID 1 or RAID 10. Additionally, it is recommended that the
transaction logs, databases, and tempdb be placed on separate physical drives. Finally, if the
hardware controller supports write caching, it is recommended that write caching be enabled.
However, be sure to use a caching controller that guarantees that the controller cache contents will
be written to disk in case of a system failure.
Server Configuration Options (sp_configure)
No specific changes need to be made to the server configuration options in order to optimize
performance for an ETL system. It is recommended that these options be left at their default settings
unless there is a specific reason to modify them.
Database Configuration
In SQL Server 2000, database performance can be tuned by proper selection of settings for data file
growth and adjusting database configuration options.
Data File Growth
When creating a database, an initial size for the data files for the database and transaction log must
be specified. By default, SQL Server 2000 allows the data files to grow as much as necessary until
disk space is exhausted. It is important to size the database appropriately before loading any data
into it to avoid the I/O intensive operation of auto-growing data files. Failure to appropriately size the
data files initially for the database means that SQL will be forced to frequently increase the size of the
data files, which will degrade performance of the ETL processes.
Appropriate initial sizing of the data files can reduce the likelihood of SQL being forced to increase the
size of the database, which eliminates an intensive I/O operation. If a data file is allowed to
automatically grow, the file growth may be specified by a percentage or a set number value. The
growth value can be specified in megabytes (MB), kilobytes (KB), or percent. If percent is specified,
the increment size is the specified percentage of the file size at the time the increment occurs. If the
data file is too small, the growth increments will be frequent. For example, if a data file is initially
created at 10 MB and set to grow in 10 percent increments until it reaches 20 MB, SQL Server 2000
will perform eight auto-grow operations as the data file size increases to 20 MB. Therefore, it is
recommended that a fixed MB value be chosen for data file growth increments.
Finally, if the server uses SCSI disks, special care should be paid to preventing disk space
consumption from increasing beyond 85 percent of the capacity of the drive. Beyond 85 percent
consumption, SCSI disk performance begins to degrade. Therefore, it is recommended that the data
files for the database are set to grow automatically, but only to a predefined maximum size, which
should be no more than 85 percent capacity of the drive.
Database Configuration Options
Several database options can be adjusted to enhance the performance of an ETL database. For a
complete discussion of these options, see SQL Server Books Online. For more information about
database performance tuning, see Chapter 20, "RDBMS Performance Tuning Guide for Data
Warehousing."
The following table lists some database options and their setting that may be used to increase ETL
performance.
Option name
Setting
AUTO_CREATE_STATISTICS
Off
AUTO_UPDATE_STATISTICS
On
AUTO_SHRINK
Off
CURSOR_DEFAULT
LOCAL
RECOVERY Option
Bulk_Loaded
TORN_PAGE_DETECTION
On
Caution Different recovery model options introduce varying degrees of risk of data loss. It is
imperative that the risks be thoroughly understood before choosing a recovery model.
MANAGING SURROGATE KEYS
Surrogate keys are critical to successful data warehouse design: they provide the means to maintain
data warehouse information when dimensions change. For more information and details about
surrogate keys, see Chapter 17, "Data Warehouse Design Considerations."
The following are some common characteristics of surrogate keys:
Used as the primary key for each dimension table, instead of the original key used in the source
data system. The original key for each record is carried in the table but is not used as the primary
key.
May be defined as the primary key for the fact table. In general, the fact table uses a composite
primary key composed of the dimension foreign key columns, with no surrogate key. In schemas
with many dimensions, load and query performance will improve substantially if a surrogate key is
used. If the fact table is defined with a surrogate primary key and no unique index on the composite
key, the ETL application must be careful to ensure row uniqueness outside the database. A third
possibility for the fact table is to define no primary key at all. While there are systems for which this
is the most effective approach, it is not good database practice and should be considered with
caution.
Contains no meaningful business information; its only purpose is to uniquely identify each row.
There is one exception: the primary key for a time dimension table provides human-readable
information in the format "yyyymmdd ".
Is a simple key on a single column, not a composite key.
Should be numeric, preferably integer, and not text.
Should never be a GUID.
The SQL Server 2000 Identity column provides an excellent surrogate key mechanism.
ETL CODE EXAMPLES
Code examples in these sections use the pubs sample database included with SQL Server 2000 to
demonstrate various activities performed in ETL systems. The examples illustrate techniques for
loading dimension tables in the data warehouse; they do not take into consideration separate
procedures that may be required to update OLAP cubes or aggregation tables.
The use of temporary and staging tables in the ETL database allows the data extraction and loading
process to be broken up into smaller segments of work that can be individually recovered. The
temporary tables allow the source data to be loaded and transformed without impacting the
performance of the source system except for what is necessary to extract the data. The staging
tables provide a mechanism for data validation and surrogate key generation before loading
transformed data into the data warehouse. Transformation, validation, and surrogate key
management tasks should never be performed directly on dimension tables in the data warehouse.
The code examples in this chapter are presented as Transact-SQL, in order to communicate to the
widest audience. A production ETL system would use DTS to perform this work. A very simple system
may use several Execute SQL tasks linked within a package. More complex systems divide units of
work into separate packages, and call those subpackages from a master package. For a detailed
explanation of how to use DTS to implement the functionality described in this chapter, please see
SQL Server Books Online.
Tables for Code Examples
The examples use the authors table in the pubs database as the source of data. The following three
tables are created for use by the code examples.
Table name
Purpose
Authors_Temp Holds the data imported from the source system.
Authors_Stagin Holds the dimension data while it is being updated. The data for the authors will be
g
updated in this table and then the data will be loaded into the data warehouse
dimension table.
Authors_DW
Simulates the Authors dimension table in the data warehouse.
These are key points regarding the structures of these tables:
There is no difference between the structure of the authors table in the pubs database and
theAuthors_Temp table in the staging area. This allows for straightforward extraction of data from
the source system with minimum impact on source system performance.
The Author_Staging table is used to generate the surrogate key (Author_Key column) that is
used by the data warehouse. This table is also used to validate any data changes, convert data
types, and perform any other transformations necessary to prepare the data for loading into the
data warehouse.
The structure of the Author_Staging table in the staging area is the same as that of
theAuthors_DW table in the data warehouse. This allows for straightforward loading of the
dimension data from the staging database to the data warehouse. If the dimension table in the data
warehouse is small enough, it can be truncated and replaced with data from the staging table. In
many data warehouses, dimension tables are too large to be efficiently updated by dropping and
reloading them in their entirety. In this case, the tables in both the staging area and data warehouse
should contain a datetime column, which can be used to determine which records need to be
updated, inserted, or deleted in the data warehouse table.
The staging and data warehouse tables are identical after the data is loaded into the data
warehouse. This fact can be considered for use in backup strategy planning.
Define Example Tables
The following three Transact-SQL statements create
the Authors_Temp and Authors_Stagingtables, and the Authors_DW table that simulates the
Authors dimension table in the data warehouse:
Code Example 19.1
CREATE TABLE [Authors_Temp] (
[au_id] [varchar] (11) PRIMARY KEY CLUSTERED,
[au_lname] [varchar] (40) DEFAULT ('Missing'),
[au_fname] [varchar] (20) DEFAULT ('Missing'),
[phone] [char] (12) DEFAULT ('000-000-0000'),
[address] [varchar] (40) DEFAULT ('Missing'),
[city] [varchar] (20) DEFAULT ('Missing'),
[state] [char] (2) DEFAULT ('XX'),
[zip] [char] (5) DEFAULT ('00000'),
[contract] [bit] NOT NULL DEFAULT (0))
ON [PRIMARY]
GO
CREATE TABLE Authors_Staging (

[Author_Key] int NOT NULL IDENTITY (1,1) PRIMARY KEY CLUSTERED ,
[au_id] varchar (11) NOT NULL ,
[au_lname] varchar (40) NOT NULL DEFAULT ('Missing'),
[au_fname] varchar (20) NOT NULL DEFAULT ('Missing'),
[phone] char (12) NOT NULL DEFAULT ('000-000-0000'),
[address] varchar (40) NULL DEFAULT ('Missing'),

[city] varchar (20) NOT NULL DEFAULT ('Missing'),
[state] char (2) NOT NULL DEFAULT ('XX') ,
[zip] char (5) NOT NULL DEFAULT ('00000') ,
[contract] bit NOT NULL,
[DateCreated] smalldatetime NOT NULL DEFAULT (getdate()),
[DateUpdated] smalldatetime NOT NULL DEFAULT (getdate())
)ON [PRIMARY]
GO
CREATE TABLE [Authors_DW] (

[Author_Key] [int] NOT NULL PRIMARY KEY CLUSTERED,
[au_id] [varchar] (11) NOT NULL,
[au_lname] [varchar] (40) NOT NULL DEFAULT ('Missing'),
[au_fname] [varchar] (20) NOT NULL DEFAULT ('Missing'),
[phone] [char] (12) NOT NULL DEFAULT ('000-000-0000'),
[address] [varchar] (40) NULL DEFAULT ('Missing'),

[city] [varchar] (20) NOT NULL DEFAULT ('Missing'),
[state] [char] (2) NOT NULL DEFAULT ('XX'),
[zip] [char] (5) NOT NULL DEFAULT ('00000'),
[contract] [bit] NOT NULL,
[DateUpdated] smalldatetime NOT NULL DEFAULT (getdate())
) ON [PRIMARY]
GO
Populate Example Tables
The following three Transact-SQL statements populate the temporary, staging, and data warehouse
sample tables by loading all of the author records except the record for author Johnson White, which
will be inserted later to illustrate a technique for adding records to the data warehouse dimension
table:
Code Example 19.2
--Populate the Authors_Temp table with all author records except Johnson White's
INSERT INTO Authors_Temp
SELECT * FROM Authors
WHERE AU_ID <> '172-32-1176'
GO
--Populate the Authors_Staging table from Authors_Temp

INSERT INTO Authors_Staging (au_id, au_lname, au_fname, phone, address, city, state,
zip, contract)
SELECT au_id, au_lname, au_fname, phone, address, city, state, zip, contract
FROM Authors_Temp
GO
--Populate the simulated data warehouse dimension table, Authors_DW

INSERT INTO Authors_DW
SELECT * FROM Authors_Staging
GO
The contents of the three tables now simulate the state following the completion of all previous ETL
processing before the author Johnson White is added to the source data table.
Inserting New Dimension Records
Loading new author records is a relatively simple task. If the extraction method is capable of
generating a change set (a set of records that have been altered since the last data extraction) from
the source system, we load the change set into the temporary table. If we cannot generate a change
set from the source system, we will have to load the entire data set from the source system into the
temporary table, even if only a single record has changed.
The following Transact-SQL code demonstrates a simple technique for loading new rows into the
Authors dimension. This example assumes that there is a primary key on the source system that we
can use and it assumes that we do not have a change set.
Code Example 19.3
--Truncate any data that currently exists in the Authors_Temp table
TRUNCATE TABLE Authors_Temp
GO
--Load all of the data from the source system into the Authors_Temp table
INSERT INTO Authors_Temp
SELECT * FROM Authors
GO
--Set a starting value for the Contract field for two records
-- for use by future examples
UPDATE Authors_Temp
SET Contract = 0
WHERE state = 'UT'
GO
--Locate all of the new records that have been added to the source system by
--comparing the new temp table contents to the existing staging table contents
--and add the new records to the staging table
zip, contract)
SELECT T.au_id, T.au_lname, T.au_fname, T.phone, T.address, T.city, T.state, T.zip,
T.contract
FROM Authors_Temp T LEFT OUTER JOIN
Authors_Staging S ON T.au_id = S.au_id
WHERE (S.au_id IS NULL)
GO
--Locate all of the new records that are to be added to the data warehouse
--and insert them into the data warehouse by comparing Authors_Staging to Authors_DW
INSERT INTO Authors_DW (Author_Key, au_id, au_lname, au_fname, phone, address, city,
state, zip, contract,
DateCreated, DateUpdated)
SELECT S.Author_Key, S.au_id, S.au_lname, S.au_fname, S.phone, S.address, S.city,
S.state, S.zip, S.contract,
S.DateCreated, S.DateUpdated
FROM Authors_Staging S LEFT OUTER JOIN
Authors_DW D ON S.au_id = D.au_id
WHERE (D.au_id IS NULL)
GO
Managing Slowly Changing Dimensions
This section describes various techniques for managing slowly changing dimensions in the data
warehouse. "Slowly changing dimensions" is the customary term used for dimensions that contain
attributes that, when changed, may affect grouping or summarization of historical data. Design
approaches to dealing with the issues of slowly changing dimensions are commonly categorized into
the following three change types:
Type 1: Overwrite the dimension record
Type 2: Add a new dimension record
Type 3: Create new fields in the dimension record
Type 1 and Type 2 dimension changes are discussed in this section. Type 3 changes are not
recommended for most data warehouse applications and are not discussed here. For more
information and details about slowly changing dimensions, see Chapter 17, "Data Warehouse Design
Considerations."
Type 1 and Type 2 dimension change techniques are used when dimension attributes change in
records that already exist in the data warehouse. The techniques for inserting new records into
dimensions (discussed earlier in the section "Inserting New Dimension Records") apply to all
dimensions regardless of whether changes to dimension attributes are incorporated using Type 1 or
Type 2 change techniques.
The code examples in the following sections demonstrate techniques for managing Type 1 and Type
2 dimension changes. The examples have been kept simple to maintain clarity for technique
illustration purposesthe examples assume that all changes for a dimension will be of the same
type, whereas, in reality, most dimensions include some attributes that require Type 2 changes and
other attributes that can be maintained using Type 1 changes. For example, a retailer may decide that
a change in the marital status of a customer should be treated as a Type 2 change, whereas a
change of street address for the same customer should be treated as a Type 1 change. Therefore, it
is important to document all of the attributes in a dimension and, for each attribute, whether a value
change should be applied as Type 1 or a Type 2 change.
Type 1: Overwrite the Dimension Record
A change to a dimension attribute that is never used for analysis can be managed by simply changing
the data to the new value. This type of change is called a Type 1 change. For example, a change to a
customer's street address is unlikely to affect any summarized information and the previous street
address can be discarded without consequence.
Type 1 dimension changes are straightforward to implement. The following Transact-SQL code
demonstrates a simple Type 1 technique for updating existing rows in the Authors dimension. For this
example, we will change some data in the Authors_Temp table to simulate changed records
received as a result of updates to the authors table in the source database. The value for the
Contract field is assumed to be eligible for Type 1 changes in this example. In a later section, the
Contract field will be updated using a Type 2 change. The following example assumes that there is a
primary key on the source system that we can use and it assumes that we do not have a change set:
Code Example 19.4
--Change the Authors_Temp table to simulate updates received from the source system
UPDATE Authors_Temp
SET Contract = 0
WHERE state = 'UT'
GO
--Update the Authors_Staging table with the new values in Authors_Temp

UPDATE Authors_Staging
SET Contract = T.Contract,
DateUpdated = getdate()
FROM Authors_Temp T INNER JOIN Authors_Staging S
ON T.au_id = S.au_id
WHERE T.Contract <> S.Contract
GO
--Update the Author_DW with the new data in Authors_Staging

UPDATE Authors_DW
SET Contract = S.Contract,
FROM Authors_Staging S INNER JOIN Authors_DW D
ON S.Author_Key = D.Author_Key
WHERE S.Contract <> D.Contract
GO
Type 2: Add a New Dimension Record
Type 2 changes cause history to be partitioned at the event that triggered the change. Data prior to
the event continues to be summarized and analyzed as before; new data is summarized and
analyzed in accordance with the new value of the data. The technique for implementing a Type 2
change is to keep the existing dimension record and add a new record that contains the updated data
for the attribute or attributes that have changed. Values are copied from the existing record to the new
record for all fields that have not changed. A new surrogate key value is created for the new record
and the record is added to the dimension table. Fact records that apply to events subsequent to the
Type 2 change must be related to the new dimension record.
Although it is relatively straightforward to implement Type 2 change techniques in the ETL process to
manage slowly changing dimensions, the data associated with a dimension member becomes
fragmented as such changes are made. Data warehouse analysis and reporting tools must be
capable of summarizing data correctly for dimensions that include Type 2 changes. To minimize
unnecessary fragmentation, a Type 2 change should not be used if a Type 1 change is appropriate.
The techniques used to insert new records into a Type 2 dimension are the same as the ones used to
insert new records into a Type 1 dimension. However, the techniques used to track updates to
dimension records are different.
The following Transact-SQL code demonstrates a simple technique for applying Type 2 changes for
existing rows in the Authors dimension. Unlike a Type 1 change, existing records are not updated in a
Type 2 dimension. Instead, new records are added to the dimension to contain the changes to the
source system records. In this example, the values of the contract field changed in the Type 1
example are changed to different values and we now assume the contract field is to be managed as a
Type 2 change.
Notice that the Transact-SQL statement used to load updated records into the staging table is the
same as the one used to insert new records into the staging table except that the predicate clause in
the two statements differ. When loading new records, the WHERE clause uses the auth_id field to
determine which records are new. When inserting records for Type 2 changes, the WHERE clause
causes new records to be added when the value of the attribute of interest (contract) in the temporary
table differs from the attribute value in the staging table.
Code Example 19.5
--Change the Authors_Temp table to simulate updates received from the source system
--This change reverses the change made in the Type 1 example by setting Contract to 1
UPDATE Authors_Temp
SET Contract = 1
WHERE state = 'UT'
GO
--For example purposes, make sure the staging table records have a different value
--for the contract field for the UT authors
UPDATE Authors_Staging
SET Contract = 0
WHERE state = 'UT'
GO
--Insert new records into the Staging Table for those records in the temp table
--that have a different value for the contract field
zip, contract)
SELECT T.au_id, T.au_lname, T.au_fname, T.phone, T.address, T.city, T.state, T.zip,
T.contract
FROM Authors_Temp T
LEFT OUTER JOIN Authors_Staging S ON T.au_id = S.au_id
WHERE T.Contract <> S.Contract
GO
--Insert the new records into the data warehouse Table

INSERT INTO Authors_DW (Author_Key, au_id, au_lname, au_fname, phone, address, city,
state, zip, contract,
SELECT S.Author_Key, S.au_id, S.au_lname, S.au_fname, S.phone, S.address, S.city,
S.state, S.zip, S.contract,
S.DateCreated, S.DateUpdated
FROM Authors_Staging S LEFT OUTER JOIN
Authors_DW D ON S.Author_Key = D.Author_Key
WHERE (D.Author_Key IS NULL)
GO
Managing the Fact Table
After all dimension records have been loaded and updated, the fact table also must be loaded with
new data. The fact table must be loaded after the dimension tables so the surrogate keys added to
the dimension records during the ETL processes can be used as foreign keys in the fact table. This
section demonstrates techniques for loading the fact table.
For purposes of these examples, a table (Fact_Source) is created that simulates a data table in a
source system from which fact data can be extracted. The Fact_Source table data is a combination
of data found in the Sales and TitleAuthor tables in the pubs database.
The following table lists definitions of the tables created for use with the examples that follow.
Table name Purpose
Fact_Source A simulated source data table that will be used in the example code. This table is a
combination of Sales and TitleAuthor in the pubs database.
Fact_Temp Receives data imported from the source system.
Fact_Stagin Holds the fact table data during transformation and surrogate key operations. Data is
g
loaded to the data warehouse fact table after ETL operations are complete.
Fact_DW
The fact table in the data warehouse.
Titles_DW A dimension table for Titles to demonstrate the use of surrogate keys in the fact table.
Store_DW
A dimension table for Stores to demonstrate the use of surrogate keys in the fact
tables.
Several key points about the structures of these tables should be noted:
There is no difference between the structures of the Fact_Source table and the Fact_Temp tables.
This allows for the easiest method to extract data from the source system so that transformations
on the data do not impact the source system.
The Fact_Staging table is used to add the dimension surrogate keys to the fact table records. This
table is also used to validate any data changes, convert any data types, and so on.
The structures of the Fact_Staging and Fact_DW tables do not match. This is because the final
fact table in the data warehouse does not store the original keysjust the surrogate keys.
The fact table key is an identity column that is generated when the transformed data is loaded into
the fact table. Since we will not be updating the records once they have been added to
theFact_DW table, there is no need to generate the key prior to the data load into the fact table.
This is not how the key column is generated in dimension tables. As discussed above, the decision
to use an identity key for a fact table depends on the complexity of the data warehouse schema and
the performance of load and query operations; this example implements an identity key for the fact
table.
The following Transact-SQL statements create the tables defined above:
Code Example 19.6
--Create the simulated source data table
CREATE TABLE [Fact_Source] (
[stor_id] [char] (4) NOT NULL ,
[ord_num] [varchar] (20) NOT NULL ,
[ord_date] [datetime] NOT NULL ,
[qty] [smallint] NOT NULL ,
[payterms] [varchar] (12) NOT NULL ,
[title_id] [tid] NOT NULL
) ON [PRIMARY]
GO
--Create the example temporary source data table used in the ETL database
CREATE TABLE [Fact_Temp] (

[qty] [smallint] NOT NULL ,
[title_id] [tid] NOT NULL
) ON [PRIMARY]
GO
--Create the example fact staging table

CREATE TABLE [Fact_Staging] (
[qty] [smallint] NOT NULL DEFAULT (0),
[title_id] [tid] NOT NULL,
[Store_Key] [int] NOT NULL DEFAULT (0),
[Title_Key] [int] NOT NULL DEFAULT (0)
) ON [PRIMARY]
GO
--Create the example data warehouse fact table

CREATE TABLE [Fact_DW] (
[Store_Key] [int] NOT NULL DEFAULT (0),
[Title_Key] [int] NOT NULL DEFAULT (0),

[qty] [smallint] NOT NULL DEFAULT (0),
[Fact_Key] [int] IDENTITY (1, 1) NOT NULL PRIMARY KEY CLUSTERED
) ON [PRIMARY]
GO
--Create the example titles dimension table

CREATE TABLE [Titles_DW] (
[title_id] [tid] NOT NULL,
[title] [varchar] (80) NOT NULL,
[type] [char] (12) DEFAULT ('UNDECIDED'),
[pub_id] [char] (4) NULL,
[price] [money] NULL,
[advance] [money] NULL,
[royalty] [int] NULL,
[ytd_sales] [int] NULL,
[notes] [varchar] (200) NULL,
[pubdate] [datetime] NOT NULL DEFAULT (getdate()),
[Title_Key] [int] NOT NULL IDENTITY (1,1) PRIMARY KEY CLUSTERED
) ON [PRIMARY]
GO
--Create the example stores dimension table

CREATE TABLE [Stores_DW] (
[stor_id] [char] (4) NOT NULL,
[stor_name] [varchar] (40) NULL,
[stor_address] [varchar] (40) NULL,
[city] [varchar] (20) NULL,

[state] [char] (2) NULL,
[zip] [char] (5) NULL,
[Store_Key] [int] IDENTITY (1,1) PRIMARY KEY CLUSTERED
) ON [PRIMARY]
GO
The following statements populate the sample fact and dimension tables and provide a base set of
data for the remainder for the examples. The Fact_Temp and Fact_Source may appear to be
redundant, but Fact_Source is only used to simulate a source table in an OLTP system.
Code Example 19.7
--Load the simulated Fact_Source table with example data
INSERT INTO Fact_Source
SELECT S.*
FROM titleauthor TA INNER JOIN sales S ON TA.title_id = S.title_id
GO
--Load the Fact_Temp table with data from the Fact_Source table
INSERT INTO Fact_Temp
SELECT *
FROM Fact_Source
GO
--Load the example dimension for Titles

INSERT INTO Titles_DW
SELECT *
FROM Titles
GO
--Load the example dimension for Stores
INSERT INTO Stores_DW

SELECT *
FROM Stores
GO
This completes the preparation of the sample data. The remaining examples demonstrate the tasks
that prepare data for loading and load it into the data warehouse fact table.
The following code loads the Fact_Staging table. Notice that the Store_Key and Title_Key columns
that are used for surrogate keys contain zeros when the data is first loaded into the staging table.
This is because NULLs are not allowed in these columns. The prevention of NULLs allows for a very
clean data load and it negates the need to do NULL logic checks in the ETL code or the reporting
system. The zeros in the column also provide an easy mechanism for locating invalid data in the
dimension and fact table data. If a zero appears in either column in the final fact table, then the ETL
logic failed to handle a dimension attribute. It is good practice to always add a dimension record with
zero key and assign it the description of "unknown." This helps preserve relational integrity in the data
warehouse and allows reporting systems to display the invalid data, so that corrections can be made
to the ETL logic or the source data.
Code Example 19.8
--Load the Fact_Staging table with data in the Fact_Temp table
INSERT INTO Fact_Staging (stor_id, ord_num, ord_date, qty, payterms, title_id,
Store_Key, Title_Key)
SELECT stor_id, ord_num, ord_date, qty, payterms, title_id, 0, 0
FROM Fact_Temp
GO
Now that the Fact_Staging table is loaded, the surrogate keys can be updated. The techniques for
updating the surrogate keys in the fact table will differ depending on whether the dimension contains
Type 2 changes. The following technique can be used for Type 1 dimensions:
Code Example 19.9
--Update the Fact_Staging table with the surrogate key for Titles
--(Type 1 dimension)
UPDATE Fact_Staging
SET Title_Key = T.Title_Key
FROM Fact_Staging F INNER JOIN
Titles_DW T ON F.title_id = T.title_id
GO
--Update the Fact_Staging table with the surrogate key for Store
--(Type 1 dimension)
UPDATE Fact_Staging
SET Store_Key = S.Store_Key
Stores_DW S ON F.Stor_id = S.Stor_id
GO
The technique above will not work for dimensions that contain Type 2 changes, however, because
there may be more than one dimension record that contains the original source key. The following
technique is appropriate for Type 2 dimensions:
Code Example 19.10
--Add a few new rows to the Stores_DW table to demonstrate technique
--Duplicate Store records are added that reflect changed store names
INSERT INTO Stores_DW (stor_id, stor_name, stor_address, city, state, zip)
SELECT stor_id, 'New ' + stor_name, stor_address, city, state, zip
FROM Stores_DW
WHERE state = 'WA'
GO
--Add some new rows to fact table to demonstrate technique

Store_Key, Title_Key)
SELECT stor_id, ord_num, ord_date, qty, payterms, title_id, 0, 0
FROM Fact_Temp
GO
--Update the fact table. Use the maximum store key

--to relate the new fact data to the latest store record.
BEGIN TRANSACTION
--get the maximum store_key for each stor_id

SELECT MAX(STORE_KEY) AS Store_Key, stor_id
INTO #Stores
FROM Stores_DW
GROUP BY stor_id
ORDER BY stor_id
--update the fact table

UPDATE Fact_Staging
SET Store_Key = S.Store_Key
#Stores S ON F.stor_id = S.stor_id
WHERE F.Store_Key = 0
--drop the temporary table

DROP TABLE #Stores
COMMIT TRANSACTION
GO
After the fact data has been successfully scrubbed and transformed, it needs to be loaded into the
data warehouse. If the ETL database is not on the same server as the data warehouse database,
then the data will need to be transferred using DTS, bcp, or another mechanism. An efficient
approach is to use bcp to export the data from the ETL database, copy the data to the target server,
and then use BULK INSERT to update the target database. However, if the databases are on the
same server, a simple INSERT statement will load the new fact table rows:
Code Example 19.11
--Load the new fact table rows into the data warehouse
INSERT INTO Fact_DW (ord_num, ord_date, qty, payterms, Store_Key, Title_Key)
SELECT ord_num, ord_date, qty, payterms, Store_Key, Title_Key
FROM Fact_Staging
GO
Finally, the following SELECT statement shows the data warehouse fact table, complete with Type 2
dimension for the stores dimension:
Code Example 19.12
--Demonstrate the success of the technique
SELECT S.stor_id, S.Store_Key, S.stor_name, F.ord_num, F.ord_date, F.qty, F.payterms
FROM Stores_DW S INNER JOIN
Fact_DW F ON S.Store_Key = F.Store_Key
ORDER BY S.stor_id, S.Store_Key
GO
Advanced Techniques
While the sample techniques described above will work for small to medium-sized dimensions, they
will not work for large dimensions. For large dimensions, variations of these techniques can provide
greater efficiency. The code examples in this topic show some advanced techniques for Type 2
dimensions.
One of the key design decisions in the above techniques centers on the use of the staging tables. In
the techniques illustrated above, the staging tables are exact copies of the final data warehouse
dimension tables. However, the efficiency of the above techniques decreases as the number of rows
in the dimension increase due to records added for Type 2 changes. For very large dimensions
(millions of rows), the above technique will require massive amounts of processing power to
complete. Therefore, for large dimensions, we need to introduce a variation of the above technique
that will allow the system to scale with the data warehouse.
This variation involves creating a "current version" dimension table for use in the ETL process that
contains only a single row for each of the dimension members. This record contains the current
attributes of the dimension member. For example, if we have a Type 2 dimension for stores, and the
data for the store Bookbeat has undergone three Type 2 changes, then the current version table
would not contain all four records for the store. Instead, the table contains a single row for Bookbeat
that contains all of the current information for it, including the current surrogate key value for the
dimension member. This creates a smaller table with fewer rows that allows for faster access during
the ETL process.
The following code incorporates a Store_Current table to demonstrate this technique for the Stores
dimension. The table below describes each of the tables used in the example.
Table name
Purpose
AuthorsStores_TempHolds the data imported from the source system.
Stores_Staging
Holds the dimension data while it is being updated. The data for the stores will
be updated in this table and then the data will be loaded into the data
warehouse dimension table.
Stores_Current
Contains a single record for each store to track the current information for the
store.
Stores_DW
Simulates the Stores dimension table in the data warehouse.
The following statements create the four tables:
Code Example 19.13
DROP TABLE Stores_DW
GO
CREATE TABLE [Stores_Temp] (

[zip] [char] (5) NULL
) ON [PRIMARY]
GO
CREATE TABLE [Stores_Staging] (

[DateUpdated] smalldatetime NOT NULL DEFAULT (getdate()),
[Store_Key] [int] IDENTITY (1,1) PRIMARY KEY CLUSTERED
) ON [PRIMARY]
GO
CREATE TABLE [Stores_Current] (


[Store_Key] [int] PRIMARY KEY CLUSTERED
) ON [PRIMARY]
GO
CREATE TABLE [Stores_DW] (

[Store_Key] [int] PRIMARY KEY CLUSTERED
) ON [PRIMARY]
GO
The following statements populate the Stores_Temp, Stores_Staging, and Stores_Currentsample
tables to provide a base set of data that will be used in the remainder of the example:
Code Example 19.14
--Load the Stores_Temp table with the default set of data
INSERT INTO Stores_Temp
SELECT * FROM Stores
GO
--Load the Stores_Staging table with the default set of data

INSERT INTO Stores_Staging (stor_id, stor_name, stor_address, city, state, zip,
SELECT stor_id, stor_name, stor_address, city, state, zip, getdate(), getdate()
FROM Stores_Temp
GO
--Load the Stores_Current table with the default set of data

INSERT INTO Stores_Current (stor_id, stor_name, stor_address, city, state, zip,
DateCreated, DateUpdated, store_key)
SELECT stor_id, stor_name, stor_address, city, state, zip, DateCreated, DateUpdated,
store_key
FROM Stores_Staging
GO
The following code adds some new records into the Stores_Staging table to simulate Type 2
changes to the Stores dimension. The new records reflect changes to existing store data; no new
store records are added.
Code Example 19.15
--Insert some change records into Store_Staging to demonstrate the technique
--Duplicate records are added that reflect changes to store names for some stores
INSERT INTO Stores_Staging (stor_id, stor_name, stor_address, city, state, zip)
SELECT stor_id, stor_name + ' New', stor_address, city, state, zip
FROM Stores_staging
WHERE state <> 'ca'
GO
Records for new stores are loaded into Stores_Current before starting to process stores with change
records. The following Transact-SQL code loads new stores in the Stores_Staging table into
the Stores_Current table. This technique is the same as the one documented earlier in the chapter.
Code Example 19.16
--Insert any new stores in Stores_Staging into the Stores_Current table
--In this example there should not be any new stores
INSERT INTO Stores_Current (stor_id, stor_name, stor_address, city, state, zip,

SELECT S.stor_id, S.stor_name, S.stor_address, S.city, S.state, S.zip, S.DateCreated,
S.DateUpdated
FROM Stores_Staging S LEFT OUTER JOIN Stores_Current C ON S.stor_id = C.stor_id
WHERE (c.Store_Key IS NULL)
GO
The real change in this technique involves changing the way that dimension members are updated.
The following Transact-SQL code demonstrates the alternative way to update the dimension
members and load them into the data warehouse dimension. Once the new members of the
dimension have been loaded, the next step is to check existing members for attributes changes that
require Type 2 changes to the dimension. This example checks the stor_name attribute and updates
the row in theStores_Current table for every store that has had a name change (in this example, all
stores that do not exist in CA).
Code Example 19.17
--Update Store_Current table for all stores that have had a name change
UPDATE Stores_Current
SET stor_name = S.stor_name,
Store_key = S.Store_Key,
FROM Stores_Staging S
INNER JOIN Stores_Current C ON S.stor_id = C.stor_id
WHERE S.stor_name <> C.stor_name
GO
Now that all of the dimension records have been updated with the latest data, the surrogate keys can
be updated for the fact table data with the following Transact-SQL statement. This technique is more
efficient because a temporary table does not have to be created to determine the current value of the
dimension table key.
Code Example 19.18
--generate some fact data rows that do not have a store_key
store_key, title_key)
SELECT stor_id, ord_num, getdate(), qty, payterms, title_id, 0, 0
FROM Fact_Staging
WHERE QTY < 20
GO
--Update the fact data using the Store_Key key from the Store_Current table
--to relate the new fact data to the latest store record
UPDATE Fact_Staging
SET Store_Key = C.Store_Key
Stores_Current C ON F.stor_id = C.stor_id
WHERE F.Store_Key = 0
GO
Meta Data Logging
A critical design element in successful ETL implementation is the capability to generate, store and
review meta data. Data tables in a data warehouse store information about customers, items
purchased, dates of purchase, and so on. Meta data tables store information about users, query
execution times, number of rows retrieved in a report, etc. In ETL systems, meta data tables store
information about transformation execution time, number of rows processed by a transformation, the
last date and time a table was updated, failure of a transformation to complete, and so on. This
information, if analyzed appropriately, can help predict what is likely to occur in future transformations
by analyzing trends of what has already occurred.
In the code examples that follow, the terms "Job" and "Step" are used with the following meanings:
A "Job" is an ETL element that is either executed manually or as a scheduled event. A Job contains
one or more steps.
A "Step" is an individual unit of work in a job such as an INSERT, UPDATE, or DELETE operation.
A "Threshold" is a range of values defined by a minimum value and a maximum value. Any value
that falls within the specified range is deemed acceptable. Any value that does not fall within the
range is unacceptable. For example, a processing window is a type of threshold. If a job completes
within the time allotted for the processing window, then it is acceptable. If it does not, then it is not
acceptable.
Designing meta data storage requires careful planning and implementation. There are dependencies
between tables and order of precedence constraints on records. However, the meta data information
generated by ETL activities is critical to the success of the data warehouse. Following is a sample set
of tables that can be used to track meta data for ETL activities.
Job Audit
ETL jobs produce data points that need to be collected. Most of these data points are aggregates of
the data collected for the job steps and could theoretically be derived by querying the job step audit
table. However, the meta data for the job itself is important enough to warrant storage in a separate
table. Below are sample meta data tables that aid in tracking job information for each step in an ETL
process.
tblAdmin_Job_Master
This table lists all of the jobs that are used to populate the data warehouse. These are the fields
intblAdmin_Job_Master:
Field
Definition
JobNumber
A unique identifier for the record, generally an identity column.
JobName
The name (description) for the job. For example, "Load new dimension data."
MinThreshRecords The minimum acceptable number of records affected by the job.
MaxThreshRecords The maximum acceptable number of records affected by the job.
MinThreshTime
The minimum acceptable execution time for the job.
MaxThreshTime
The maximum acceptable execution time for the job.
CreateDate
The date and time the record was created.
tblAdmin_Audit_Jobs
This table is used to track each specific execution of a job. It is related to
the tblAdmin_Job_Mastertable using the JobNumber column. These are the fields
in tblAdmin_Audit_Jobs:
Field
Definition
JobNumber
A unique identifier for the record, generally an identity column.
JobName
The name (description) for the job. For example, "Load new dimension data."
StartDate
The date and time the job was started.
EndDate
The date and time the job ended.
NumberRecords The number of records affected by the job.
Successful
A flag-indicating if the execution of the job was successful.
This data definition language will generate the above audit tables:
Code Example 19.19
CREATE TABLE [dbo].[tblAdmin_Job_Master] (
[JobNumber] [int] IDENTITY (1, 1) NOT NULL
CONSTRAINT UPKCL_Job PRIMARY KEY CLUSTERED,
[JobName] [varchar] (50) NULL DEFAULT ('Missing'),
[MinThreshRecords] [int] NOT NULL DEFAULT (0),
[MaxThreshRecords] [int] NOT NULL DEFAULT (0),
[MinThreshTime] [int] NOT NULL DEFAULT (0),
[MaxThreshTime] [int] NOT NULL DEFAULT (0)
GO
CREATE TABLE [dbo].[tblAdmin_Audit_Jobs] (

[JobNumber] [int] IDENTITY (1, 1) NOT NULL
CONSTRAINT UPKCL_Job PRIMARY KEY CLUSTERED,
[JobName] [varchar] (50) NULL DEFAULT ('Missing'),
[StartDate] [smalldatetime] NOT NULL DEFAULT (getdate()),
[EndDate] [smalldatetime] NOT NULL DEFAULT ('01/01/1900'),

[NumberRecords] [int] NOT NULL DEFAULT (0),
[Successful] [bit] NOT NULL DEFAULT (0),
GO
Step Audit
Many ETL jobs are multi-step, complicated transformations that involve INSERT, UPDATE, and
DELETE statements with branching logic and an execution dependency. It is important to record meta
data that tracks the successful completion of an operation, when it happened and how many rows it
processed. This information should be stored for every step in an ETL job. Below are sample meta
data tables that aid in tracking information for each step in an ETL job.
tblAdmin_Step_Master
This table lists all of the steps in a job. These are the fields in tblAdmin_Step_Master:
Field
Definition
JobNumber
The unique number of the job that this step is associated with.
StepSeqNumber The step number within the object that executed the unit of work. Frequently, ETL
jobs contain more than a single unit of work and storing the step number allows
for easy debugging and specific reporting. If the object only has a single step,
then the value of this field is "1".
StepDescription
A description of the step. For example, "Inserted records into tblA."
Object
The name of the object. For example, the name of a stored procedure or DTS
package that accomplishes the step.
NumberRecords The number of records affected by the step.
MinThreshRecords The minimum "acceptable" number of records affected by the step.
MaxThreshRecord The maximum "acceptable" number of records affected by the step.
s
MinThreshTime
The minimum "acceptable" execution time for the step.
MaxThreshTime
The maximum "acceptable" execution time for the step.
CreateDate
The date and time the record was created.
StepNumber
A unique value assigned to the record, generally an identity column.
tbl_Admin_Audit_Step
This table is used to track each specific execution of a job step. It is related to
thetblAdmin_Step_Master table using the StepNumber column. These are the fields
intblAdmin_Audit_Step:
Field
Definition
RecordID
JobAuditID
Used to tie the specific execution on a job step to the specific execution of a job.
StepNumber
The step number executed.
Parameters
Any parameters sent to the job step for the specific execution instance. These are
the parameter values, not a list of the parameters.
NumberRecord The number of records affected by the job step.
s
StartDate
The date and time the job step was started.
EndDate
The date and time the job step ended.
UserName
The name of the user that executed the job step.
Below is the data definition language that will generate the job step audit tables above.
Code Example 19.20
CREATE TABLE [dbo].[tblAdmin_Step_Master] (
[JobNumber] [int] NOT NULL DEFAULT (1),

[StepSeqNumber] [int] NOT NULL DEFAULT (1),
[StepDescription] [varchar] NOT NULL DEFAULT ('Missing'),
[Object] [varchar] (50) NULL DEFAULT ('Missing'),
[MinThreshRecords] [int] NOT NULL DEFAULT (0),
[MaxThreshRecords] [int] NOT NULL DEFAULT (0),
[MinThreshTime] [int] NOT NULL DEFAULT (0),
[MaxThreshTime] [int] NOT NULL DEFAULT (0),
[StepNumber] [int] IDENTITY (1, 1) NOT NULL
CONSTRAINT UPKCL_PrimaryKey PRIMARY KEY CLUSTERED) ON [PRIMARY]
GO
CREATE TABLE [dbo].[tblAdmin_Audit_Step] (

[RecordID] [int] IDENTITY (1, 1) NOT NULL
CONSTRAINT UPKCL_PrimaryKey PRIMARY KEY CLUSTERED,
[Object] [varchar] (50) NULL DEFAULT ('Missing'),
[StepNumber] [tinyint] NOT NULL DEFAULT (1),
[StepDescription] [varchar] NOT NULL DEFAULT ('Missing'),
[Parameters] [varchar] (100) NULL,
[NumberRecords] [int] NOT NULL DEFAULT (1),
[EndDate] [smalldatetime] NOT NULL DEFAULT ('01/01/1900'),
[UserName] [varchar] (20) NOT NULL DEFAULT ('Missing')) ON [PRIMARY]
GO
Error Tracking
Another important type of meta data about transformations is information that tracks what failed and
why. ETL jobs produce errors. Just as tracking successful execution is important, tracking failures is
equally important. Below is a sample meta data table that aids in tracking error information for each
step in an ETL job. This table is designed to track SQL Server 2000 errors, although it could be
modified to track OLE DB errors as well.

Note In SQL Server 2000, only the error number can be trapped, not the generated error message.
tblAdmin_Audit_Errors
This table lists all of the errors that are generated during a job step. These are the fields
intblAdmin_Audit_Errors:
Field
Definition
RecordID
StepNumber The step number executed that generated the error.
Parameters Any parameters sent to the job step for the specific execution instance. These are the
parameter values, not a list of the parameters.
ErrorNumber The error number raised by SQL Server 2000 (generally the @@ERROR number).
RecordCoun The number of records affected by the job step.
t
UserName The name of the user that executed the job step.
EndDate
The date and time the job step ended.
Code Sample: Job Audit
The following stored procedures demonstrate one method of logging job level meta data. The
usp_Admin_Audit_Job_Start procedure indicates the start of an ETL job and should be the very first
stored procedure executed in the ETL job:
Code Example 19.21
ALTER PROCEDURE usp_Admin_Audit_Job_Start
@JobNumber int = 1 --The number of the job being executed (from the mast job table)
AS
SET NOCOUNT ON --SET NoCount ON
--DECLARE variables
DECLARE @ErrorNumber int --the number of the SQL error generated
DECLARE @ErrorRowCount int --the number of rows in the unit of work affected by
error
DECLARE @Startdate smalldatetime --the datetime the load job started
DECLARE @EndDate smalldatetime --the datetime the load job ended
--INSERT the first record (start time) for the job into the tblAdmin_Audit_Jobs table
BEGIN TRANSACTION
SET @StartDate = getdate() --set a start date for the batch
SET @EndDate = '01/01/1900' --set a bogus endate for the batch
insert into tblAdmin_Audit_Jobs (JobNumber, StartDate, EndDate, NumberRecords,

Successful)
values (@JobNumber, @StartDate, @EndDate, 0, 0)
SELECT @ErrorNumber = @@error, @ErrorRowCount = @@rowcount
If @ErrorNumber <> 0
BEGIN
ROLLBACK TRANSACTION
GOTO Err_Handler
END
COMMIT TRANSACTION
RETURN (0)
Err_Handler:
exec usp_AdminError @@ProcID, 'none', @ErrorNumber, @ErrorRowCount
RETURN (1)
GO
The following stored procedure indicates the end of an ETL job and should be the last stored
procedure executed in the ETL job. It is important to note that in addition to updating
thetblAdmin_Audit_Jobs table, this stored procedure also updates the tblAdmin_Audit_Step table
with the threshold information for each step. The threshold information is stored with each step in the
table because over time, the acceptable thresholds for the step may change. If the threshold
information is only stored in the master step table (a Type 1 dimension), any changes to the table
affect meta data generated for historical steps.
Therefore, storing the threshold with the step (a Type 2 dimension) allows us to maintain historical
execution records without affecting their integrity if the master step information is changed. For
example, if a step initially loads 1,000 rows but over time the number of rows increases to 1 million,
the acceptable threshold information for that step must be changed as well. If the threshold data is
stored only in the tblAdmin_Step_Master table and not stored with each record, the context of the
data will be lost, which can cause inaccuracies in reports built on the meta data information. For
simplicity, to illustrate the technique, the sample code does not maintain threshold information
automatically. In order to change the threshold information for a step, an administrator will need to
modify the master step record manually. However, it would be possible to automate this process.
Code Example 19.22
CREATE PROCEDURE usp_Admin_Audit_Job_End

@JobNumber int = 1, --The number of the job (from the master job table) being
executed
@Successful bit --A flag indicating if the job was successful
AS
--DECLARE variables
DECLARE @ErrorNumber int --the number of the SQL error generated
DECLARE @ErrorRowCount int --the number of rows in the unit of work affected by
error
DECLARE @Startdate smalldatetime --the datetime the load job started
DECLARE @EndDate smalldatetime --the datetime the load job ended
DECLARE @JobAuditID int --the # for the instance of the job
DECLARE @RowCount int --the number of rows affected by the job
--UPDATE the job record (end time) in the Audit table

BEGIN TRANSACTION
SET @EndDate = getdate() --set the end date for the batch
SET @JobAuditID = (SELECT MAX(JobAuditID) FROM tblAdmin_Audit_Jobs
where JobNumber = @JobNumber) --get the job number
SET @RowCount = (SELECT SUM(NumberRecords) --get the total job record count
FROM tblAdmin_Audit_Step WHERE JobAuditID = @JobAuditID)
UPDATE tblAdmin_Audit_Jobs --Update the Job record with the end time
SET EndDate = @EndDate,
NumberRecords = @RowCount,
Successful = 1
WHERE JobAuditID = @JobAuditID

UPDATE tblAdmin_Audit_Step --Update all steps for the job with the
SET MinRecords = T.MinThreshRecords, --threshold information for each step
MaxRecords = T.MaxThreshRecords,
MinTime = T.MinThreshTime,
MaxTime = T.MaxThreshTime,
TimeTarget = CASE
WHEN DATEDIFF(mi, A.StartDate, A.EndDate) BETWEEN T.MinThreshTime AND T.MaxThreshTime
THEN 'On'
WHEN DATEDIFF(mi, A.StartDate, A.EndDate)< T.MinThreshTime THEN 'Under'
WHEN DATEDIFF(mi, A.StartDate, A.EndDate) > T.MaxThreshTime THEN 'Over'
ELSE 'Unknown'
END,
RecordTarget = CASE
WHEN A.NumberRecords BETWEEN T.MinThreshRecords AND T.MaxThreshRecords THEN 'On'
WHEN A.NumberRecords < T.MinThreshRecords THEN 'Under'
WHEN A.NumberRecords > T.MaxThreshRecords THEN 'Over'
ELSE 'Unknown'
END
FROM tblAdmin_Step_Master T
RIGHT OUTER JOIN tblAdmin_Audit_Step A ON T.StepNumber = A.StepNumber
WHERE A.JobAuditID = @JobAuditID
BEGIN
GOTO Err_Handler
END
COMMIT TRANSACTION
RETURN (0)
Err_Handler:
exec usp_AdminError @@ProcID, 'none', @ErrorNumber, @ErrorRowCount
RETURN (1)
GO
Code Sample: Step Audit
The following stored procedures demonstrate one method of logging step records from within ETL
stored procedures. Notice that the @@ProcID is used to retrieve the object id of the executing stored
procedure. Also note that the values of @@error and @@rowcount are retrieved immediately after
the INSERT statement.
Code Example 19.23
ALTER PROCEDURE usp_Admin_Audit_Step
@StepNumber tinyint = 0, --the uniue number of the step
@Parameters varchar(50) = 'none', --any parameters used in the SP
@RecordCount int = 0, --the number of records modified by the step
@StartDate smalldatetime, --the date & time the step started
@EndDate smalldatetime --the date & time the step ended
AS
--DECLARE variables
DECLARE @ErrorNumber int
DECLARE @ErrorRowCount int
DECLARE @JobAuditID int
BEGIN TRANSACTION --INSERT the audit record into the tblAdmin_Audit_Step table
SET @JobAuditID = (SELECT MAX(JobAuditID) FROM tblAdmin_Audit_Jobs)
INSERT INTO tblAdmin_Audit_Step ( JobAuditID, StepNumber, Parameters, NumberRecords,

StartDate, EndDate, Username)
VALUES (@JobAuditID, @StepNumber, @Parameters, @RecordCount,
@StartDate, @EndDate, user_name())
SELECT @ErrorNumber = @@error, @ErrorRowCount = @@RowCount
BEGIN
GOTO Err_Handler
END
COMMIT TRANSACTION
RETURN (0)
Err_Handler:
exec usp_Admin_Log_Error @@ProcID, 1, 'none', @ErrorNumber, @ErrorRowCount
RETURN (1)
GO
The following stored procedure demonstrates the use of the auditing stored procedure detailed
above:
Code Example 19.24
CREATE PROCEDURE usp_AuditSample
AS
--DECLARE variables
DECLARE @ErrorNumber int

DECLARE @RecordCount int
DECLARE @StartDate smalldatetime
DECLARE @EndDate smalldatetime
BEGIN TRANSACTION
SET @StartDate = getdate() --get the datetime the step started
insert into tblTest
select * from tblTest
SELECT @ErrorNumber = @@error, @RecordCount = @@rowcount
If @ErrorNumber <> 0 --error handler

BEGIN
GOTO Err_Handler
END
SET @EndDate = getdate() --get the datetime the step finished
--log the audit record into the tblAdmin_Audit_Step table

exec usp_Admin_Audit_Step 1 , 'test from SP', @RecordCount, @StartDate, @EndDate
COMMIT TRANSACTION
RETURN (0)
Err_Handler:
exec usp_Admin_Log_Error @@ProcID, 'none', @ErrorNumber, @RecordCount
RETURN (1)
GO
Code Sample: Error Tracking
The following stored procedures demonstrate one possible method of logging errors in ETL stored
procedures. Notice that the stored procedure uses the OBJECT_NAME function to retrieve the name
of the object (table, view, stored procedure, and so on). This introduces a level of abstraction so that
the code is only useful for stored procedures.
Code Example 19.25
CREATE PROCEDURE usp_Admin_Log_Error
@ObjectID int,
@StepNumber int,
@Parameters varchar(50) = 'none',
@ErrorNumber int = 0,
@RecordCount int = 0
AS
--SET NoCount ON
SET NOCOUNT ON
--RETRIEVE the NAME of the object being audited

DECLARE @ObjectName varchar(50)
SET @ObjectName = OBJECT_NAME(@ObjectID)
--INSERT the audit record into the tblAdmin_Audit_Errors table

BEGIN TRANSACTION
insert into tblAdmin_Audit_Errors (Object, StepNumber, Parameters, ErrorNumber,
RecordCount, UserName, RecordDate)
values (@ObjectName, @StepNumber, @Parameters, @ErrorNumber, @RecordCount,
user_name(), getdate())
COMMIT TRANSACTION
GO
Once an error is generated and passed to the error logging stored procedure, it is logged to
thetblAdmin_Audit_Errors table. Notice that the @@ProcID is used to retrieve the objected of the
executing stored procedure. Also note that the values of @@error and @@rowcount are retrieved
immediately after the INSERT statement. With the exception of modifying the value of @Step, this
logic may be deployed with no other alterations to the code. The following stored procedure
demonstrates how to deploy the error logging method detailed above:
Code Example 19.26
CREATE PROCEDURE usp_ErrorSample
AS
--SET NoCount ON
SET NOCOUNT ON
--DECLARE Variables
DECLARE @ObjectName varchar(50)
DECLARE @ErrorNumber int, @RecordCount int
DECLARE @Step int
--INSERT the audit record into the Authors table

BEGIN TRANSACTION
insert into Authors
Select * from authors
SELECT @ErrorNumber = @@error, @RecordCount = @@rowcount, @Step = 2
BEGIN
GOTO Err_Handler
END
COMMIT TRANSACTION
RETURN (0)
Err_Handler:
exec usp_Admin_Log_Error @@ProcID, @Step, 'none', @ErrorNumber, @RecordCount
RETURN (1)
GO
CONCLUSION
The ETL system efficiently extracts data from its sources, transforms and sometimes aggregates data
to match the target data warehouse schema, and loads the transformed data into the data warehouse
database. A well-designed ETL system supports automated operation that informs operators of errors
with the appropriate level of warning. SQL Server 2000 Data Transformation 1048527521Services
can be used to manage the ETL operations, regardless of the techniques used to implement
individual ETL tasks.
While it is tempting to perform some transformation on data as it is extracted from the source system,
the best practice is to isolate transformations within the transformation modules. In general, the data
extraction code should be designed to minimize the impact on the source system databases.
In most applications, the key to efficient transformation is to use a SQL Server 2000 database for
staging. Once extracted data has been loaded into a staging database, the powerful SQL Server
2000 database engine is used to perform complex transformations.
The process of loading fact table data from the staging area into the target data warehouse should
use bulk load techniques. Dimension table data is usually small in volume, which makes bulk loading
less important for dimension table loading.
The ETL system is a primary source of meta data that can be used to track information about the
operation and performance of the data warehouse as well as the ETL processes.
Changing of LO Extractor in Production System

How to Retain Deltas When you change LO Extractor in Production System
Requirement may come up to add new fields to LO cockpit extractor which is up & running in production environment.
This means the extractor is delivering daily deltas from SAP R/3 to BW system .
Since this change is to be done in R/3 Production system, there is always a risk that daily deltas of LO cockpit extractor
would get disturbed. If the delta mechanism is disturbed (delta queue is broken) then there no another way than doing reinitialization for that extractor. However this re-init is not easy in terms of time & resource. Moreover no organization would
be willing to provide that much downtime for live reporting based on that extractor.
As all of us know that initialization of LO Extractor is critical, resource intensive & time consuming task. Pre-requisites to
perform fill setup tables are - we need to lock the users from transactional updates in R/3 system, Stop all batch jobs that
update the base tables of the extractor. Then we need to schedule the setup jobs with suitable date ranges/document
number ranges.
We also came across such scenario where there was a requirement to add 3 new fields to existing LO cockpit extractor
2LIS_12_VCITM. Initialization was done for this extractor 1 year back and the data volume was high.
We adopted step by step approach to minimize the risk of delta queue getting broken /disturbed. Hope this step by step
procedure will help all of us who have to work out similar scenarios.
Step by Step Procedure:1.Carry out changes in LO Cockpit extractor in SAP R/3 Dev system.
As per the requirement add new fields to Extractor.
These new fields might be present in standard supporting structures that you get when you execute "Maintain Data
source" for extractor in LBWE. If all required fields are present in supporting structure mentioned above then just add
these fields using arrow buttons provided and there is no need to write user exit code to populate these new fields.
However if these fields (or some of the required fields) are not present in supporting structures then you have to go for
append structure and user exit code. The coding in user exit is required to populate the newly added fields. You have to
write ABAP code in User exit under CMOD & in Include ZXRSAU01.
All above changes will ask you for transport request. Assign appropriate development class/Package and assign all these
objects into a transport request.
2.Carry out changes in BW Dev system for objects related to this change.
Carry out all necessary changes in BW Dev system for objects related to this change (Info source, transfer rules, ODS,
Info cubes, Queries & workbooks). Assign appropriate development class/Package and assign all these objects into a
transport request
3.Test the changes in QA system.
Test the new changes in SAP R/3 and BW QA systems. Make necessary changes (if needed) and include them in followup transports.
4.Stop V3 batch jobs for this extractor.
V3 batch jobs for this extractor are scheduled to run periodically (hourly/daily etc) Ask R/3 System Administrator to put on
hold/cancel this job schedule.
5.Lock out users, batch jobs on R/3 side & stop Process chain schedule on BW.
In order to avoid the changes in database tables for this extractor and hence possible risk of loss of data, ask R/3 System
Administrator to lock out the users. Also batch job schedule need to be put on hold /cancel.
Ask System Administrator to clear pending queues for this extractor (if any) in SMQ1/SMQ2. Also pending /error out v3
updates in SM58 should be processed.
On BW production system the process chain related to delta Info package for this extractor should be stopped or put on
hold.
6.Drain the delta queue to Zero for this extractor.
Execute the delta Info package from BW and load the data into ODS & Info cubes. Keep executing delta Info package till
you get 0 records with green light for the request on BW side. Also you should get 0 LUW entries in RSA7 for this
extractor on R/3 side.
7.Import R/3 transports into R/3 Production system.
In this step we import R/3 transport request related to this extractor. This will include user exit code also. Please ensure
that there is no syntax error in include ZXRSAU01 and it is active. Also ensure that objects such as append structure is
active after transport.
8.Replicate the data source in BW system.
On BW production system, replicate the extractor (data source).
9.Import BW transport into BW Production system.
In this step we import BW transport related to this change into BW Production system.
10.Run program to activate transfer rules
Execute program RS_TRANSTRU_ACTIVATE_ALL. Enter the Info source and source system name and execute. This
will make sure that transfer rules for this Info source are active
11.Execute V3 job Manually in R/3 side
Go to LBWE and click on Job Control for Application area related to this extractor (for 2LIS_12_VCITM it is application
12). Execute the job immediately and it should finish with no errors.
12.Execute delta Info package from BW system
Run delta Info package from BW system. Since there is no data update, this extraction request should be green with zero
records (successful delta extraction)
13.Restore the schedule on R/3 & BW systems

Ask System Administrator to resume V3 update job schedule, batch job schedule and unlock the users. On BW side,
restore the process chains schedule.
From next day onwards (or as per frequency), you should be able to receive the delta for this extractor with data also
populated for new fields.
Delta Mechanism In LO Extraction V3 Update

After examining the conception and technical background of delta extraction in Logistic Cockpit by using this method
(Episode one: V3 Update, the serializer), we also examined all peculiar restrictions and problems related to its usage
(Episode two: V3 Update, when some problems can occur...).
Since performance and data consistency are very critical issues for a datawarehouse and, from this point of view, the
serializer started showing its not so reliable face, with advent of PI 2002.1 (or PI-A 2002.1) three new update
methods came up and, as of PI 2003.1 (or PI-A 2003.1), these methods have been completely replaced Serialized
V3 update method, that its no longer offered.Fig.1: LBWE, Update Mode Selection Screen as PI.2002.1
Fig.2: LBWE, Update Mode Selection Screen as PI.2003.1
Now lets meet our new guests and try to discover when its better to make use of one than another one!
1. New "direct delta" update method: when in R/3, life is not so berserk...
With this update mode, extraction data is transferred directly to the BW delta queues with each document posting.
As a consequence, each document posted with delta extraction is converted to exactly one LUW in the related BW delta
queues.
Just to remember that LUW stands for Logical Unit of Work and it can be considered as an inseparable sequence of
database operations that ends with a database commit (or a roll-back if an error occurs).Fig.3: Direct Delta update
mechanism
Starting from the definition of this method we can see at once what are advantages and disadvantages resulting from
direct delta usage.
BENEFITS AND...
As we can see from the picture above, theres no need to schedule a job at regular intervals (through LBWE Job
control) in order to transfer the data to the BW delta queues; thus, additional monitoring of update data or extraction
queue is not require.
Logically, restrictions and problems described in relation to the "Serialized V3 update" and its collective run do not apply to
this method: by writing in the delta queue within the V1 update process, the serialization of documents is ensured by
using the enqueue concept for applications and, above all, extraction is independent of V2 update result.
...LIMITATIONS !
The number of LUWs per datasource in the BW delta queues increases significantly because different document changes
are not summarized into one LUW in the BW delta queues (as was previously for V3 update).
Therefore this update method is recommended only for customers with a low occurrence of documents (a maximum
of 10000 document changes - creating, changing or deleting - between two delta extractions) for the relevant application.
Otherwise, a larger number of LUWs can cause dumps during extraction process and, anyway, V1 update would be too
much heavily burdened by this process.
Besides, note that no documents can be posted during delta initialization procedure from the start of the
recompilation run in R/3 (setup tables filling job) until all records have been successfully updated in BW: every document
posted in the meantime is irrecoverably lost.
(Remember that stopping the posting of documents always applies to the entire client).
2. New "queued delta" update method: how to easily forget our old serializer...
With queued delta update mode, the extraction data (for the relevant application) is written in an extraction queue
(instead of in the update data as in V3) and can be transferred to the BW delta queues by an update collective run, as
previously executed during the V3 update.
After activating this method, up to 10000 document delta/changes to one LUW are cumulated per datasource in the BW
delta queues.Fig.4: Queued Delta update mechanism
If you use this method, it will be necessary to schedule a job to regularly transfer the data to the BW delta queues (by
means of so-called "update collective run") by using the same delivered reports as before (RMBWV3<Appl.No.>); instead,
report RSM13005 will not be provided any more since it only processes V3 update entries.
As always, the simplest way to perform scheduling is via the "Job control" function in LBWE.
SAP recommends to schedule this job hourly during normal operation after successful delta initialization, but there is no
fixed rule: it depends from peculiarity of every specific situation (business volume, reporting needs and so on).
BENEFITS, BUT...
When you need to perform a delta initialization in the OLTP, thanks to the logic of this method, the document
postings(relevant for the involved application) can be opened again as soon as the execution of the
recompilation run (or runs, if several and running in parallel) ends, that is when setup tables are filled, and a delta init
request is posted in BW, because the system is able to collect new document data during the delta init uploading too (with
a deeply felt recommendation: remember to avoid update collective run before all delta init requests have been
successfully updated in your BW!).
By writing in the extraction queue within the V1 update process (that is more burdened than by using V3),
theserialization is ensured by using the enqueue concept, but collective run clearly performs better than the
serialized V3 and especially slowing-down due to documents posted in multiple languages does not apply in this method.
On the contrary of direct delta, this process is especially recommended for customers with a high occurrence of
documents (more than 10,000 document changes - creation, change or deletion - performed each day for the application
in question.
In contrast to the V3 collective run (see OSS Note 409239 Automatically trigger BW loads upon end of V3 updates in
which this scenario is described), an event handling is possible here, because a definite end for the collective run is
identifiable: in fact, when the collective run for an application ends, an event (&MCEX_nn, where nn is the number of the
application) is automatically triggered and, thus, it can be used to start a subsequent job.
Besides, dont omit that queued delta process extraction is independent of success of the V2 update.
...REMEMBER TO EMPTY THE QUEUE !
The queued delta is a good friend, but some care is required to avoid any trouble.
First of all, if you want to take a look to the data of all extract structures queues in Logistic Cockpit, use
transactionLBWQ or "Log queue overview" function in LBWE (but here you can see only the queues currently
containing extraction data).
In the posting-free phase before a new init run in OLTP, you should always execute (as with the old V3) the update
collective run once to make sure to empty the extraction queue from any old delta records (especially if you are already
using the extractor) that, otherwise, can cause serious inconsistencies in your data.
Then, if you want to do some change (through LBWE or RSA6) to the extract structures of an application (for which
you selected this update method), you have to be absolutely sure that no data is in the extraction queue before executing
these changes in the affected systems (and especially before importing these changes in production environment !).
To perform a check when the V3 update is already in use, you can run in the target system the RMCSBWCC check report.
The extraction queues should never contain any data immediately before to:
perform an R/3 or a plug-in upgrade
import an R/3 or a plug-in support packages
3. New "unserialized V3 update" update method: when a right delta sequence is not a requirement...
With this update mode, that we can consider as the serializers brother, the extraction data continues to be written to the
update tables using a V3 update module and then is read and processed by a collective update run (through LBWE).
But, as the name of this method suggests, the V3 unserialized delta disowns the main characteristic of his brother: data is
read in the update collective run without taking the sequence into account and then transferred to the BW delta
queues.Fig.5: (Unserialized) V3 Delta update mechanism
Its pleonastic to say that all (performance) problems related to the serialized V3 update cant apply to the
unserialized one !
However, already known V2 dependency keep on subsist.
When this method can be used ?
Only if its irrelevant whether or not the extraction data is transferred to BW in exactly the same sequence (serialization) in
which the data was generated in R/3 (thanks to a specific design of data targets in BW and/or because functional data
flow doesnt require a correct temporal sequence).Fig.6: Comparison among call function hierarchies involved in different
update methods
Little recap of essential points to consider in migration process

If you want to select a new update method, you have to implement specific OSS notes, otherwise, even if you have
selected another update method, data will still be written to the V3 update and it can no longer be processed !
Here is an OSS collection related to update method switch divided for application:
PURCHASING (02) -> OSS note 500736
INVENTORY MANAGEMENT (03) -> OSS note 486784
PRODUCTION PLANNING AND CONTROL (04) -> OSS note 491382
AGENCY BUSINESS (45) -> OSS note 507357
If the new update method of an application is the queued delta, its better to have the latest qRFC version installed.
However, before changing, you must make sure that there are no pending V3 updates (as suggested before, run the
RMCSBWCC during a document posting free phase and switch the update method if this program doesnt return any
open V3 updates).
Early delta initialization in the logistics extraction and final considerations
Not always on our systems an important downtime is possible in the initialization process during the reconstruction and
the delta init request (just thinking when you need to ask a billing stop period...a real nightmare in some company !)
For this reason, as of PI 2002.1 and BW Release 3.0B, you can use the early delta initialization to perform the
initialization for selected datasources (just checking in infopackage update mode tab if this function is available): in this
way you can readmit document postings in the OLTP system as early as possible during the initialization procedure.
In fact, if an early delta initialization infopackage was started in BW, data may be written immediately to the delta queue.
But, if you are working with the queued delta method, using early delta initialization function doesnt make sense: as
described before, its the same method definition that permits to reduce downtime phase.
But leaving aside that, dont forget that, regardless of the update method selected, its ALWAYS necessary to stop any
document postings (for the relevant application) during setup tables recompilation run !
LO Extractor in Production system
How to retain deltas when you change LO extractor in Production

system
Requirement may come up to add new fields to LO cockpit extractor
which is up & running in production environment. This means the
extractor is delivering daily deltas from SAP R/3 to BW system .Since this
change is to be done in R/3 Production system, there is always a risk that
daily deltas of LO cockpit extractor would get disturbed. If the delta
mechanism is disturbed (delta queue is broken) then there no another way
than doing re-initialization for that extractor. However this re-init is not
easy in terms of time & resource. Moreover no organization would be
willing to provide that much downtime for live reporting based on that
extractor.
As all of us know that initialization of LO Extractor is critical, resource
intensive & time consuming task. Pre-requisites to perform fill setup
tables are - we need to lock the users from transactional updates in R/3
system, Stop all batch jobs that update the base tables of the extractor.
Then we need to schedule the setup jobs with suitable date
ranges/document number ranges.We also came across such scenario where
there was a requirement to add 3 new fields to existing LO cockpit
extractor 2LIS_12_VCITM.
Initialization was done for this extractor 1 year back and the data volume
was high.We adopted step by step approach to minimize the risk of delta
queue getting broken /disturbed. Hope this step by step procedure will
help all of us who have to work out similar scenarios.
Step by Step Procedure
1.Carry out changes in LO Cockpit extractor in SAP R/3 Dev system.As
per the requirement add new fields to Extractor.These new fields might be
present in standard supporting structures that you get when you execute
"Maintain Data source" for extractor in LBWE. If all required fields are
present in supporting structure mentioned above then just add these fields
using arrow buttons provided and there is no need to write user exit code
to populate these new fields.However if these fields (or some of the
required fields) are not present in supporting structures then you have to
go for append structure and user exit code.
The coding in user exit is required to populate the newly added fields.
You have to write ABAP code in User exit under CMOD & in Include
ZXRSAU01.All above changes will ask you for transport request. Assign
appropriate development class/Package and assign all these objects into a
transport request.
2.Carry out changes in BW Dev system for objects related to this
change.Carry out all necessary changes in BW Dev system for objects
related to this change (Info source, transfer rules, ODS, Info cubes,
Queries & workbooks). Assign appropriate development class/Package
and assign all these objects into a transport request
3.Test the changes in QA system.Test the new changes in SAP R/3 and
BW QA systems. Make necessary changes (if needed) and include them in
follow-up transports.
4.Stop V3 batch jobs for this extractor.V3 batch jobs for this extractor are
scheduled to run periodically (hourly/daily etc) Ask R/3 System
Administrator to put on hold/cancel this job schedule.
5.Lock out users, batch jobs on R/3 side & stop Process chain schedule on
BW.In order to avoid the changes in database tables for this extractor and
hence possible risk of loss of data, ask R/3 System Administrator to lock
out the users. Also batch job schedule need to be put on hold /cancel.Ask
System Administrator to clear pending queues for this extractor (if any) in
SMQ1/SMQ2. Also pending /error out v3 updates in SM58 should be
processed.On BW production system the process chain related to delta
Info package for this extractor should be stopped or put on hold.
6.Drain the delta queue to Zero for this extractor.Execute the delta Info
package from BW and load the data into ODS & Info cubes. Keep
executing delta Info package till you get 0 records with green light for the
request on BW side. Also you should get 0 LUW entries in RSA7 for this
extractor on R/3 side.
7.Import R/3 transports into R/3 Production system.In this step we import
R/3 transport request related to this extractor. This will include user exit
code also. Please ensure that there is no syntax error in include
ZXRSAU01 and it is active. Also ensure that objects such as append
structure is active after transport.
8.Replicate the data source in BW system.On BW production system,
replicate the extractor (data source).
9.Import BW transport into BW Production system.In this step we import
BW transport related to this change into BW Production system.
10.Run program to activate transfer rulesExecute
program RS_TRANSTRU_ACTIVATE_ALL. Enter the Info source and
source system name and execute. This will make sure that transfer rules
for this Info source are active
11.Execute V3 job Manually in R/3 sideGo to LBWE and click on Job
Control for Application area related to this extractor (for
2LIS_12_VCITM it is application
12. Execute the job immediately and it should finish with no
errors.12.Execute delta Info package from BW systemRun delta Info
package from BW system. Since there is no data update, this extraction
request should be green with zero records (successful delta extraction)
13.Restore the schedule on R/3 & BW systemsAsk System Administrator
to resume V3 update job schedule, batch job schedule and unlock the
users. On BW side, restore the process chains schedule.From next day
onwards (or as per frequency), you should be able to receive the delta for
this extractor with data also populated for new fields.
LO-COCKPIT Extraction
The first and the most important extraction method. LO-COCKPIT
Positioning of LO-COCKPIT:
What is Logistics Cockpit?

We can say that its a new technique to extract logistics information and consists of a series of a standard extract
structures (that is, from a more BW perspective, standard datasources), delivered in the business content.
Mind you that, the navigation part will no be discussed here but only the theory part which is required for understanding
LO extraction process. The navigation steps will be mentioned in brief.
Here we start...........
Data Extraction is the process of loading data from OLTP to OLAP (BW/BI). Here is an illustration...
I have a company, in which daily thousands of transactions happen all over the world. So to analyze my business on
yearly or monthly basis, i am moving to SAP BW/BI, so that i can generate reports and take business decisions.
Tomorrow i am going to load all the data which was captured till yesterday, from SAP R/3 to BW/BI. I do a full load for this.
After completing this task, i need to load the transactions that will happen from tomorrow to BW. This can be done either
daily or weekly or monthy based on the volume of transactions.
If there are 1000s of transactions per day, i can use daily load, 10000s - weekly, if in lakhs- monthly.
So, in precise, data has to be extracted in two modes:
1. Full load - Entire data which is available at source is loaded to BW/BI
2. Delta load - Only the new/changed/deleted data is loaded.
Full Load Data FLow:
Let us see, how the data is loaded to BW in Full mode.
Here we need to understand few basic things which happen on R/3 side.
Document Posting means creating a transaction, writing into the application/transaction tables.
So whenever sales order is created ( document posted), it transaction is written into the database tables/application
tables/transaction tables (Ex. EKPO, EKKO, VBAK, VBAP)
Whenever you are doing a full load, setup tables are used.
setup tables:
Access to application tables are not permitted, hence setup tables are there to collect the required data from the
application tables.
When a load fails, you can re-run the load to pull the data from setup tables. Data will be there in setup tables. Setup
tables are used to Initialize delta loads and for full load. Its part of LO Extraction scenario.
With this option, you avoid pulling from R/3 directly as we need to bring field values from multiple tables. You can see the
data in the setup tables.Setup table table name wiil be extract structure name followed by SETUP. Set up table names
starts with 'MC' followed by application component '01'/'02' etc and then last digits of the datasource name and then
followed by SETUP
Also we can say the communication structure (R/3 side,you can check it in LBWE also) name followed by 'setup'
example: MC13VD0HDRSETUP
If you want to check data in set up tables you better look at the transaction NPRT here you can see the table
name from which data is picking.
Setup tables are cluster tables and are used to extract the data from R/3 Tables.(LO Extractors)
Basically, for entire application like SD-Billing we have got it's own setup Tables...so while filling the set-up tables,
we usually fill for the entire application.
Ex: OLI7BW is for filling setup Tables for SD application.
OLI9BW T-code is for Billing Application,
When u fill the setup Tables, the data from different tables..VBAK, VBAP, VBRK, VBRP...etc will come through
communication Structures and saved in SetupTables...
The main advantage of having setup Tables is, we can read the data in different levels..Header level as well as
Item level.
when we run init load or Full load in BW, the data will be read from Setup Tables for the first time( Entire data will
be read).... and the delta records will be updated to Delta Queue once the v3 job runs... and we can extract the delta
records from Delta Queue.
once we succefully run the init, we can delete setup Tables.
Filling up the set up tables depends on the datasource.

There are different T-codes for the respective extract structures
OLIIBW transaction PM data
OLI1BW INVCO Stat. Setup: Material Movemts
OLI2BW INVCO Stat. Setup: Stor. Loc. Stocks
OLI3BW Reorg.PURCHIS BW Extract Structures
OLI4BW Reorg. PPIS Extract Structures
OLI7BW Reorg. of VIS Extr. Struct.: Order
OLI8BW Reorg. VIS Extr. Str.: Delivery
OLI9BW Reorg. VIS Extr. Str.: Invoices
OLIABW Setup: BW agency business
OLIFBW Reorg. Rep. Manuf. Extr. Structs
OLIIBW Reorg. of PM Info System for BW
OLIQBW QM Infosystem Reorganization for BW
OLISBW Reorg. of CS Info System for BW
OLIZBW INVCO Setup: Invoice Verification
OLI7BW is the tcode for Sales Order.
Steps for full load:
*************** This will be updated later - the navigation part*********************
1.
Delta Load:
As a prerequisite we need to discuss various update methods for delta load.
1.
2.
3.
4.
Serialized V3
Queued Delta
Direct Delta
Unserialized V3
Before that we need to understand V1, V2, V3 updates. These are different work processes on the application server that
makes the update LUW from the running program and execute it. These is separated to optimize the transaction
processing capabilities.
These are different work processes on the application server that makes the update LUW from the running program and
execute it. These is separated to optimize the transaction processing capabilities.
For Example :
If you create/change a purchase order (me21n/me22n), when you press 'SAVE' and see a success message (PO....
changed..), the update to underlying tables EKKO/EKPO has happened (before you saw the message). This update was
executed in the V1 work process.
There are some statistics collecting tables in the system which can capture data for reporting. For example, LIS table
S012 stores purchasing data (it is the same data as EKKO/EKPO stored redundantly, but in a different structure to
optimize reporting). Now, these tables are updated with the transaction you just posted, in a V2 process. Depending on
system load, this may happen a few seconds later (after you saw the success message). You can see V1/V2/V3 queues
in SM12 or SM13.
Statistical tables are for reporting on R/3 while update tables are for BW extraction. And is data stored redundantly in
these two (three if you include application tables) sets of table.
Difference is the fact that update tables are temporary, V3 jobs continually refreshes these tables (as I understand). This
is different from statistics tables which continue to add all the data. Update tables can be thought of as a staging place on
R/3 from where data is consolidated into packages and sent to the delta queue (by the V3 job).
Update tables can be bypassed (if you use 'direct' or 'queued' delta instead of V3) to send the updates (data) directly to
the BW queue (delta queue). V3 is however better for performance and so it is an option along with others and it uses
update tables.
Statistical table existed since pre BW era (for analytical reporting) and have continued and are in use when customers
want their reporting on R/3.
The structure of statistical table might be different from the update table/BW queue, so, even though it is based on same
data, these might be different subsets of the same superset.
V3 collective update means that the updates are going to be processed only when the V3 job has run.
At the time of oltp transaction, the update entry is made to the update table. Once you have posted the txn, it is available
in the update table and is waiting for the V3 job to run. When V3 job runs, it picks up these entries from update table and
pushes into delta queue from where BW extraction job extracts it.
Synchronous update (V1 update): Statistics update is carried out at the same time (synchronous) as the
document update (in the application tables).
Asynchronous update (V2 update): Document update and the statistics update take place in different tasks.
So, V1 and V2 updates dont require any scheduling activity.
Collective update (V3 update):As for the previous point (V2), document update is managed in a separate moment
from the statistics update one, but, unlike the V2 update, the V3 collective update must be scheduled as a job (via LBWE).
Remember that the V3 update only processes the update data that is successfully processed with the V2 update.
------------------------------------------------------------------------------------Serialized V3:
Take an example of the same PO item changing many times in quick succession.
V1 (with enqueue mechanism) ensures that the OLTP tables are updated consistently. Update table gets these update
records which may or may not end up in correct sequence (as there is no locking) when it reaches BW. 'Serialized V3' was
to ensure this correct sequence of update records going from update tables to delta queue (and then to BW).
Since update table records have the timestamp, when the V3 job runs, it can sequence these records correctly and thus
achieve 'serialization'.
The problems in Serialized V3 update are:
Several changes in one second: For technical reasons, collective run updates that are generated in the same
second cannot be serialized. That is, the serialized V3 update can only guarantee the correct sequence of extraction data
in a document if the document did not change twice in one second.
Different instances and times synchronization: I think its easy to verify how much it is probable that in a
landscape in which there are several application servers for the same environment different times can be displayed.The
time used for the sort order in our BW extractions is taken from the R/3 kernel which uses the operating system clock as a
time stamp. But, as experience teaches, in general, the clocks on different machines differ and are not exactly
synchronized.The conclusion is that the serialized V3 update can only ensure the correct sequence in the extraction of a
document if the times have been synchronized exactly on all system instances, so that the time of the update record
(determined from the locale time of the application server) is the same in sorting the update data.
The V2 update dependence: Not to be pitiless, but the serialized V3 update have also the fault of depending
from the V2 processing successful conclusion.Our method can actually only ensure that the extraction data of a document
is in the correct sequence (serialized) if no error occurs beforehand in the V2 update, since the V3 update only processes
update data for which the V2 update is successfully processed.Independently of the serialization, its clear that update
errors occurred in the V2 update of a transaction and which cannot be reposted, cause that the V3 updates for the
transaction that are still open can never be processed.This could thus lead to serious inconsistencies in the data in the
BW system.
Example:
Take a case where the first update (based on earliest timestamp) to be processed is in language EN (for same PO item).
V3 job is then going to process all the update records of language EN in chronological sequence before going to next
language records. If another language update (for same PO item) happened in between two EN language updates, this is
going to be processed later after all EN updates are processed and thus become out of sequence.
In the above figure, all the documents in red color (EN language) will be processed first and later blue colored (IT
language), which is an inconsistancy in sequence.
Direct Delta ( 2nd delta update method in our list)
With this update mode,

Each document posting is directly transferred into the BW delta queue
Each document posting with delta extraction leads to exactly one LUW in the respective BW delta queues
Just to remember that LUW stands for Logical Unit of Work and it can be considered as an inseparable sequence of
database operations that ends with a database commit (or a roll-back if an error occurs).
Benifits:
Theres no need to schedule a job at regular intervals (through LBWE Job control) in order to transfer the data to
the BW delta queues; thus, additional monitoring of update data or extraction queue is not required.
Logically, restrictions and problems described in relation to the "Serialized V3 update" and its collective run do not
apply to this method: by writing in the delta queue within the V1 update process, the serialization of documents is ensured
by using the enqueue concept for applications and, above all, extraction is independent of V2 update result.
Limits:
The number of LUWs per datasource in the BW delta queues increases significantly because different document
changes are not summarized into one LUW in the BW delta queues (as was previously for V3 update).Therefore this
update method is recommended only for customers with a low occurrence of documents (a maximum of 10000 document
changes - creating, changing or deleting - between two delta extractions) for the relevant application. Otherwise, a larger
number of LUWs can cause dumps during extraction process.
No documents can be posted during delta initialization procedure from the start of the recompilation run in R/3
(setup tables filling job) until all records have been successfully updated in BW: every document posted in the meantime is
irrecoverably lost.
V1 update would be too much heavily burdened by this process.
(Remember that stopping the posting of documents always applies to the entire client).
-----------------------------------------------------------------------------------------Queued Delta ( the third update method)

With queued delta update mode, the extraction data (for the relevant application) is written in an extraction queue (instead
of in the update data as in V3) and can be transferred to the BW delta queues by an update collective run, as previously
executed during the V3 update.
After activating this method, up to 10000 document delta/changes to one LUW are cumulated per datasource in the BW
delta queues.
If you use this method, it will be necessary to schedule a job to regularly transfer the data to the BW delta queues
As always, the simplest way to perform scheduling is via the "Job control" function in LBWE.
SAP recommends to schedule this job hourly during normal operation after successful delta initialization, but there is no
fixed rule: it depends from peculiarity of every specific situation (business volume, reporting needs and so on).
Benifits:
When you need to perform a delta initialization in the OLTP, thanks to the logic of this method, the document
postings (relevant for the involved application) can be opened again as soon as the execution of the recompilation run (or
runs, if several and running in parallel) ends, that is when setup tables are filled, and a delta init request is posted in BW,
because the system is able to collect new document data during the delta init uploading too (with a deeply felt
recommendation: remember to avoid update collective run before all delta init requests have been successfully updated in
your BW!).
By writing in the extraction queue within the V1 update process (that is more burdened than by using V3), the
serialization is ensured by using the enqueue concept, but collective run clearly performs better than the serialized V3 and
especially slowing-down due to documents posted in multiple languages does not apply in this method.
On the contrary of direct delta, this process is especially recommended for customers with a high occurrence of
documents (more than 10,000 document changes - creation, change or deletion - performed each day for the application
in question.
In contrast to the V3 collective run (see OSS Note 409239 Automatically trigger BW loads upon end of V3
updates in which this scenario is described), an event handling is possible here, because a definite end for the collective
run is identifiable: in fact, when the collective run for an application ends, an event (&MCEX_nn, where nn is the
number of the application) is automatically triggered and, thus, it can be used to start a subsequent job.
Extraction is independent of V2 update.

Limits:
V1 is more heavily burdened compared to V3.
Administrative overhead of extraction queue.

Note:
1.
if you want to take a look to the data of all extract structures queues in Logistic Cockpit, use transaction LBWQ or
"Log queue overview" function in LBWE (but here you can see only the queues currently containing extraction data).
2.
In the posting-free phase before a new init run in OLTP, you should always execute (as with the old V3) the
update collective run once to make sure to empty the extraction queue from any old delta records (especially if you are
already using the extractor) that, otherwise, can cause serious inconsistencies in your data.
3.
Then, if you want to do some change (through LBWE or RSA6) to the extract structures of an application (for
which you selected this update method), you have to be absolutely sure that no data is in the extraction queue before
executing these changes in the affected systems (and especially before importing these changes in production
environment !).
To perform a check when the V3 update is already in use, you can run in the target system the RMCSBWCC check report.
-------------------------------------------------------------------------------------Unserialized V3 : (The last one)

With this update mode, that we can consider as the serializers brother, the extraction data continues to be written to the
update tables using a V3 update module and then is read and processed by a collective update run (through LBWE).
But, as the name of this method suggests, the V3 unserialized delta disowns the main characteristic of his brother: data is
read in the update collective run without taking the sequence into account and then transferred to the BW delta queues.
Issues:
Only suitable for data target design for which correct sequence of changes is not important e.g. Material
Movements
V2 update has to be successful
When this method can be used ?
Only if its irrelevant whether or not the extraction data is transferred to BW in exactly the same sequence (serialization) in
which the data was generated in R/3 (thanks to a specific design of data targets in BW and/or because functional data
flow doesnt require a correct temporal sequence).
Here ends the update methods.

************************************************************
Some important points :
The setup tables are the base tables for the Datasource used for Full upload.So if you are going for only full
uploadfull update is possible in LO extractors.
Full update is possible in LO extractors.In the full update whatever data is present in the setup tables(from the last
done init) is sent to BW.
But setup tables do not receive the delta data from the deltas done after the init.So if ur full update should get ALL
data from the source system,u will need to delete and re-fill setup tables.
**************************************************************
Some Questions:
Question:
The serialized V3 update can guarantee the correct sequence for the extraction data of a document only if there were no
errors in the V2 update. This is because the V3 update only processes update data for which a V2 update has be carried
out successfully.
Why is V3 dependent on V2, what is V2 and V1 update?
Answer:
V1 - Synchronous update
V2 - Asynchronous update
V3 - Batch asynchronous update
These are different work processes on the application server that takes the update LUW (which may have various DB
manipulation SQLs) from the running program and execute it. These are separated to optimize transaction processing
capabilities.
Taking an example -
If you create/change a purchase order (me21n/me22n), when you press 'SAVE' and see a success message (PO....
changed..), the update to underlying tables EKKO/EKPO has happened (before you saw the message). This update was
executed in the V1 work process.
There are some statistics collecting tables in the system which can capture data for reporting. For example, LIS table
S012 stores purchasing data (it is the same data as EKKO/EKPO stored redundantly, but in a different structure to
optimize reporting). Now, these tables are updated with the txn you just posted, in a V2 process. Depending on system
load, this may happen a few seconds later (after you saw the success message). You can see V1/V2/V3 queues in SM12
or SM13.
V3 is specifically for BW extraction. The update LUW for these is sent to V3 but is not executed immediately. You have to
schedule a job (eg in LBWE definitions) to process these. This is again to optimize performance.
V2 and V3 are separated from V1 as these are not as realtime critical (updating statistical data). If all these updates were
put together in one LUW, system performance (concurrency, locking etc) would be impacted.
Serialized V3 update is called after V2 has happened (this is how the code running these updates is written) so if you
have both V2 and V3 updates from a txn, if V2 fails or is waiting, V3 will not happen yet.
BTW, 'serialized' V3 is discontinued now, in later releases of PI you will have only unserialized V3.
-------------------------------------------------------------------Question:
There are following tables
1. Application tables (R/3 tables)
2. Statistical tables (for reporting purpose)
3. update tables
4. BW queue
For Application tables its V1 update, statistical tables its V2 update and is it that the same information is again redundantly
stored in update tables?
How are statistical tables different from update tables? I mean i understood what statistical tables are, my question is "Is
the same information again redundantly stored in update tables for Collective V3 update to pull the records to BW Queue".
I mean is V3 collective update same as Synchronous V3 update? How does the records get saved in update tables?
Answer:
Statistical tables are for reporting on R/3 while update tables are for BW extraction. Is data stored redundantly in these
two (three if you include application tables) sets of table?, yes it is.
Difference is the fact that update tables are temporary, V3 jobs continually refresh these tables (as I understand). This is
different from statistics tables which continue to add all the data. Update tables can be thought of as a staging place on
R/3 from where data is consolidated into packages and sent to the delta queue (by the V3 job).
Update tables can be bypassed (if you use 'direct' or 'queued' delta instead of V3) to send the updates (data) directly to
the BW queue (delta queue). V3 is however better for performance and so it is an option alongwith others and it uses
update tables.
Statistical table existed since pre BW era (for analytical reporting) and have continued and are in use when customers
want their reporting on R/3.
The structure of statistical table might be different from the update table/BW queue, so, even though it is based on same
data, these might be different subsets of the same superset.
V3 collective update means that the updates are going to be processed only when the V3 job has run. I am not sure about
'synchronous V3'. Do you mean serialized V3?
At the time of oltp transaction, the update entry is made to the update table. Once you have posted the txn, it is available
in the update table and is waiting for the V3 job to run. When V3 job runs, it picks up these entries from update table and
pushes into delta queue from where BW extraction job extracts it.
-----------------------------------------------------------------------------------
Question:
what do you mean by serialization?is it the serialization beween sequence of records in update tables to the sequence in
BW Queue?
and
Can you explain little more about the Collective run performance with different languages.
Answer:
The requirement in 'delta' capturing on R/3 side is to be able to capture the delta 'exactly once in order'.
Take an example of the same PO item changing many times in quick succession.
V1 (with enqueue mechanism) ensures that the OLTP tables are updated consistently. Update table gets these update
records which may or may not end up in correct sequence (as there is no locking) when it reaches BW. 'Serialized V3' was
to ensure this correct sequence of update records going from update tables to delta queue (and then to BW).
Since update table records have the timestamp, when the V3 job runs, it can sequence these records correctly and thus
achieve 'serialization'. However, there is a technical problem with this. The timestamp recorded in update record is sent by
the application server (where user executed the txn) and if there are multiple app servers there might be some
inconsistency in their system time which can cause incorrect serialization.
Another problem is in the fundamental design of the V3 process. V3 Job sequences the updates on timestamp, and then
processes the update records from update table (to send it to delta queue), but it does so for one language at a time
(update record also has user logon language stored). Why this is done is not clear to me, but it is a basic design feature
and can not be subverted.
This causes a potential issue if multiple logon languages are used by users. Serialization may not happen correctly in
such a case. Take a case where the first update (based on earliest timestamp) to be processed is in language EN (for
same PO item). V3 job is then going to process all the update records of language EN in chronological sequence before
going to next language records. If another language update (for same PO item) happened in between two EN language
updates, this is going to be processed later after all EN updates are processed and thus become out of sequence. The
weblog mentions this scenario.
These two constraints remain for 'serialized V3' where 'serialization' couldn't be truly achieved. Hence newer PIs have
discarded 'serialized V3' altogether and now you do not have this option (if you are using a newer PI).
If you use 'serialized V3', you have to be clear that the 'serialization' may not always work in the above two scenarios
(multi language environment, and multiple app servers or updates to same records in same second(as timestamp has
granularity upto second level only)).
***************************************************************************
Now we will discuss about the functions of LO-COCKPIT:

Maintain Extract Strucutres: Here you can add additional fields from the communication structures available to the
extract structure.
Maintain Data Sources: In the Data source maintenance screen, you can customize the data source by using the
following fields: field name, short text, selection, hide field, inversion or cancellation field or reverse posting, and field only
known in customer exit.
Activating update: By Setting as active, data is written into extract structures both online as well as during
completion of setup tables or restructure table or LO initialization tables. Depending on the update mode a job has to be
scheduled with which the updated data is transferred in the background into the central delta management (Delta Queue).
Controlling update:This talks about the delta update mode you are using and how do you control the data load
based on the volume of data. LO Cockpit supports 4 types of update modes ( delta modes, which we have already
discussed):Serialized V3 update,Direct Delta,Queued Delta,Unserialized V3 update.
SAP BW - LO EXTRACTION MADE SIMPLE

LO EXTRACTION:
CONTENTS:
1) INTRODUCTION
2) WHY GOING EXTRACTION?
3) DIMENSIONS OF EXTRACTION PROCESS?
4) FREQUENTLY USED IMPORTANT TERMS IN LO:
5) EXTRACTION TYPES
6) LO Cockpit
7) TRANSACTION CODES IN USE FOR LO EXTRACTION
8) LO- EXTRACTION APPLICATIONS
9) DELTA EXTRACTION-ABR MECHANISM
10) UPDATES & UPDATE MODES AVAILABLE IN LO- EXTRACTION

A. DELTA UPDATES:
i. V1
ii. V2
iii. V3
B. DELTA UPDATE MODES:
i. DIRECT DELTA
ii. QUEUED DELTA
iii. UNSERIALIZED V3 UPDATE
11) FEATURES & DIFFERENCES BETWEEN V1 V2 & V3 UPDATES
12) LO-EXTRACTION (logistics data)
13) NAMING CONVENTIONS
a. NAMING CONVENTION FOR TRANSACTION DATA SOURCE
b. NAMING CONVENTION FOR EXTRACT STRUCTURE
c. NAMING CONVENTION FOR SETUP TABLES
d. TABLE REPRESENTATION:
NAMING CONVENTION FOR TRANSACTION DATA SOURCE, EXTRACT STRUCTURE & SETUP TABLES
Extraction:
Extraction? What is extraction?
Extraction itself means extracting data.
From where?
For what?
Why?
Just think! Not arduous. But not easier said than done!
I think you are familiar with modeling, which we can say that, its an art of designing. So what u has designed?
Answer: modeling of InfoCubes & D.S.O DataStoreObject or ODS which are well known as DataTargets.
By what? InfoObjects, InfoAreas, Idoc, PSA transfer methods, and also u know update rules, transfer rules. ohhhh. Its
gud. Gradually after building a data model i.e. InfoCube-just an ESS an extended Star Schema, just another step is how u
fillllllllllll the InfoCube?
Now u can get an answer, thats by extraction-extractor-extractor mechanism.
Am I right?
Yeah! Just its filling the InfoCube!
How u fill?
Much more about SAP interfaces!
SAP has an amazing feature i.e. it supports external data from third party ETL Tools. E.g. Ascential Software, ETI. And
Informatica through Staging BAPI to load data and meta data from many different databases and file formats. SAP
entered into strategic partnership with Ascential Software and bundling their DataStage ETL software package with SAP
BW.
Extraction programs read the data from EXTRACT STRUCTURES and send it to DATA STAGING MECHANISMS in a
required format.
Logistics is an important aspect in an enterprise and forming a core part of ERP application. For every application above
stated in table the applications such as Purchasing, Inventory Controlling, Shop Floor Controlling, Quality Management,
Invoice Verification, Shipment SD Sales BW, LE Shipping BW, SD Billing BW, Plant Maintenance BW, Customer Service
BW etc.
Day to day thousands of transactions occur form as a database volume. Important point to notice here is that, use of
Microsoft Excel is opted by small organizations, having low volume of data transactions or transaction data.
But when we have large number of transaction data updated daily, ERP software must be chosen. Thus we are choosing
SAP implementation and as we know it has its own pre delivered business content for all applications, and according to
the requirement we must install and change the business content.
After data modeling we go to Extraction and then reporting. So we must opt a different mechanism in extraction so called
extraction mechanism i.e. alternate method to reduce or decrease performance related issues and minimize the impact on
OLAP (Transaction System). Thus we choose LO-Extraction as an alternate to meet the real time analytic requirements.
The main conclusion here is that we need exact data for reporting, i.e. eliminate unwanted data or repeated data from
millions of records. I mean to say, Right data for right Reporting.
E.g. for more clarification, we will not recruit same person many times for one interview process. Here discussing about
data. Here person may be data, interview may be InfoCube, interviewer may be BW Delta Queue, and organization may
be BI or BW. Here, so the extracted data is analyzed in reporting making drill down across and so on etc in reporting.
SAP BW can extract and load data from a variety of source systems including Non-SAP source systems:
Data from non-SAP data sources can be loaded into SAP BW in one of three ways:
1. Flat file source systems (loaded through flat files)
2. External source systemsconnect to external systems with third-party custom extraction tools (using staging BAPIs).
3. External data providers such as Nielsen
SAP source systems:
1. SAP R/3 source systems (after release 3.1H)
2. SAP BW systemsusing Myself
3. Other SAP products
Methods of Data Load into SAP BW:

There are four methods to update the DataTargets (ODS or InfoCube) in SAP BW:
1. Update PSA, then data targets. This is the standard data transfer method. Data are first updated in the PSA and can be
subsequently updated.
2. Update PSA and data targets in parallel. This is a way of carrying out a high-performance update of data in the PSA
and one or more InfoCubes.
3. Update PSA only. You can store data in the PSA and bypass InfoCube or ODS objects. For example, if a high volume of
data needs to be extracted into SAP BW, the PSA can be loaded several times a day (e.g., every 15 minutes) and the
InfoCube and ODS can be loaded subsequently once a day (outside of business hours when there is limited on-line
activity and load on the BW server). This is one method to improve overall performance and manage high volumes of
data.
4. Update data targets only. You can load the InfoCubes and bypass the PSA. This is one way to reduce the overhead
associated with the PSA. The drawback of this method is that you lose the ability to do the following:
The preferred method for loading data into SAP BW is to use the PSA if possible.
2) What are the dimensions of extraction process?

Four dimensions are generally used to describe the different methods and properties of extraction processes:
Every extraction process can be viewed along these four dimensions.
1. Extraction mode.
2. Extraction scenario.
3. Data latency.
4. Extraction scope.
For further explanation see sap online guidance or Willeys Mastering the SAP Business Information Warehouse
3) FREQUENTLY USED IMPORTANT TERMS IN LO:

ABR MECHANISM : ABR Delta Mechanism is a PUSH DELTA, pushes the data from APPLICATION to the QUEUED
DELTA by means of V1 or V2 Update.
COMMUNICATION STRUCTURE : In the Communication Structure, data from an InfoSource is staged in the SAP (BW)
System. The Communication Structure displays the structure of the InfoSource. It contains all (logically grouped) of the
InfoObjects belonging to the InfoSource of the SAP (BW) System. Data is updated in the InfoCubes from the
Communication Structure. InfoCubes and Characteristics interact with InfoSources to get Source system data.
DATA SOURCE : The table ROOSOURCE has all details about the DataSource. You can give the input as your
DataSource name and get all relevant details about the DataSource
EXTRACTOR : Extractor enables the upload of business data from source systems into the data warehouse
EXTRACT RULES : Extract rules define how the data will be moved from extract structure to transfer structure.
EXTRACT STRUCTURE Extract Structure is a record layout of InfoObjects. In the Extract Structure, data from a
DataSource is staged in the SAP (R/3) SourceSystem. The Extract Structure contains the amount of fields that are offered
by an Extractor in the SAP (R/3) Source System for the data loading process. For flat files there is no Extract Structure
and Transfer structure.
EXTRACTION : Extraction is an alternate method for extracting data from the direct database (which is the result of n
number of transaction data updated daily through OLTP, resulting huge volume of data) mainly to meet OLAP
requirements.
SETUP TABLES : Lo uses the concept of Setup tables, from where the application tables data is collected. Setup tables
are used as data storage. Setup tables are used to Initialize delta loads and for full load.
TRANSFER RULES : Transfer rules transform data from several transfer structures from source system into a single
communication structure or InfoSource.
Data is transferred 1:1 from the Transfer Structure of the SAP (R/3) Source System into the Transfer Structure of the SAP
(BW) System, and is then transferred into the SAP (BW) System Communication Structure using the TransferRules. In the
transfer rules maintenance, you determine whether the communication structure is filled with fixed values from the transfer
structure fields, or using a local conversion routine.
TRANSFER STRUCTURE Transfer Structure maps data source fields to InfoSource InfoObjects.
The Transfer Structure is the structure in which the data is transported from the SAP (R/3) Source System into the SAP
(BW) System. In the Transfer Structure maintenance, you determine which Extract Structure fields are to be transferred to
the SAP (BW) System. When you activate the DataSource of the SAP (R/3) Source System from the SAP (BW) System,
an identical Transfer Structure like the one in the SAP (R/3) Source System is created in the SAP (BW) System.
UPDATE MODES : DELTA DIRECT DELTA, QUEUED DELTA, UNSERIALIZED V3 UPDATE, SERIALIZED UPDATE
UPDATE RULES Update rules transform data from communication structure in an InfoSource into one or more data
targets using chars, KF and time char
The update rules specify how the InfoObjects (Key Figures, Time Characteristics, and Characteristics) are updated in the
DataTargets from the Communication Structure of an InfoSource. You are therefore connecting an InfoSource with an
InfoCube or ODS object. The update rules assign InfoObjects in the InfoSources to InfoObjects in data targets. Update
rules are independent of the source system or data source and are specific to data targets.
Example: you can use update rules to globally change data independent of the source system.
UPDATES : V1- Synchronous, V2-Asynchronous, V3- Batch asynchronous updates
4) EXTRACTION TYPES:
SAP BW Extractors:
SAP BW extractors are the programs in SAP R/3 that enable the extraction and load of data from SAP R/3 to SAP BW.
SAP provides extractors that are programmed to extract data from most applications of SAP R/3 and delivered as part of
the business content. Extractors exist in release 3.1H and up.
There are three types of extractors in SAP BW:
1. SAP BW content extractors. For extraction of data from SAP R/3 application-specific data (e.g., FI, CO, HR, LO cockpit)
2. Customer generated, application specific (generic extractors). For extraction of cross-application data in SAP R/3 (e.g.,
LIS, FI/SL, CO/PA)
3. Generic extractors, cross application. For extraction of data from SAP R/3 across application database table/views and
SAP query.
5) LO COCKPIT-WHAT IS IT, WHAT IT DO & WHAT IT CONSIST OF?

The LO cockpit is a tool to manage the extract of logistics data from SAP R/3 and It consists of:
New standard extract structures
New DataSources
The functionality of the LO cockpit includes
Maintaining extract structure.
Maintaining DataSources.
Activating the update.
Controlling the V3 update.
6) TRANSACTION CODES IN USE FOR LO EXTRACTION:

TCODE DESCRIPTION
LBWE : LO DATA EXTRACTION: CUSTOMIZING COCKPIT
LBWF : BW LOG
LBWG : DELETION OF SETUP DATA MANUALLY
NPRT : LOG FOR SETUP OF STATISTICAL DATA
OLI*BW : INITIALIZATION OR SETUP (STATISTICAL)
LBWQ :
OLI8BW : DELIVERY
OLI9BW : BILLING
RSA2 : DATA SOURCE METADATA IN SAP ECC 6.0 9DATA SOURCE REPOSITORY)
RSA3 : TO CHECK ANY DATA THAT IS AVAILABLE RELATED TO YOUR DATA SOURCE
RSA5 : ACTIVATION OF BUSINESS CONTENT DATA SOURCE
RSA6 TO SEE THE ACTIVATED DATASOURCE
RSA7 : DELTA QUEUE ( DETA IMAGE & REPEAT DELTA)
SBIW : IMG FOR SAP BI EXTRACTION
SE11 : TO SEE APPLICATION TABLES STORED
SE37 CHECKING EXTRACTOR
SM13 : TO CHECK THE VARIOUS UPDATES THAT GETS PROCESSED DURING THE PROCESS OF UPDATING THE
SALES DOCUMENT CHANGE
SM37 : TO KNOW THE JOB STATUS
SMQ1 : CLEAR EXTRACTOR QUEUES
SMQ2 : TO CHECK THE VARIOUS UPDATES THAT GETS PROCESSED DURING THE PROCESS OF UPDATING THE
SALES DOCUMENT CHANGE
PROCESS KEYS
PROGRAM RMCVNEUA : SETUP THE INFORMATION FOR SALES DOCUMENTS
TMCLVBT : IT MAINTAINS THE PROCESS KEYS & THEIR DESCRIPTION
TMCLVBW : IT MAINTAINS THE PROCESS KEYS FOR EACH APPLICATION
7) LO- EXTRACTION APPLICATIONS:

As we know that SAP BW has its own business content i.e. SAP Pre Delivered Business Content. This is the touch stone
of SAP BW making pioneer in the ERP market. LO makes it as one stop for all i.e. The Transaction Data Source available
in LO COCKPIT. We can see this source data by entering Transaction Code: LBWE
LO- EXTRACTION APPLICATIONS

APPLICATION NUMBER : APPLICATION NAME APPLICATION NUMBER : APPLICATION NAME
02 : Purchasing 11 : SD Sales BW
03 : Inventory Controlling 12 : LE Shipping BW
04 : Shop Floor Controlling 13 : SD Billing BW
05 : Quality Management 17 : Plant Maintenance BW
06 : Invoice Verification 18 : Customer Service BW
08 : Shipment
8) DATA FLOW IN BW & BI:
Here we can see that BI-System extracts the data as a ONE TIME activity for the INIT DATA LOAD. After successful data
load, the SETUPTABLES/RESTRUCTING TABLES to be deleted to avoid redundant storage.
Important points here to know in SAP BW is
i. Installing Business Content (which contains SAP delivered Standard Data Sources comprising of data structures and
Extraction Programs. Actually SAP delivered Business Content is D Version, so we must activate the Business Content A
Version to use.
ii. Customizing
iii. Deploying
LO uses concept of SETUPTABLES to carry out the INITIAL DATA EXTRACTION process. The data Extractors for HR, FI
extracts data directly by accessing APPLICATION TABLES. I.e. LO Data Sources do not use application tables directly.
SETUPTABLES are also known as RESTRUCTURING TABLES, which are CLUSTER TABLES, which holds the
respective APPLICATION DATA. When we run init load or Full load in BW, the data will be read from Setup Tables for the
first time, total data will be read and the delta records will be updated to Delta Queue by V3 job runs and delta records can
be extracted from Delta Queue. After successful run of the init, setup Tables are deleted.
Important terms:
INIT/FULL LOAD: First Time (START)
DELTA: Only Changes (AFTER START)
9) DELTA EXTRACTION-ABR MECHANISM:

ABR MECHANISM:
The LO data sources support ABR Delta Mechanism (Push Delta Mechanism) which is compatible to ODS/DSO &
InfoCube. USE OF ABR DELTA:
The ABR Delta creates delta with:
AFTER: No MINUS symbol, updates records
BEFORE: MINUS SYMBOL, Delta Before Image
So what are pull data and Push data?
Data is pulled from delta queue (SourceSystem) and pushed into Data Ware House when a transaction is saved. SAP BW
system follows Pull Scenario & SAP BI system follows Push Data. The ABR Delta creates (-) symbol, Delta before Image
for the data which is deleted & After Image provides status after change with addition of records. The ABR delta
Mechanism uses V1 or V2 update where a delta is generated for a document change with addition of change (i.e.
document postings done), the program LO Updates the application tables for a transaction pushes/triggers the data. Data
extraction in BW is extracting data from various tables in the R/3 systems or BW systems. There are standard delta
extraction methods available for master data and transaction data. You can also build them with the help of transaction
codes provided by SAP. The standard delta extraction for master data is using change pointer tables in R/3. For
transaction data, delta extraction can be using LIS structures or LO cockpit. Delta Queue for an LO DataSource is
automatically generated after successful initialization and can be viewed in transaction RSA7 or in Transaction SMQ1
under the name MCEX.
10) UPDATES & UPDATE MODES AVAILABLE IN LO- EXTRACTION:

First when we are going to update methods, first we must know about updates, because updates are used in update
methods!
A. UPDATES:
i. V1 - Synchronous update
ii. V2 - Asynchronous update
iii. V3 - Batch asynchronous update
i. V1 - Synchronous update: V1 update is defined as direct delta is synchronous type uses time critical technology and
its nature of update is automatic, where the process done only once and never 2nd time.
It is used for single update work process & belongs to a same LUW (Logical Unit Of Work), reads data from documents
& uses direct delta & writes to application tables.
V1 update can be scheduled any time, with a mandatory step of locking users to prevent simultaneous updates.
V1 is carried out for critical/primary changes & these affects objects that has controlling functions in SAP System
During the process of creation of an order, the V1 update writes data into the application tables and the order gets
processed.
ii. V2 - Asynchronous update:
V2 update is defined as queued delta, is asynchronous statistical update type uses non-time critical (for updating
statistical tables of transaction tables) and its nature of update is never automatic, uses report RMBW311.
The process not done in a single update & carried out in separate LUW, reads data from documents and uses extractor
Queue & writes to application tables.
V2 update is scheduled hourly, with a mandatory step of locking users to prevent simultaneous updates.
A V2 is executed for less critical secondary changes & are pure statistical updates resulting from the transaction.
V1 updates must be processed before V2
iii. V3 - Batch asynchronous update: V3 is defined as Unserialized is collective update / synchronous with background
schedule or batch synchronous update, uses delta queue technology and the nature of update is never automatic, which
runs in background uses report RSM13005.
The process can be done at any time after V1, V2 updates. i.e. V3 uses V1 or V2 +v3 update, reads data from
documents & uses delta queue collective run call and writes to application tables.
V3 can be scheduled at any time, with a mandatory step of locking users to prevent simultaneous updates.
B. UPDATE MODES:
The LO DataSource implements its delta functionality using the above discussed V1, V2 & V3 update methods,
individually or by combination of them.
P1 2002.1 is the upgraded version. So what are the update modes that are available with LO DataSource as of P1
2002.1?
i. DIRECT DELTA
ii. QUEUED DELTA
iii. SERIALIZED V3 UPDATE
iv. UNSERIALIZED V3 UPDATE
i. DIRECT DELTA : V1 update:
Document postings and update sequence is 1:1. I.e. the direct delta V1 updates directly the document positions to the BW
Delta Queue and this is extracted to BI System periodically.
In doing so each document posting with delta extraction is posted for exactly one LUW in the respective BW delta
queues.
Users are locked but any postings are done so is completely lost. Because from start of recompilation run in OLTP until all
init requests have been successfully updated in BW.
Advantages:
SUITABILITY : Suitable for customers with fewer documents and no monitoring of update data/extraction queue required
SERIALIZATION BY DOCUMENT : Posting process ensures serialization of document by document, while writing delta to
Delta Queue within V1
EXTRACTION : Extraction is independent of V2 update
Disadvantages:
SUITABILITY : V1 is more heavily burden& Not suitable f high number of documents
Re-initialization process
(User Locks) : Setup and initialization required before document postings are resumed.
IMPLEMENTATION : less
ii. QUEUED DELTA V2 (V1+V3) update: **SAP RECOMMENDS**
Delta queues essentially are tables capturing the key values of changed or inserted records or the entire transaction
recordsData is pushed to extraction queue by means of V1 update and to delta queue same as V3 update. As we know
that V3 is a collective run the data collected in extraction queue and scheduled background.SAP recommends queued
delta for customers with high amount of documents.
Uses REPORT RMBWV311 for collective run and the naming convention is MCEX_UPDATE_ for example for sales its
MCEX_UPDATE_11s.
Advantages:
SUITABILITY : Suitable for customers with high amount of documents (SAP RECOMMENDS)
SERIALIZATION BY DOCUMENT : No serialization as its a statistical update, which is run after V1
EXTRACTION : Extraction is independent of V2 update
IMPLEMENTATION : **SAP RECOMMENDS**
Disadvantages : No disadvantages
iii. UNSERIALIZED V3 UPDATE

Data extracted from application tables is written to update tables using un-serialized V3 update mode. By using collective
update the so extracted data is processed. The Important point to consider here is that the data is read without a
sequence from update tables and are transferred to BW delta queue. Un serialized update, which is run after V1 & V2
doesnt guarantee serialisation of the document data posted to delta queue. This method is not suggestible as the entries
in the delta queue are not equal to the updates that made to application. This method V3 results erroneous data, when
data from DataSource is updated to DSO in overwrite mode, so the previous data is over written by the last update. But
this V3 is suggestible when updating data to ODS or InfoCube. V3 runs and process the data after V2 with which the data
is processed.
Advantages
SUITABILITY : Suitable for customers with high amount of documents
SERIALIZATION BY DOCUMENT : No serialization as its a statistical update, which is run after V1 & V2 but the un
serialised V3 doesnt guarantee serialisation of the document data.
EXTRACTION : collective
DOWN TIME : Not efficient
IMPLEMENTATION : LESS
Disadvantages
SUITABILITY : Not suggested if documents subjected to large changes with tracking changes
iv. SERIALIZED V3 UPDATE

Here the tables are updated consistently and the main problem in this method is, the user locks not there and the
document postings occur as we are extracting data. Its arduous as the data will not have consistency. E.g. document
changed twice or thrice when extracting data.
Hey: the important difference we to analyze here is!
Serialized V3 Vs Un-serialized V3
As the term serialisation itself mean, sequentially i.e. Data is read sequentially and transferred to the BW Delta Queue.
Un-serialised mean, the process is not in a sequence i.e. data is read in the update collective run without taking the
sequence into account and then transferred to the BW delta queues.
As per the data flow, serialized and un-serialized V3 updates, I can say that these are twins, with different process.
11) DATA FLOW CHARTS OF DIRECT DELTA, QUEUED DELTA, SERIALIZED V3 & UNSERIALIZED V3 UPDATES:
12) FEATURES , DIFFERENCES BETWEEN V1 V2 & V3 UPDATES: (COMPARISION TABLE):
13) NAMING CONVENTIONS:

NAMING CONVENTION FOR TRANSACTION DATA SOURCE
NAMING CONVENTION FOR SETUP TABLES
NAMING CONVENTION FOR EXTRACT STRUCTURE
TABLE REPRESENTATION:
PART 2:
14) PRACTICAL NOTES:
a) How to generate DataSource for LO
b) Migrating data from SAP R/3 to SAP (BW) using Direct Delta.
c) Data migration with Serialized V3 Update (Asynchronous with Background)
15) Sales and Distribution Tables : Commonly used SD Tables
16) KEY POINTS
Generic Extraction
GENERICEXTRACTION
It is mainly used for extracting the data form other Source Systems like R/3 etc.
DataSource:
Defines as extract structure and transfer structure from source system (like SAP R/3) and transfer structure from BW side.
ES -> What data to be extracted from source system

TS -> What data to be extracted from source system to BW. TS is a group of fields which indicated how the data is coming
from the source system
GENERIC EXTRACTION
1. Why do you go for Generic extraction?
- When the data source is not readily available in the Business content
- Even if the Business content data source is available and it is already being used up. And if we want to simulate
the same kind of extractor, we can go for generic extraction.
2. The T-code for creating Generic Extraction is RSO2.
3. We can create 3 types of data sources using generic extraction
- Transaction data
- Master data attribute
- Text
4. We can generate generic datasource using the following
- Table
- View
- Function module
- Infoset Query
STEPS TO CREATE GENERIC EXTRATION:

(MASTER DATA)
1. Go to T-Code SBIW otherwise go to T-Code RSO2.
-> Select the radio button Master data Attribute, give name and click on CREATE
-> Select the Application Component by pressing F4 (depending up on the table which we are extracted). It is just like
infoarea.
-> Give the Master data table name like MARA etc.
-> Give short, medium, long descriptions and click on SAVE.
-> It shows 4 options like

- SELECTION - > What ever the fields we select here those fields can appear at DATA SELECTION tab of
INFOPACKAGE.
- HIDE -> What ever the fields we select here those fields cant be appear in BW side.
- INVERSION -> This option is available for Key figures. It takes ve sign to the existing values.
- FIELD ONLY -> This option also available for Key figures. If there is any Enhancement for a particular field at that time
this is enabled.
So select whatever the fields we want as data selection and also to transfer rules otherwise hide other fields and click on
SAVE. Then it says, Data source has been saved successfully.
(Here I selected some fields like MATNR, ERSDA, MATKL,MEINS)
-> Then go to BW side -> Create all the Infoobjects that we want to extract to BW system.
-> Go to source system -> select R/3 to BW connection -> select your application component which we selected while
creating the generic datasource and click on Replicate datasource.
-> Select your datasource, which we are extracted and select Assign infosource
When we are seeing in the monitor it shows error. It shows the error message like the following.
Error message when processing in the Business Warehouse Diagnosis

An error ocurred in the SAP BW when processing the data. The error is Documented in an error message.
System response
The error message(s) was.weresent by:
Update Procedure
Chech the error message (pushbutton below the text).
Select the message in the message dialog box and select the long text for the further information
Follow the instructions in the message
When we click on ERROR MESSAGE it shows the following.
First see in the DETAILS tab in monitor then go to PSA and see in which places the error may occur.
Record 1: Value '00107 ' for characteristic YMATKL does not have the right internal format
Record 1: YMATKL: Data record 1 ('000000000000000038 '): Version '00107 ' is not valid
Record 4: Value '013 ' for characteristic YMATKL does not have the right internal format
Record 4: YMATKL: Data record 4 ('000000000000000068 '): Version '013 ' is not valid
LIKE THIS IT SHOWS.
-> Go and see in the PSA also it shows like this if we select only error records in PSA.
-> By seeing this error in PSA and error message in DETAILS tab, I think that the variable YMATKL must start with a letter.
So then go to the Transfer rule for YMATKL and write the ROUTINE.
-> Click on change to infosource -> select transfer rule for YMATKL -> select ROUTINE ->Click on CREATE -> write the
code and click on SAVE. The code is as follows.
DATA : NUM (20) TYPE C.
DATA : A TYPE C VALUE 'C'.
CONCATENATE A TRAN_STRUCTURE-MATKL INTO NUM.
RESULT = NUM.
And again click on ACTIVATE.
-> And again SCHEDULE the infopackage. Then see the MONITOR.
***** THIS IS THE PROCESS OF LOADING THE DATA USING FULL UPDATE *****
ALE DELTA
Now see loading data using DELTA UPLOAD. For this upload, if we change any data that changes will be effected in BW
while scheduling the DELTA update. For that also first we can INIT delta. Now see the procedure of ALE DELTA. First go
to MM02 (Change the material management) and do some changes.
-> Select the MATERIAL by pressing F4
-> Click on SELECT VIEWS and then select some views like BASIC DATA1 etc.
-> Modify the records from the fields, which we extracted already to BW side like BASIC UNIT OF MEASUREMENT,
Material Group etc
> After all modifications are done click on SAVE.
-> Go to RSO2 and select your data source for master data and click on CHANGE
---> Go to DATASOURCE menu tab and select ALE DELTA option.
-> Then it asks the TABLE NAME and CHANGE DOC OBJECT.
-> Here TABLE NAME is the table, which we extracted already.

-> And CHANGE DOC OBJECT is nothing but a field in that table. How can we know that is the field that we are giving in
the place of CHANGE DOC OBJECT? [Go to SE11 and give the table name as TCDOB and click on DISPLAY. Then click
on CONTENTS.
-> Whatever the object shows here that can be taken here as CHANGE DOC OBJECT.
-> Click on TRANSFER tab. Click on SAVE and again SAVE.
->Go to BW side and select the DATASOURCE and click on REPLICATE DATASOURCE again.
-> Go to Infosource and double click on your transfer rules and click on ACTIVATE transfer rules
-> Go to infopackage and SCHEDULE again by selecting the UPDATE as INIT DELTA without data transfers.
-> It shows 1 from 1 record after init delta. That means that one record is used for simulation means in order to enable
delta we can run the init delta option.
-> After completion of init delta successfully come back again and see now DELTA option is enabled. Select the DELTA
option and again SCHEDULE the infopackage.
***** THIS IS THE ALE DELTA FOR GENERIC EXTRATION FOR MASTER DATA *****
CO-PA EXTRACTION
Step 1: Log on to R/3
1. In the command area type /nSBIW. This takes you to the Dispay IMG screen.
2. The next step is to generate the DataSource. For this follow the path:
Business Information Warehouse ---> Settings for Application-Specific DataSources --->
Profitability Analysis ---> Create DataSource. Click on it to create a DataSource.
3. You will be taken to the CO-PA / SAP BW: DataSource for Transaction Data Screen.
DataSource : 1_CO_PA_800_GPK
Check CREATE
Operating Concern = BGIS
Check Costing-Based
4. Now click the EXECUTE button.
5. The next step is to select the Characteristics and the Key Figures. All the grayed once
are mandatory and are included in the COPA.
Short Text : COPA for 800_GPK
Medium Text : COPA for 800_GPK
Long Text : COPA for 800_GPK
Field Name for Partitioning : LAND1
6. Select the Calculated Key Figures from the Key Figure Scheme, like the once in the next
screen shot. Now click on the InfoCatalog icon available on the toolbar or F7.
7. Now you see a small window pops up.

Package : Z_TRAIN
8. After clicking the save button, another window pops up
Click the TICK mark.

9. The next screen is DataSource: Customer Version Edit. Here with respect to your
concern check the boxes.
10. Now in the tool bar, you see a icon to SAVE. So click the icon SAVE. A small window
pops up, stating
Click the YES BUTTON.
11. The next screen is InfoObject <--> Field Assignment. Check the selection fields of you project choice.
12. Click the SAVE button on the toolbar. Now click the BACK Button. You see a small
window pop up.
13. Now click on the Tick Mark.

14. Now click the icon GENERATE available on the toolbar.
15. Now click on the Back Button.
16. Click the back button, once again.
16. In the command field type /nRSA3. This is used to check the extraction.
DataSource = 1_CO_PA_800_GPK
Click the Extraction Button
Step 2: Log on to the BW Server
1. Now the DataSource has to be replicated. For this follow the path
Source Systems ---> EPC Client 800 ---> BW DataSources ---> SAP Application
Components ---> Controlling ---> Profitability Analysis
2. You dont find the DataSource 1_CO_PA_800_GPK, as the DataSource is not replicated.
So right click the mouse pointer and select the option Replicate DataSource
3. Now try to find the DataSource 1_CO_PA_800_GPK.

4. The next step is to Assign InfoSource. For this, select 1_CO_PA_800_GPK, right click
the mouse pointer and select the option Assign DataSources
5. A small window pops up.
6. Click the TICK mark.

7. Click the Yes Button
8. Now check the transfer rules and the communication structure.
9. Click on the Activate Button to activate the InfoSource.
10. Now you are supposed to create an InfoArea. Follow the path:
InfoProvider ---> InfoProvider
11. Right click the mouse pointer and select the option Create InfoArea
InfoArea = GPK_COPA
Description = InfoArea for 0CO_PA_800_GPK
12. Now place the mouse pointer on GPK_COPA InfoArea and select the option
Create InfoCube.
13. Click the create button.
14. Click on the InfoSource, select the InfoSource 1_CO_PA_800_GPK.

15. You see that all the Characteristics and Key Figures are mapped.
16. Now click on the ACTIVATE button to activate the InfoCube.

17. The next step is to create the Update Rules.
18. Click on the TICK Mark.

19. Now check that everything is mapped.
20. Click the Activate Button.
21. Now you need to Load the data into the InfoCube. For this you need to create an
InfoPackage
22. Follow the path:
InfoSource ---> Controlling ---> 1_CO_PA_800_GPK
Delta Mechanism In LO Extraction

For extracting logistic transactional data from R/3, a new generation of datasources and extractors, no longer based on
LIS (Logistic Information System) information structures, was developed starting from BW Release 2.0B and PI 2000.1 or
PI-A 2000.1 (valid from R/3-Release 4.0B ).
The tools for the logistics extract structures can be found in the IMG for BW (transaction SBIW): access into your OLTP
system, choose Customer-Defined DataSources -> Logistics -> Managing Extract Structures.
The Logistics Extract Structures Customizing Cockpit (you can directly see it by transaction LBWE) represents the
central tool for the administration of extract structures.
Ok...but, in other words, what is this the Logistic Cockpit (LC)?

We can say that its a new technique to extract logistics information and consists of a series of a standard extract
structures (that is, from a more BW perspective, standard datasources), delivered in the business content thanks to a
given plug-in.
But, what is the logic behind these datasources that allows to manage the logistic flows towards BW in delta mode after
an initial load of the historical data (done with update mode Delta Initialization, by retrieving data from the setup tables,
which are assigned to each extract structure, and are filled using special setup transactions in OLTP) ?
Following from many questions posted until now in SDN BW Forums, in this weblog we will focus only on the delta
mechanism of the LC and not on the other tasks we can manage inside it, like the necessary steps for activating and
carrying out successful data extraction or the maintenance of extract structures and datasources (but, dont worry, it will
arrive also a summary weblog dedicated to these important procedures in the next days !).
The V3 Update
Unlike the LIS update (well, I know that you are asking but how does this old LIS update work ???...my dear, another
weblog will arrive very soon also for this topic...sorry for the waiting, but I have a lot of things to do !), data is transferred
from the LIS communication structure, using extract structures (e.g. MC02M_0HDR for the header purchase documents),
into a central delta management area.
This transfer takes place thanks to the V3 update with a specific (scheduled) job and is therefore temporally detached
from the daily application operations; the main consideration is that the delta management acts as a buffer (not depending
from the application business) containing data that can be requested from BW via infopackage with update mode delta.
The following picture shows (with an high-level view) the interaction between the LIS communication structure and the V3
extraction technology.
e said that for updating the extraction of transactional data from the different logistics applications (MM, PP, SD and so
on), the technology for collective updates (V3 updates) is used (until PI 2003.1).
This means that the data is collected in the R/3 update tables before the transfer to the interface: the data is retrieved
there by means of a periodic update process that needs to be started in order to transfer delta records to the the BW
system delta queue.
During this V3 collective run (that you can start and schedule from LBWE for each application component), the data is
transferred to the BW delta queue (that you can see from RSA7 (see the picture below) or LBWQ transactions), from
which they are retrieved by means of (delta) requests from the BW system.
V1, V2, V3...

When scheduling what
Normally in R/3 there are three types of update available:
Synchronous update (V1 update)
Statistics update is carried out at the same time (synchronous) as the document update (in the application tables).
Asynchronous update (V2 update)
Document update and the statistics update take place in different tasks.
So, V1 and V2 updates dont require any scheduling activity.
Collective update (V3 update)
As for the previous point (V2), document update is managed in a separate moment from the statistics update one, but,
unlike the V2 update, the V3 collective update must be scheduled as a job (via LBWE).
Remember that the V3 update only processes the update data that is successfully processed with the V2 update.
This is a key task in order to properly manage the right working of the BW logistic flows.
In fact, scheduling timing process is very important and it should be based on the basis of 1) the amount of activities on a
particular OLTP system and on 2) the particular requirements related to the updating needs of data displayed in BW
reports.
For example (relating to the first point), a development system with a relatively low/medium of new/modified/deleted
documents may only need to run the V3 update on a weekly/daily basis.
Instead, a full production environment, with really many thousands of transactions everyday, may have to be updated
hourly, otherwise postings will queue and can affect performance heavily.
About the second point: if, for example, the reporting timing refers to a monthly periodic view, successfully monthly
scheduling the V3 update will ensure that all the necessary information structures are properly updated when new or
existing documents are processed in the meanwhile.
Finally, the right choice will be the result of all these considerations; by doing so, the information structures in BW will be
current and overall performance will be improved.
Its possible to verify that all V3 updates are successfully completed via transaction SM13.
SM13 transaction will take you to the Update Records: Main Menu screen:
On this screen, enter asterisk as your user (for all users), flag the radio button V2 executed, select a range date and hit
enter.
Any outstanding V3 updates will be listed.
At this point, its clear that, considering the V3 update mechanism, the main requirement is that the delta info have to
be transferred in the same sequence to the BW system as it occurred in the OLTP system.
Just a consideration...if we had to load our delta records only in a cube, there would be no problem: everything goes in
append and, in the end, well find the final situation right displayed thanks to the OLAP processor!
But since updating in ODS objects is permitted in the logistics extraction for almost all DataSources, we have to consider
any effects that can derive from the overwrite update mode (specific of the ODS object).
For example, the consistent storage of a status field (e.g. delivery status) in ODS objects can only be ensured only with a
right (serialized) delta sequence: if the record with open delivery status (created as first record in R/3) arrives later than
the record with closed delivery one (created as second one in R/3), we would have a false representation of the reality.
Considering that, the sequence of the existing data records is recognized by and taken into account when reading and
processing the update data (step A of the picture), as well as when transferring data to the BW system (step B).
Since the normal existing update methods actually does not recognize the serialized processing of update data, the
Serialized V3 Update function was created (also thanks to subsequent several corrections in SAP Basis) in order to be
able to serialize step A.
InfoCube Performance
Use
Here you can find the options that allow you to affect not only the
load performance, but also the query performance.
Features
Database Indexes
With an increasing number of data records in the InfoCube, not only
the load, but also the query performance can be reduced. This is
attributed to the increasing demands on the system for maintaining
indexes. The indexes that are created in the fact table for each
dimension allow you to easily find and select the data. When
initially loading data into the InfoCube, you should not create the
indexes at the same time as constructing the InfoCube, rather only
afterwards.
The indexes displayed are the secondary indexes of the F and E
fact tables for the InfoCube. The primary indexes and those defined
by the user are not displayed. The aggregates area deals with the
associated indexes of the fact table for all aggregates of an
InfoCube.
Checking Indexes
Using the Check Indexes button, you can check whether indexes
already exist and whether these existing indexes are of the correct
type (bitmap indexes).
Yellow status display: There are indexes of the wrong type
Red status display: No indexes exist, or one or more indexes are
faulty
You can also list missing indexes using transaction DB02,
pushbutton Missing Indexes. If a lot of indexes are missing, it can

be useful to run the ABAP reports SAP_UPDATE_DBDIFF and
SAP_INFOCUBE_INDEXES_REPAIR.
Deleting Indexes
For delta uploads with a large quantity of data (more than a million
records), you should not align the database indexes of the
InfoCube with every roll up, rather delete the indexes first, and then
completely reconstruct them after rolling up.
Repairing Indexes
Using this function you can create missing indexes or regenerate
deleted indexes. Faulty indexes are corrected.
Build Index
You can delete indexes before each data load and then rebuild
them again afterwards. You can also set this automatic index build
for delta uploads.
Aggregate Table Indexes
Just as for the indexes of the fact table, you can maintain the
indexes of the aggregation tables. Maintaining the indexes of the
aggregation tables affects all the aggregation tables of the
InfoCube. If the InfoCube contains more than 50 million records,
you should, as a rule, refrain from deleting and recreating indexes.
Database Statistics
The database statistics are used by the system to optimize the
query performance. You should keep the database statistics up-todate for this reason.
It is recommended that you always update the statistics if you have
loaded more than a million new records into the InfoCube since the
last update.
You can automatically recalculate the database statistics after each
load or after each delta upload.
Using the pushbutton Check Statistics, you can check the InfoCube
to see if statistics exist. If no statistics exist

yet, the status display changes to red. Using Recalculate Statistics,
you can add the missing InfoCube statistics in the background.
You can determine the percentage of InfoCube data that is used to
create the statistics. The percentage is set to 10% by default. The
larger the InfoCube, the smaller you should choose the percentage,
since the demand on the system for creating the statistics
increases with the change in size.
For up to 10 million entries in the InfoCube, you should set the
percentage of InfoCube data, that is used for creating the statistics,
to 100%.
Automatic deletion and building of indexes, as well as automatic
calculation of database statistics
and cause conflicts with automatic compression and rollup (see
Automatic Subsequent Processing ). These processes lock one
another. See Functional Constraints of Processes.
If you use process chains to guarantee serial processing of the
process, these settings are not
supported for the object. Then you have to include the automatic
processes as process types in the process chain.

Data Extraction

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Data Extraction

Transféré par

Droits d'auteur :

Formats disponibles

Data Extraction, Transformation, and Loading

Data Extraction, Transformation, and Loading Techniques

Most homogenous ETL architectures have the following characteristics:

Most heterogeneous ETL architectures have the following characteristics:

Map Source Data to Target Data

CREATE TABLE Authors_Staging (

[address] varchar (40) NULL DEFAULT ('Missing'),

CREATE TABLE [Authors_DW] (

[address] [varchar] (40) NULL DEFAULT ('Missing'),

--Populate the Authors_Staging table from Authors_Temp

--Populate the simulated data warehouse dimension table, Authors_DW

--Update the Authors_Staging table with the new values in Authors_Temp

--Update the Author_DW with the new data in Authors_Staging

--Insert the new records into the data warehouse Table

CREATE TABLE [Fact_Temp] (

--Create the example fact staging table

--Create the example data warehouse fact table

[ord_date] [datetime] NOT NULL ,

--Create the example titles dimension table

--Create the example stores dimension table

[city] [varchar] (20) NULL,

--Load the example dimension for Titles

--Load the example dimension for Stores

INSERT INTO Stores_DW

--Add some new rows to fact table to demonstrate technique

--Update the fact table. Use the maximum store key

--get the maximum store_key for each stor_id

--update the fact table

--drop the temporary table

CREATE TABLE [Stores_Temp] (

CREATE TABLE [Stores_Staging] (

CREATE TABLE [Stores_Current] (

[city] [varchar] (20) NULL,

CREATE TABLE [Stores_DW] (

--Load the Stores_Staging table with the default set of data

--Load the Stores_Current table with the default set of data

INSERT INTO Stores_Current (stor_id, stor_name, stor_address, city, state, zip,

CREATE TABLE [dbo].[tblAdmin_Audit_Jobs] (

[EndDate] [smalldatetime] NOT NULL DEFAULT ('01/01/1900'),

[JobNumber] [int] NOT NULL DEFAULT (1),

CREATE TABLE [dbo].[tblAdmin_Audit_Step] (

modified to track OLE DB errors as well.

insert into tblAdmin_Audit_Jobs (JobNumber, StartDate, EndDate, NumberRecords,

SELECT @ErrorNumber = @@error, @ErrorRowCount = @@rowcount

CREATE PROCEDURE usp_Admin_Audit_Job_End

SET NOCOUNT ON --SET NoCount ON

--UPDATE the job record (end time) in the Audit table

SELECT @ErrorNumber = @@error, @ErrorRowCount = @@rowcount

SET @JobAuditID = (SELECT MAX(JobAuditID) FROM tblAdmin_Audit_Jobs)

INSERT INTO tblAdmin_Audit_Step ( JobAuditID, StepNumber, Parameters, NumberRecords,

SELECT @ErrorNumber = @@error, @ErrorRowCount = @@RowCount

DECLARE @ErrorNumber int

SELECT @ErrorNumber = @@error, @RecordCount = @@rowcount

If @ErrorNumber <> 0 --error handler

SET @EndDate = getdate() --get the datetime the step finished

--log the audit record into the tblAdmin_Audit_Step table

--RETRIEVE the NAME of the object being audited

--INSERT the audit record into the tblAdmin_Audit_Errors table

--INSERT the audit record into the Authors table

SELECT @ErrorNumber = @@error, @RecordCount = @@rowcount, @Step = 2

Changing of LO Extractor in Production System

13.Restore the schedule on R/3 & BW systems

Delta Mechanism In LO Extraction V3 Update

Fig.2: LBWE, Update Mode Selection Screen as PI.2003.1