Performance Optimization in BODS

SAP Business Objects Data Services delivers a powerful, enterprise-class data integration and data
quality solution. It provides one development, run-time, and management environment that allows you
to integrate, transform, improve, and deliver data anywhere, at any frequency.
With SAP BusinessObjects Data Services, you can:
Respond to information needs and accelerate time to market
Deliver the most trusted and accurate information
Maximize operational efficiency with a single platform
Leverage a collaborative, complete, and open solution scalable for any project size
Features and Functions:
The comprehensive features and functions of SAP BusinessObjects Data Services can enable the
following:
Agility Leverage features that can accelerate time to market, encourage reuse and collaboration,
increase developer productivity, and create unified skill sets.
Trusted information Improve the reliability of your data with functions for data governance
including transparency, change management, security, and controls global data support, and end-to-
end metadata management.
Operational excellence Gain a single data repository, platform, and environment so that costs
associated with supporting multiple applications can be reduced; Leverage multi-platform support for
64bit, UNIX, Linux, and Windows.
Enterprise class Benefit from highly scalable performance with support for parallelization and grid
computing, enterprise connectivity, and support for service-oriented architecture (SOA).
It is recommended that the BODS jobs should be tuned accordingly to be able to complete in less time and utilize
less resource.
This document deals with the different Performance Optimization techniques needed to fine tune the BODS jobs
there by enabling the BODS jobs to complete in less time with less rates of failure.
2 BODS Performance Optimization
BODS is an ETL (Extraction, Transformation and Load) tool that selects data from one data base,
processes it in a job server by applying the transformation rules specified in the mapping and loads the
data into the specified target database.
There are 7 touch points that need to be considered for performance optimization. They are given
below:
1. Source Operating System.
2. Source Data Base.
3. Target Operating System.
4. Target Database.
5. Network.
6. Job Server OS.
7. BODS Repository database.


The BODS Job Server and the BODS Repository Database can reside on the same server. If installed on
different servers, Network throughput configured between Job Server and BODS Repository Database
will play a key role. The performance tuning methods for each identified touch point are given below.
2.1.1 Source Operating System
The Source Operating System should be tuned to quickly read data from disks. This can be done by the
following methods:
Set the read-ahead protocol to 64 KB to make the I/O operations fast.
The size of read-ahead protocol is by default set to 4-8 KB.
2.1.2 Source Database
The source database should be tuned to make SELECTs as quickly as possible. This can be done by the
following methods:
Increase the size of Database I/O to match with the OS read-ahead protocol; otherwise it may
cause bottle neck there by affecting the performance.
Increase size of shared buffer to cache more data in the database server.
Cache tables that are small enough to fit in the shared buffer by enabling cache at table level.
Caching reduces the number of I/Os thereby improving the speed of access to the data.
Turn off PARALLEL for small tables.
Create indexes on appropriate columns by looking into the dataflow to identify the columns
used in the join criteria.
Create Bitmap indexes on the columns with low cardinality as the index does not take much
space in the memory, there by resulting in faster SELECTs.
2.1.3 Target Operating System
The Target Operating System should be tuned to quickly write data to disks. This can be done by the
following methods:
Turn on the asynchronous I/O to make the Input/output operations as fast as possible.
2.1.4 Target Database
The Target Database should be tuned to perform INSERTs and UPDATEs as quickly as possible. This can
be done by the following methods:
Turn off archive logging.
Turn off redo logging for all tables.
Tune rollback segments for better performance.
Place redo log files and data files on a raw file if possible.
Increase the size of the shared buffer.
2.1.5 Network
Even if the Source Database and Target Database are tuned if the network band width is small, there is
possibility for the occurrence of bottle neck which hampers the performance.
Adjust the size of the Network buffer in the database client so that each client request
completely fills a small number of network packets, there by reducing the number of round trips across
databases which in turn improves the performance.
2.1.6 Job Server OS
Data Services jobs are multi-threaded applications. Typically a single data flow in a job initiates one
al_engine process that in turn initiates four threads. For maximum performance benefits:
Consider a design that runs one al_engine process per CPU at a time.
Tune the Job Server OS so that the Data Services threads spread to all
available CPUs.

The above performance optimization methods need to be implemented during the environment and
infrastructure preparation of BODS components.
2.1.7 Data Services Jobs
The following execution options can be used to improve the performance of BODS jobs:
Monitor Sample Rate: If the job processes large amount of data set the Monitor Sample Rate
to a higher value (maximum being 50,000, default is 1000) to reduce the number of I/O calls to the log
file there by improving the performance.
If virus scanner is configured on the BODS JobServer, exclude the Data Services log from the
virus scan. Otherwise the virus scan scans the Data Services log repeatedly during the execution, which
causes performance degradation.
Collect Statistics for self-tuning: BODS has a self-tuning capability to determine the cache
type by looking into the statistics of previous job executions. The Collect Statistics option needs to be
selected during the first execution of the job. BODS collects the statistics for that job and stores in the
metadata for that job. In the next execution select the Use Collected statistics option to allow BODS
to decide the type of cache to be used to execute the job, there by improving the performance of the
job.
Set the data flow properties like Degree of Parallelism depending upon the number of CPUs
available for processing and set the cache type to in-memory if the data volume is less.
If source tables are from same schema and using same Data Store, identify the joins in the
early phase so that the join can be pushed to the database.
Create synonyms for the tables in other schemas to pushdown join conditions of tables
belonging to different schemas in a database.
Use data base links to connect different schemas in different databases to pushdown the joins
of tables available in these schemas to database.
Use data transfer (type = TABLE) to pushdown the complex logic in a dataflow to the
database.
Do not use advanced features like Run Distinct as Separate process etc available in
Advanced tab in Query transform, as it starts multiple sub-processes which causes heavy traffic
between the processes and there by lead to the termination of job.
Do not use Data Transfer transform unless required. (Use table type if required as it is more
reliable.). SAP suggests that Data Transform is not a reliable transform and hence recommends to not
using it unless required.
Turn off the Cache option for the tables with larger amounts of data. Cache is turned on by
default for every source table. Make sure that there are indexes created on key columns for these
tables on which cache is turned off.
Do not use BODS functions like job_name(), instead initialize a variable in a script and use that
variable for mapping in query transforms.
Use Join where ever applicable in place of look up transform as the look up transform has to
access the table for each and every record which increases the number of I/O operations to database.
Use Query transforms to split the data in place of Case transforms as it is a costly transform in
BODS.


SAP Business Objects Data Services Target Table
Options:
created by Ankit Sharma on Sep 18, 2013 11:37 AM, last modified by Ankit Sharma on Sep 23, 2013 12:48 PM
Version 3

This will help you to understand how to make use of the different options available in the target table
while loading data.

Below is the descriptions of all the options with the sample data.

Auto Correct Load:

Auto correct loading ensures that the same row is not duplicated in a target table

Input Data Screen Short:

Target Option Window: Following steps need to be done


Uncheck the Delete sata from Table before Loading

Before Updating in Update Control:

After Updating:


Now go to Source Data and add two Duplicate rows
Before Updating:

After Updating:


After Creation of Jobs in Data Services Designer



Now Apply Auto Correct Load on Target Tables



Note: After apply Auto Correct Load Duplicate values are removed from Target Tables
Now Execute the Job


Output:



Ignore Column Option:


If a matching row exists, the row is updated depending on the values of Ignore columns with value,
and Ignore columns with null:

Ignore Columns Options are Worked with Auto Correct Load:

When the Ignore columns with null option is set to Yes and the column data from the source is
NULL, then the corresponding column in the target table is not updated. Otherwise, the corresponding
target column is updated as NULL since the source column is NULL.



Now Go To Target Table Properties:

Before Update in the Target Table Properties:



After Update in the Target Table Properties:



Now Source Data Information:

Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID
as a Primary key


View of Source table Before Loading:



View of Source table After Null Values Before Loading:


Now update in the Target Table before loading the data



Then Finally Execute the Job



Output after executing the job:



Ignore Column with Value Option:

If a matching row exists, the row is updated depending on the values of Ignore columns with value,
and Ignore columns with null:

When the column data from the source matches the value in Ignore columns with value, the
corresponding column in the target table is not updated. The value may be spaces. Otherwise, the
corresponding column in the target is updated with the source data.


Now Go To Target Table Properties:

Before Update in the Target Table Properties:

Before Update in the Target Table Properties:



After Update in the Target Table Properties:


Now Source Data Information:


Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID
as a Primary key



View of Source table Before Loading:



Now update in the Target Table before loading the data



Then Finally Execute the Job


Output after executing the job:


Update Key Columns in Update Control:
Now Go To Target Table Properties:

Before Update in the Target Table Properties:




After Update in the Target Table Properties:



Now Source Data Information:
Before Update:



After Update in Source Table:



Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID
as a Primary key


View of Source table Before Loading:


Now update in the Target Table before loading the data


Then Finally Execute the Job


Output after executing the job:













Sending IDOC from BODS to SAP


Document Purpose
This document provides steps required to load data in SAP using one of the SAP interfaces. This document will be
used at the time of proof of concept to demonstrate the ability of Business Objects Data Services (BODS) to invoke
IDOC interface to load data in SAP.

The main objectives are:
Loading data in SAP using IDOC
Demonstrate how BODS can be used to load data using IDOC
Display required configurations step-by-step & invocation
Demonstrate actual sample load using BODS job

Overview:
This document provides systematic approach for the data load in three sections:
1. Configuration Required at SAP Side:
2. Configuration Required at BODS Side:
3. Demonstration Sample Load using BODS Job:
Each of above sections provide information as in how to configure the specific parameters required to get both SAP &
BODS ready to communicate with each other. Screen shots provided for better understanding of the parameters &
their configuration.


Configuration Required On SAP (ECC 6.0) Side

Following is the required configuration on SAP side.
1.Display IMG [ /nsale ]
Connect to SAP System using SAP GUI
Use Transaction Code (TCODE) /nsale to Display IMG




Screenshot 1: Display IMG screen
Expand Logical Systems tree & click on the Clock icon in front of Define Logical System

2. Defining Logical System


Screenshot 2: Defining Logical System screen
Click on the New Entry Button


For SAP Inbound IDOC One Logical system (FRMDICL100) is required
Create a Logical system as shown below


Screenshot 3: Overview of added entries

Click on the save button to save both the Logical System.

Verify that the LS FRMDICL100 is present in the SAP system



Screenshot 4: Overview of Logical Systems

3. Create Partner Profile [ /nwe20 ]
Use TCODE /nwe20 to configure partner profile.



Screenshot 05: Partner Profile screen
Click on Create button to create new partner profile
Create partner profile as shown below

Note :
1. Provide partner no. FRMDICL100 i.e same as BODS Logical System.
2. Agent 50010120 EDI Department is necessary for communication with non-SAP systems like BODS. Without this
agent IDOC communication will not be successful.


Screenshot 06: Partner Profile deatil screen

Click on the Save button to save the partner profile
After saving; the Create Inbound parameter will be activated in Inbound paramtrs


Screenshot 07: Partner Profile create Inbound paramtrs screen

Click on Create Inbound parameter Green Plus button in Inbound paramtrs section




Screenshot 08: Partner Profile Inbound paramtrs screen
Note :
In Inbound options tab, select appropriate process code relevant to message type.
(In this case; for MATMAS message type, process code is MATM)



Screenshot 09

In "Post processing : permitted agent" tab provide details as shown above.
Save "Partner profile: Inbound parameters" using Save button
After saving verify Partner Profile & Inbound Parameter added.


Screenshot 10

Configuration Required On BODS Side

Following is the required configuration on BODS side
1. Create SAP Datastores

Create SAP Datastore by providing required SAP Application Server Information & creadentials


Screenshot 11: SAP Datastore in BODS


Right click on IDOCs and select Import By Name


Screenshot 12: IDOC list under SAP datastore in BODS


Screenshot 13: Import IDOC screen

Provide Type and Name as shown above and click on Import to import IDOC in datastore

2. Generating input XML schema
To create the input XML structure with the correct datatypes quickly drop the IDOC as message source onto the
dataflow, go into this object and rightclick the schema. There, you can ask the Designer to generate the entire
structure as XML schema, thus you get all columns, schemas including the datatypes.




Screenshot 14: MATMAS01 IDOC screen


MATMAS01_OUT this XML file source is created using generated schema inFile Format tab.


Screenshot 15: XML Schema screen

3. Build Batch Job
Build a Batch Job with a DataFlow inside
DataFlow is as Follows




Screenshot 16: Batch Job screen
Double click on source XML File Source configure it as shown below.
Note: For demonstration purpose we have used XML file that was output of the Real-Time Job designed in article
"Receiving IDOC In BODS from SAP".


Important : Make sure that in XML file Sending Partner setting i.e. <SNDPRN> should be same as BODS logical
system name. (FRMIDCL100) as shown below.


Screenshot 18: IDOC out XML file strucutre screen

Unnest by right clicking on MATMAS01_OUT schema in "Query" as shown below


Screenshot 19: IDOC structre screen
Drag the IDOC MATMAS01 from SAP DataStore as shown


Screenshot 20: IDOC in SAP Datastore
Note: Make sure to select option "Make IDOC Message Target"
Configure IDOC Message Target

Screenshot 21: BODS side IDOC configuration



Receiving IDOC in Business Objects Data Services
(BODS) from SAP
created by ROSHAN CHHAJED on Jul 16, 2012 10:51 AM, last modified by ROSHAN CHHAJED on Jul 16, 2012
5:43 PM
Version 1

Document Purpose

This document provides steps required to load data in BODS from SAP using IDOC interface. This document will be
used at the time of proof of concept to demonstrate the ability of Business Objects Data Services (BODS) Real-Time
job to receive data using IDOC interface from SAP.

The main objectives are:
Loading data in BODS from SAP using IDOC
Demonstrate how BODS can be used to receive data using IDOC
Display required configurations step-by-step & invocation
Demonstrate actual sample load using BODS Real-Time job


Overview:

This document provides systematic approach for the data load in three sections:
1. Configuration Required at SAP Side:
2. Configuration Required at BODS Side:
3. Demonstration Sample Load using BODS Job:

Each of above sections provide information as in how to configure the specific parameters required to get both SAP &
BODS ready to communicate with each other. Screen shots provided for better understanding of the parameters &
their configuration.



SAP [ECC 6.0] Configuration for Outbound IDOC Communication

1. Display IMG [/nsale]

Connect to SAP System using SAP [ECC 6.0] GUI
Use Transaction Code (TCODE) /nsale to Display IMG


1.0 Display IMG


2. Defining Logical System

Expand Logical Systems tree & click on the Clock icon in front of Define Logical Systems

2.0 Logical Systems: overview

Click on the New Entries Button
For SAP Outbound IDOC, two Logical Systems are required
First Logical System for SAP (ID4CLNT100)
Second Logical System for BODS (ID100CBODS)

Create these two Logical System as shown below


2.1 New Entries for Logical System

Click on the "SAVE" Button to save both the Logical Systems.
Confirmation message Data was Saved will be displayed as shown below.


2.2 Data Saved Conformation


Verify two (2) Logical Systems are present in the SAP


2.3 Verifying Logical System


3. Configure RFC Connection [/nsm59]

Use TCODE "/nsm59" to configure RFC Connection


3.0 Configurations of RFC Connection


Select TCP/IP Connection as shown above & click on the Create Button to configure new RFC Connection.


3.1 Creation of New RFC Destination: Technical Settings

Provide required information as shown above

Note: We are using registered Program ID as DI_RFC_Client. This could be any simple name, but the same name
should be used while configuring RFC Clients using BODS Administrator. (Ref: 2.Client Interfaces, page no 21)


Click on the MDMP & Unicode tab


3.2 Creation of New RFC Destination: MDMP & Unicode

Note: We have selected Unicode radio button because BODS Target RFC Client is Unicode.
Important: There are two buttons Connection Test and Unicode Test we will test RFC Connection after
Configuring RFC Client using BODS Administrator in Steps provided later in this document. (Ref: 3. Testing RFC
Client from SAP, page no 23)


4. Create Transactional RFC Port [/nwe21]
Use TCODE "/nwe21" to configure the RFC Port
Expand Ports tree to go to Transactional RFC


4.0 Creation of New Transactional RFC Port

Select Transactional RFC and click on Create Button.



4.1 Transactional RFC Port Name

Provide Name "DI_PORT" as shown above.
Click on Continue button

Port Name "DI_PORT" will appear in the list as shown below.



4.2 Transactional RFC Ports Overview

Click on the DI_PORT to provide Description & RFC destination (DI_RFC_DESTINATION ) that we have created in
earlier section (Ref: 3.Configure RFC Connection, page no 7)

5. Create Partner Profile [/nwe20]

Use TCODE "/nwe20" to configure Partner Profile.


5.0 Creation of New Partner Profile



Click on Create button to create new partner profile
Create partner profile as shown below
Note :
1. Provide partner no. ID100CBODS i.e same as BODS Logical System.
2. Agent 50010120 EDI Department is necessary to communicate with non-SAP systems like
BODS. Without this agent IDOC communication will not be successful.



5.1 Partner Profile Permitted Agent

Click on the Save button to save the partner profile
After saving the "Create outbound parameter" will be activated in Outbound paramtrs


5.2 Create outbound paramtrs.

Click on Create outbound parameter button in Outbound paramtrs section.



5.3 Partner Profile: Outbound parameters

Provide the Message Type "MATMAS" & IDoc Basic type "MATMAS01"
Save "Partner profile: Outbound parameters" using Save button
Confirm Partner Profile & Outbound Parameters added.


5.4 Outbound Parameters Overview


6. Change Distribution Model [/nbd64]

Use TCODE "/nbd64" to configure Change Distribution Model


Screen Shot 6.0 Change Distribution Model

Click on the Create Model View
Note: If this button is inactive then select the Model View in left hand tree and click on Edit
button to enable Create Model View button & active.


6.1 Create Model View

Provide Short text & Technical name as follows:
Tech name same as short text.Click continue button

The DI_MODEL will be added to the list as shown below


6.2 Distribution Model Changed overview

Select DI_MODEL and click on Add Message Type button


6.3 Add Message Type for Changed Distribution Model

Sender is SAP Logical System & Receiver is BODS Logical System.
Provide all information as shown above and click on continue button.
The Distribution Model created is shown below.


6.4 over View of Distribution Model

Check if IDOC distribution is turned on.
Run transaction "/nbd61" and make sure the checkbox is set.


6.5 Activation Status




Create Real Time Job in BODS

1. Create SAP Data store
Go to Data stores tab in BODS Designer, right click on blank area to create "NEW".
Create SAP Datastore by providing required SAP Application Server Information & creadentials


1.0 Creation of SAP Data Store


Expand the "SAP_TTL_Client100_DS" datastore
Select the IDOCs and Right click , click on Import By Name

1.1 IDOC'S importing


1.2 IDOC Import by Name

Provide Type & Name as shown above & click on Import to import IDOC in data-store

2. Build Real Time Job

Build a Real Time Job with a DataFlow inside



2.0 Real Time Job Overview.


Data Flow is as Follows

2.1 Data Flow

Query is used to Unnest as follows


2.2 Query Unnest


Note: You can unnest by right clicking on MATMAS01 as shown below


2.3 Unnest Option

The Target XML file configuration is a follows :


2.4 Target XML file Configuration

Note: Make sure Make Port checkbox is checked.


Configuring RT Services and RFC for IDOC through Web Administrator

1. Configuring Real Time Service & Service Provider
Open BODS Web Administrator
Expand Real-Time tree and select Real-Time Services to do Real-Time Service Configuration
Provide required information as shown below:


1.0 Web Administrator: Real Time Services Configuration

Job Name is Real Time Job name that you have created in earlier stage in designer (Ref: 2.Build Real Time Job,
page no 18)
Browse the Job and select Job name
In service provider settings - Click on Add button to add Job Server and make it enabled.

Once the Service is configured you can start the service by selecting the service & click on start button.


1.1 Real Time Services Status

2. Client Interfaces

Using BODS Web Administrator
Expand Real-Time tree and select Client Interfaces to do RFC Client Configuration
Provide required information as shown below


2.0 RFC Client Configurations

Click on Apply button

Note: RFC Program ID must be same as register Program ID given while configuring RFC Connection. (Ref:
3.Configure RFC Connection, page no 7)
Go to List of Supported IDOC tab & then click on Add button to add IDOC.


2.1 Supported IDOC

Provide IDOC Type which is BASIC IDOC TYPE in SAP. (Ref: Screen Shot 5.3 Partner Profile: Outbound
parameters, page no 13)
Service Name must be same as Real Time Service we have created & click on Apply button to get the IDoc
Information registered.
List of Supported IDOC will be displayed as shown below


2.2 Lists of Supported IDOC's

Note: You may add more than one iDOC type per RFC Client
Once the RFC Client configured you can start RFC Client.
Make sure RFC Client is started & status is Waiting for Idoc as shown below:


Shot 2.3 RFC Client Interface Status

3. Testing RFC Client from SAP
Use TCODE "/nsm59" to go to Configuration of RFC Connections
Open TCP/IP Connections & locate DI_RFC_DESTINATION
Double click & open DI_RFC_DESTINATION
Click on Connection Test button


3.0 Testing RFC Client from SAP

If everything configured properly, you will see success message as shown below.


3.1 Connection Test Success

Note: If Connection test failed, make sure that the RFC client on BODS side started properly with green icon.

Sending Material from SAP [/nbd10]
1. Sending Material MATMAS from SAP
Use TCODE "/nbd10" to "Send Material"


1.0 Sending Material from SAP

Click on the match code button


1.1 Material Numbers / Material Description

Click on Start Search button for finding the entries for material numbers.


1.2 Selections of Material and Material Description

Select any Material number from the list displayed above and click on OK button.
Selected material number will appear in front of "Material" on Send Material screen.
Provide required fields as shown below & make sure the "Logical system" is BODS Logical System ID100CBODS


1.3 Send Material

Click on Execute button. We will get couple of following messages.



1.4 IDOC Information after material sent
Click on OK


2. Check Received MATMAS IDOC in BODS Using BODS Web Admin
Go to BODS Web Admin, click on Real Time Services to check that one IDOC request received as shown below




2.0 Received Requests in Web Administrator: Real Time Services



1. Checking the output data in XML File
Go to BODS Designer
Open the Real-Time Job Dataflow
Go to the target MATMAS01_OUT.xml & click on view data to confirm the data received
Generated output XML file data as follows