Dataflows for each Extractor Delta Process Type

Summary
One of the nice things about Extractors is, they tell us what kind of delta is being produced. It might be a list of changed records with their
current values, it might be even the complete change information with the values before the change and after. Some contain deletes, other
sources cannot even be updated - only new records are added. Based on this information we want to build some useful dataflows.

SAP BW has built in methods to consume Extractors and apply the change information provided to the cubes. These methods depend on
the kind of delta the Extractor provides but also how the target structure is modeled.
Do you want to keep the latest version of the record only? Do you want to be able to see the entire history of the record?
Depending on the answer the logic how to use the delta and apply it to the target will be different entirely. In BW you have just little control
on how the data has to be applied which does simplify the task of loading the deltas greatly. When there is light, there is shadow.

The list of Extractor Delta Process Types

The first main differentiation is Full, Extractor based Delta or DeltaQueue based Delta. Full is obvious, it will return the entire data always.
Extractor based delta is when the ABAP representing the Extractor scans through the ERP tables and will search for changes. How? We
don't know, it is the Extractor's job to do that and fast. DeltaQueue based delta is usually a case where when the user made a change the
system did insert the change into the delta queue as well. Logically there is no difference between the two but the Extractor based one
might have timing constraints, e.g. it will return only the changes happened yesterday or older but not the changes made a minute ago.

Requirements when building a delta dataflow

When we build any kind of delta dataflow, if Extractors are used or not, we should think about exceptional cases as well.

 What will happen if the dataflow dies during the load?

 If for some reason the same delta is executed again, the dataflow should be able to deal with that - e.g. no primary key violations if the
delta said a new record should be inserted.
Dataflow for Extractor Delta Process Type A
Summary
The delta type "A" is another Extractor type where we have to use DataServices to figure out what was changed, what is new.

What does the Extractor return?

This delta method is used for master data only, for example the Cost Accounting Activity Attribute Extractor 0ACTTYPE_ATTR. Usually this
delta method is configured for Extractors based on plain tables but the tables do have a change timestamp.

Dataflow for this delta type

The dataflow is a simple 1:1 copy of the data, only the loader has the checkbox "autocorrectload" ticked. Only important thing to
doublecheck is that primary keys are configured.
Dataflow for Extractor Delta Process Type ABR
Summary
The delta type "ABR" stands for After-, Before- and Reverse-Image, so it provides you with the entire information. If a record was new, if a
record got update from old value (=before image) to new value (=after image) or got deleted.
Processing that is simple, you either feed it into a Map_CDC transform to get the current status only (BW term: Overwrite) or you append
the data to keep full history (BW term: Addition).

What does the Extractor return?

As an example the 2LIS_11_VAHDR Extractor was used here. It is the DataSource returning all Sales Order Header documents on
document level and we append the change data to the target table.

After the initial load, a new sales order got created. The customer 1300 had ordered 20 stacks of lamps with material number L-80C.

The subsequent delta load at LOAD_DATE 13:14:08 did show that, one row was received with the correct sales order document number
VBELN.

Now the sales order got modified, it got moved from the default sales organization to another (and the sold to party is a different one as
well, but that is shown in another extractor).
The next delta run at LOAD_DATE 13:31:26 did present us with two rows, one saying that for the initial sales organization the row should
be deleted and a second for the same document but in a new organization.

And finally, the sales order got deleted altogether

and obviously we received one row with a delete flag (at 13:34:59).

Dataflow for keeping the current data only

In case you would care about the most recent status of the data only, all we need is sending the data into the Map_CDC transform. As the
Extractor Reader does create the columns DI_SEQUENCE_NUMBER and DI_OPERATION_TYPE required by the Map_CDC for us
already, there is nothing else we have to do. We don't even need to sort the data again in the transform, the Extractor does send us the
data sorted.
However this dataflow does violate our requirements of dealing with errors gracefully and allow re-reading the same delta again. What
would happen if the same data is loaded again? The same document - which is the primary key - will be executed a second time and fail.
to avoid half the data being loaded it would make sense to set the loader option "include in transaction". Now all the data is loaded and only
at the very end, a single commit will be executed against the target. So either all the data or none will be loaded.
That does not help when the same delta is requested another time, then we could execute a delete statement on all data with the given
$G_SDATE.

Dataflow with history

But it is more likely you want to keep the history and then all we have to do is appending the data as is, only looking at the
ODQ_ENTITYCNTR field. The creation of a new order would be a +1 operation, an update we would get as -1 for the old value and +1 for
the new and a deletion is a -1.
So we append the data, the primary key will be the same as in the source plus a change_date column mapped to the $G_SDATE variable.

We should delete all records with CHANGE_DATE = $G_SDATE prior to the dataflow start, this way rerunning the same delta again will
delete the previously loaded data for this delta and repopulate the same rows again. And in case of a failure, same thing will happen, we
will request the same $G_SDATE again and delete the partially loaded data before running the dataflow.
Dataflow for Extractor Delta Process Type ABR1
Summary
The method ABR1 is identical to ABR but it does not allow parallel processing, the order of the incoming data has to be retained.

The only example found is 2LIS_03_BF for the material movement data.

What does the Extractor return?

So after the initial load of all 700'000 rows

a new material movement document was created. For the purchase order 4500017118 from vendor AGENCY01 we received one piece
and did put it into storage location 0001, the material movement document 5000000000 got created.

The Extractor the returned this one new document as a single line with the corresponding information with the load date 13:15:20.

As no material movement can be changed later, another cancellation document got created for this movement.
A new document, the cancel document got added, a single row with the load date 13:28:56.

So as result for this vendor no item was received effectively. But the DI_OPERATION_TYPE cannot be used with this extractor, it would tell
for the cancel that the material movement document 5000000001 has to be deleted - but that is a brand new document.

Dataflow with history

As the changes are received as new documents, we can simply append the information into the target table as it seems.
Dataflow for Extractor Delta Process Type ADD
Summary
Extractors with delta ADD are ones that append data only to the target, pure additive load. This can be either because the source does not
allow any updates or deletes just adding new rows - finance is a common example - or the extractor itself does calculate the change
amount for us already, e.g. value was 1000USD, now it got changed to 1500USD and the extractor will return one row with +500USD.
Either way, the dataflow is trivial: We append the data in the target.

The Extractor 0CO_OM_CCA_9 returns all cost center bookings on document level. As you cannot change or delete bookings, you can
only add new bookings correcting the initial one, the Extractor is of type ADD.

When a new booking got created, like this one with the document number 200152831 where we assigned costs from cost center 4230 to
cost center 4235, we will get the corresponding data.

This booking has actually two lines, one removing the costs on the old cost center, a second adding it to the new one.

And that is what we do exactly, we append the fact table with those two rows and we are done. For restart-ability reasons the bookings
$G_SDATE value will be stored in an additional column of the booking table so we can delete all rows with the requested extractor delta
prior to starting the dataflow in case the same delta is requested another time.
Warning
This particular extractor will return delta from the previous day only. This is a limitation of the Extractor itself, same thing happens in BW. So
after creating the booking, we had to wait until the next day to get the change information actually.
Dataflow for Extractor Delta Process Type ADDD
Summary
The Delta Type ADDD is an ADD delta but using the deltaqueue mechanism to store the changes. Dataflow is identical to the ADD method.

Again our system had just 2 Extractors of that kind, one is 0EC_PCA_1.

What does the Extractor return?

After the initial load a new profit center booking from Profit Center 1500 to 1300 with the amount of 500EUR was entered.

The returned data is the two booking lines obviously.

Dataflow with history

All we have to do is inserting the data into the target table, the extra column LOAD_DATE we set to $G_SDATE so we can delete records
in case a delta is to be reloaded a second time. Not sure what UPMOD is however. It is not the 0RECORDMODE column each Extractor
has.
Dataflow for Extractor Delta Process Type AIE
Summary
Extractors with delta type AIE produce an after image only, they show what the current status of the record is. As BW cubes need the
before and after image values, the DSO activation in BW is mandatory. With DataServices it depends on what you want. In the majority of
the cases you will use Table Comparison to overwrite any existing record with the new value.

For this example we use the Extractor 0FI_GL_6, one that returns the summary values per period, account and other important attributes.

During the initial load, the Extractor will aggregate all bookings as they exists of now and present you the current values. Here in this
example the Extractor said that for account 133000, period 2012/03, the credit balance (SHKZG=S) is zero.

Then a new booking was made with the amount 9999.- from the 133000 account to the 45020 account.

As result the Extractor returned a debit balance now for this account 133000 (account 45020 would show a different value as well).

The initial load dataflow

In the initial load we simply truncate the target table and append all rows. As the logical primary key of this table would be all columns
except the amount and quantity columns, we have added an additional surrogate key as the primary key. For simplicity we mapped the key
with the formula gen_row_num().
The delta dataflow
For us the goal is to insert/update the new values into our target table. As we have no information if that record exists in the target already,
what its value is, we have to do that comparison in the DataServices, the Table Comparison Transform is meant for that.
The second target table storing the changes we have added to record the changes, it is not required.

The TableComparison has as input primary key columns all attributes: account, period, credit/debit indicator,... all columns except the
measures. For each input record it will try to find the matching target row, if it exists and is different, then an update row will be generated.
For brand new rows it will be flagged as insert.
The subsequent Key Generation Transform does generate new surrogate keys for the insert rows and the loader has all the information he
needs to insert or update the incoming rows in the target.
Warning
This particular extractor does not return the delta changes immediately, only the once made the day before.
Dataflow for Extractor Delta Process Type AIED
Summary
The AIED delta is the same as AIE except that deletions are sent as deletions, not updates to the amount of zero.
So the only difference to AIE is that we might get rows with ODQ_CHANGEMODE=U and ODQ_ENTITYCNTR=0. This row might not have
all columns being set other than the primary key columns.

When I checked all Extractors in the IDES system of this type I haven't found many, only 0FI_GL_10 and 0FI_GL_20. They return actual
and plan data, but only plan data can be deleted - I don't have any plan data.

Anyway, we will get a row marked as deleted for the given primary key, we don't care, we will compare it based on the primary key in the
Table Comparison transform and update it with zero values. Same dataflow as with the AIE case.

Only if a delete is truly wanted, then we split the Extractor data into a parallel path for the deletes.
Note: An enhancement request was filed so the Table Comparison transform does pass through rows marked as delete.
Dataflow for Extractor Delta Process Type AIM
Summary
The delta process type AIM is just like AIE one where we will get the current values only, so we have no information if the row is a new one
or a change to an existing one. The only difference to AIE is, this delta comes from the delta queue, but for us as end users we don't care.
Dealing with this kind of input is simple as well, we will ask DataServices to compare.

The Extractor used for this example is Accounts Payables, the one named 0FI_AP_3.

What does the Extractor return?

After the initial load a new document got created, the document number was 1000000006 moving the amount of 1'234 EUR from account
1000 to account 133000. The payment term used here was ZB01, meaning 3% discount if paid within 14 days.

So the Extractor did return two lines, one per line item, here you can see the one for account 1000 loaded at 13:47:43.

Next the document got changed, the payment term is now ZB04, 2% discount for 10 days.
So we got the same document another time, load time was 13:57:41 with the new payment information in the three columns.

Dataflow for keeping the current data

In the previous examples of AIE and AIED we used a TableComparison transform as the Extractor did not return the data on document
level but in an aggregated manner. So our primary key was all attributes except the measures, it was many columns. Therefore I wanted to
have a surrogate key - TableComparison is the way to go.
This Extractor here does return the data on document level, so although TableComparison can be used, we do have the option to use a
simple loader flag as well, the autocorrect load.

With that our dataflow is perfect, if we request the same delta again, we will compare the data with the already loaded once thanks to the
autocorrect flag. And the same is true if the previous run failed in the middle.

Dataflow for Extractor Delta Process Type AIMD

Summary
Identical to AIM except that internally the changes are collected in a deltaqueue. But for DataServices there is no difference.
Dataflow for Extractor Delta Process Type CUBE
Summary
The delta type "CUBE" is the most simple one for DataServices, you get the data of a cube with all attributes marked as primary keys
already and we get the changes only from the Extractor. So all that is left is loading the data with AutoCorrect load option checked.

What does the Extractor return?

As an example the 0VER_DELTA_WITH_LONG_NAME Extractor was used.

Dataflow for this delta type

The dataflow is a simple 1:1 copy of the data, only the loader has the checkbox "autocorrectload" ticked.
Dataflow for Extractor Delta Process Type FULL
Summary
A FULL delta means, we will get all source rows always. This is very often the case for all the many, many small tables like list of company
codes etc.
The dataflow for that is simple, it is the same as reading from a database table all data every day.

One Extractor of this kind is 0FI_GL_1, returning the aggregated General Ledger figures from the GLT0 table.

The first think to notice is when you import that Extractor, the Changed Data Capture item is grayed out. Of course it is, it does not support
delta!

What we do with this data is entirely up to us, we could truncate the target table and replace the old data with the new one. Very likely what
you want in this instance.

The amount of data is not that much, just 400'000 rows which took little bit more than 1.5 minutes. Note, the SAP server and DataServices
did run on a tiny VMWare image.
But we could also use TableComparison Transform to compare the data, including figuring out the deleted rows. And with an additional
History Preserving Transform we could even build the history of each booking, yesterday the account had a balance of 200 USD, today it is
300 USD, something like that. As said, standard mode of operation for any ETL tool.
Dataflow for Extractor Delta Process Type NEWD
Summary
Extractors of type NEWD only add new records. No delete, no update just inserts. Hence the dataflow is a simple Extractor to target
mapping.

The extractor for Invoice Contract Account data is an example, 0FC_INVDOC_00. You cannot delete the invoice, you cannot change an
invoice record. All you do is inserting a new one.

There is one interesting side effect however for this Extractor which is not that uncommon. When you execute the dataflow with this
Extractor you will get the error message "selection criterion BUDAT missing from data request".

So while for many Extractor the selection criteria is variable, this one forces you do pick an inventory booking date or a range. So you have
to specify that during import or rightclick the Extractor in the datastore tab of the object library and in its properties, add a condition on
BUDAT.
Delta type in BW Extractors
Each BW extractor have an delta type, it can be seen in the table ROOSOURCE. The delta type determines the datasource to have DSO
or InfoCube in the dataflow. There is an important table that maintain all the delta type.
This page will describe the content of RODELTAM table and the use of the specific delta type.

BW Delta Process ( RODELTAM Table )

UseThis table defines the delta process permitted for a DataSource and the properties of this delta process. The entries are maintained
centrally by Service API development.
A delta process is defined by the following properties:

Attribute description

ONLY_FULL DataSource delivers no delta

UPDM_NIM delta with new images

UPDM_BIM delta with before images

UPDM_AIM Delta with after images

UPDM_RIM delta with additive delete records

UPDM_ADD additive delta

UPDM_DEL delta with non-additive delete records

DREQSER mandatory serialization

DELTATYPE type of delta process

A ALE Update Pointer (Master Data)

ABR Complete Delta with Deletion Flag Via Delta Queue(Cube-Comp)

ABR1 Like Method 'ABR' But Serialization Only by Requests

ADD Additive Extraction Via Extracto (e.g. LIS Info Structures)

ADDD Like 'ADD' But Via Delta Queue (Cube-Compatible)

AIE After-Images Via Extractor (FI-GL/AP/AR)

AIED After-Images with Deletion Flag Via Extractor (FI-GL/AP/AR)

AIM After-Images Via Delta Queue (e.g. FI-AP/AR)

AIMD After-Images with Deletion Flag Via Delta Queue (e.g. BtB)

CUBE InfoCube Extraction

D Unspecific Delta Via Delta Queue (Not ODS-Compatible)

E Unspecific Delta Via Extractor (Not ODS-Compatible)

FIL0 Delta Via File Import with After-Images

FIL1 Delta Via File Import with Delta Images

NEWD Only New Records (Inserts) Via Delta Queue (Cube-Compatible)

NEWE Only New Records (Inserts) Via Extractor (Cube-Compatible)

ODS ODS Extraction

X Delta Unspecified (Do Not Use!)

 0RECORDMODE & DELTA MANAGEMENT
What is Delta Management ?

Delta management is the process of extracting only new or modified records into BI system which occured in
the source system.

0RECORDMODE :

0recordmode describe the type of change that a data record contains.

If the datasource supports delta then the field ROCANCEL which is a part of datasource holds the changes from
r/3 side.
The field ROCANCEL assigns a data to the InfoObject 0RECORDMODE in the BI system.

After-Image ‘ ‘ :
It represents how the record look like after it has been changed.

Before-Image ‘x’ :
It represents how the record look like before it has been changed.It assigns the –ve sign of the original value.
New-Image ‘N’ :
The record provides a new image when a new record is created.

Add ‘A’ :
The record provides an additive image.Only the differences of the numeric values are available.
Delete ‘D’ :
This recordmode type signifies that the record must be deleted.

Reverse-image ‘R’ :
The record provides a reverse image.The content of this record equivalent to a before image.The only difference
occurs when updating a datastore object.
 How do we identify whether the data source supports delta or not ?
Goto RSA6 or SBIW then find ur datasource and double click on it u will find the following screen.

How to know the delta process for datasource ?

We can find this in table ROOSOURCE(source system) or in the table RSOLTPSOURCE(data sources in bi 3.x)
or in table RSDS (data sources in BI).

How data is brought to BI ?

We can brought the data from sap r/3 to sap bi by using one of the following delta processes.
 ABR – After before and reverse image.

 AIE/AIM – After image.

 ADD – Additive image.

What is ABR ?

Once a new entry is posted or existing record is modified then after image shows the status of the after
image,before image shows status before the change with the negative sign and reverse image shows the status
–ve sign next to the record while indicating for deletion.

What update type(for keyfigures) it supports ?

1. Addition.
2. Overwrite.

Does it supports loading to both Infocube and Dso ?

Yes.

What is AIE/AIM ?

Once a new entry is posted or an existing posting is changed at R/3 side, an after image shows the status after
the change.

What update type (for key figures) it supports?

Overwrite Only.

Does it support loading to both infocube and DSO?

No, only DSO.

What is ADD ?

Once a new entry is posted or an existing posting is changed at R/3 side, an additive image shows the difference
between the numeric values.
What update type (for key figures) it supports?

Addition Only.

Does it support loading to both infocube and DSO?

YES (In case the DSO has Update type Addition).