ODI: Automating deployment of scenarios to production in Oracle Data Integrator

October 12,2009 | 11 Comments (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/)

In this post I will show you how you can automatically deploy scenarios in ODI.

It is rather cumbersome to manually deploy scenarios from a test/development to a production environment.

Typically it involves the following steps:

Manually (re-)generate all scenarios that need to be deployed and any child scenarios referenced.
Manually export the (re-)generated scenarios
Log in to the Operator module for the production environment and manually import the scenarios.

You do the above once or twice manually before getting extremely fed up. Actually I was rather patient (unlike my normal self) and did this about
ten times before I got very, very annoyed.

In this post I will show you how you can automate the deployment process. The proposed solution will allow you to logically group scenarios
together via marker groups for automatic deployment, thus giving you more flexibility and allowing you to just deploy a subset of scenarios in your
project.

To achieve our goal we will make use of the following Oracle Data Integrator features:

Marker groups
ODIGenerateAllScen tool
ODIExportScen tool
OdiImportScen tool
Meta-information in the ODI work repository
Execution of scenarios from a Windows batch file

In a first step we create a new marker group. We will use the marker group to logically group together scenarios we want to deploy. We will
subsequently use the marker group in the ODIGenerateAllScen, ODIExportScen, and OdiImportScen tools.

Log on to Designer > Expand Markers > Right click Markers > Select Insert Marker Group
As you can see from the figure above I have named the marker group Scenario and added three markers. One of the markers is named XLS (short
for scenarios that load data from Excel sheets). Flagging these scenarios (or rather their packages) via a marker will allow us to deploy them
separately.

Next we will add the XLS marker to those packages that we wish to logically group together for deployment. In our particular case, all of the Excel
related packages.

Right click package > Add Marker > Scenario XLS

In the next step we will create a package that will (re-)generate all packages marked with XLS.

Create a new package, name it GENERATE_SCENARIOS_XLS, add the ODIGenerateAllScen tool to it, and use the following parameters for the tool:

Project: The name of your project

Mode: Replace. This will overwrite the latest version of your scenario.
Marker Group: Scenario
Marker: XLS

Leave the default values for the other parameters.

In a next step we need to export the (re-)generated scenarios to XML. Unfortunately, the OdiExportScen tool does not allow us to make use of
marker groups to logically group together scenarios for export. To achieve our goal we need to make use of a workaround.

As ODI is a meta-driven ELT tool we can retrieve information about the marker groups from the ODI work repository.
The query below will exactly do this. It returns alls packages that are marked as XLS.

SELECT
scen_nameASpack_name
FROM(
SELECT
LAST_VALUE(d.scen_name)OVER
(PARTITIONBYc.pack_nameORDERBYscen_version)ASscen_name,
MAX(scen_version)OVER
(PARTITIONBYc.pack_name)max_scen_version,
scen_version,
c.pack_name
FROM
snp_obj_statea
joinsnp_state2bon(a.i_state=b.i_state)
JOINsnp_packagecON(a.i_instance=c.i_package)
JOINsnp_scendON(c.i_package=d.i_package)
WHERE
state_code='XLS'
)
WHERE
scen_version=max_scen_version

We will employ this query as an implicit cursor in an ODI procedure at the Command on Source. You will first need to create a data server to the
ODI work repository in Topology Manager for this to work. As per figure below, set the Technology to the technology of your work repository (in
my case Oracle) and set the schema to the logical schema of your work repository (in my case ORCL_ODIWORK_SRC).

We will then use the resultset together with the OdiExportScen tool to create XMLs for our scenario and write them to disk.

OdiExportScen"SCEN_NAME=#pack_name""SCEN_VERSION=1"
"FILE_NAME=D:\ODI\SCEN_#pack_nameVersion001.xml"
"FORCE_OVERWRITE=YES""RECURSIVE_EXPORT=YES"
"XML_VERSION=1.0""XML_CHARSET=ISO88591"
"JAVA_CHARSET=ISO8859_1"
Important parameters here are

SCEN_NAME: #pack_name. This is the bind variable from our Command on Source.
SCEN_VERSION: -1. This means that we will export the scenario that was generated last.
FILE_NAME: This is the path on your file system where the XMLs will be generated.

Make sure that you have set the Technology to Sunopsis API.

In a next step add the ODI procedure we just created to our package.

Finally, manually create a scenario for this procedure via ODI Designer.

Now we are in a position to import the exported scenarios from their XML files.

Once again we will use a procedure to achieve this

For command on source use the following query:

SELECT
scen_nameASpack_name
FROM(
SELECT
LAST_VALUE(d.scen_name)OVER
(PARTITIONBYc.pack_nameORDERBYscen_version)ASscen_name,
MAX(scen_version)OVER
(PARTITIONBYc.pack_name)max_scen_version,
scen_version,
c.pack_name
FROM
snp_obj_statea
joinsnp_state2bon(a.i_state=b.i_state)
JOINsnp_packagecON(a.i_instance=c.i_package)
JOINsnp_scendON(c.i_package=d.i_package)
Home (http://sonra.io/#home)
WHERE About (http://sonra.io/#about) Services (http://sonra.io/#services)
state_code='XLS'
Partners (http://sonra.io/#partners) Blog (http://sonra.io/#blog) Contact Us (http://sonra.io/#contact)
)
(http://sonra.io)
WHERE
scen_version=max_scen_version
For Command on Target use the following command

OdiImportScen"FILE_NAME=D:\ODI\SCEN_#pack_nameVersion001.xml"
"IMPORT_MODE=SYNONYM_INSERT_UPDATE"

Name the above procedure IMPORT_SCEN_XLS and generate a scenario for it.

We now have all the components that we need for automated scenario deployment. We will just have to glue them together. We will do this via a
batch file. In my particular case this will be a Windows batch. Of course, you can achieve the same in a Linux etc. environment.

ODI allows you to execute scenarios via the startscen.bat. You can find this batch file in the oracledi\bin folder in your ODI home. Three
parameters are mandatory for executing this batch:

%1: The name of the scenario. In our case these are the GENERATE_SCENARIOS_XLS and the IMPORT_SCEN_XLS scenarios.
%2: The version number of the scenario. In our case -1, as we want to export the last version of the marked scenarios.
%3: The execution context. In our case we deploy the scenarios from UAT to PRD. So for the export of our scenarios we will use the UAT context,
as this is where we (re-)generate and export the marked scenarios. For the import of the marked scenarios we will use the PRD context as we
want to import the exported XMLs into the production environment.

You will need to either have agents installed on the same server (one for each environment). I explain how you can install multiple agents
(http://sonra.io/?p=306) on one server in a separate post. Alternatively, you should be able to use a shared folder that both agents can access (I
havent tried this out).

@echooff
cls
d:
echogeneratexlsscenarios

cdD:\app\oracle\product\odi_home\oracledi\bin\
callstartscen.batGENERATE_SCENARIOS_XLS1UAT

echoimportxlsscenarios

cdD:\app\oracle\product\odi_home\oracledi\agent_prd\
callstartscen.batIMPORT_SCEN_XLS1PRD

Need ODI Training? Learn ODI from the experts!

I am proud to announce that Sonra are now offering ODI classes.

Value-priced. Customised. Onsite-Offsite. Online.

Five day standard training course.

Online.
Starts 24 February 2014.
Book before 15 January 2014. Pay $949.

More information on our ODI courses (http://www.odi-training.com).

Related posts

July 15, 2009 Losing timestamp in ODI when using ojdbc14.jar (http://sonra.io/losing-timestamp-in-odi-when-using-ojdbc14-jar/) The ojdbc14.jar

version of the JDBC driver is losing timestamp information when []

July 16, 2010 Oracle Data Integrator: ODIInvokeWebService and Java 6 (http://sonra.io/oracle-data-integrator-odiinvokewebservice-and-java-6/)

ODI has a built in SOAP client, the ODIInvokeWebService tool. However, there is one []

March 29, 2010 Nesting ODI substitution method calls. Part 1. (http://sonra.io/nesting-odi-substitution-method-calls-part-1/) I am not getting

around doing too much blogging on ODI these days. I have recently []

May 20, 2010 ODI and analytic functions (again???) (http://sonra.io/odi-and-analytic-functions-again/) My friend and colleague Maciej Kocon has come

up with the following trick. Rather than []

June 9, 2009 Developing a Knowledge Module in ODI published on OTN (http://sonra.io/developing-a-knowledge-module-in-odi-published-on-

otn/) ODI, Oracle Data Integrator, Knowledge Module, IKM

Tags: automatic deployment odi (http://sonra.io/tag/automatic-deployment-odi/) deploy scenarios odi (http://sonra.io/tag/deploy-scenarios-odi/)

deployment odi (http://sonra.io/tag/deployment-odi/) odi (http://sonra.io/tag/odi/) oracle data integrator (http://sonra.io/tag/oracle-data-integrator/)

Author: admin (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/)

11 Comments

AG (http://Website) says:
December 9, 2010 at 10:20 pm (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/comment-page-1/#comment-9614)

Is this also way to migrate from say dev test- prod , we have diff environments. People suggest on forums to take database exports of
the master and work rep schemas and deploy them in required environments.

And them make changes to the topolgy and associate the right agents. Please guide via email

Gurcan Orhan (http://gurcanorhan.wordpress.com) says:

January 2, 2013 at 8:12 am (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/comment-page-1/#comment-16657)

Cheers,
Nice artice, I liked it a lot. Gurcan.

January 2, 2013 at 9:39 am (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/comment-page-1/#comment-16658)

Uli Bethke (http://admin) says:
Thanks Gurcan.

In my opinion this method and execution repositories in general should only be used in very small projects. At an enterprise level I
would recommend to have a full work repository in TST and PRD. I am not a big fan of smart import/export as it has various limitations
and annoyances. For enterprise level we use the old import/export as this can be automated.

Cheers
uli

Michael Rainey (http://www.rittmanmead.com/blog) says:

January 3, 2013 at 6:55 pm (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/comment-page-1/#comment-16659)

Hi Uli,

Im curious, would you recommend your automated deployment process over using Solutions in ODI? It seems you would get the
same result ability to regenerate all Scenarios, only one object to export / import (the Solution), etc. Of course, I always like the
flexibility of custom code in ODI.
Nice work!

Regards,
Michael Rainey

Uli Bethke (http://admin) says:

January 4, 2013 at 11:05 am (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/comment-page-1/#comment-16661)

Hey Michael.

This post is really old and I would not recommend to have an Execution Work Repository for TST or PRD at all unless your projects
are very small (non-enterprise stuff).

While Solutions can be used to deploy scenarios I cant see how deployment could be automated with them. I have evaluated Solutions
for deploying objects and version control in both ODI 10 and 11 and came across so many bugs and issues that I have limited
confidence in them. I also dont think that Smart Import/Export is the way to go as it will overwrite dependent objects unless you
interfere manually. The old import/export with a deployment list is the way to go to automate this.

Cheers
uli

Manohar Reddy.S (http://www.business-intelligence) says:

January 16, 2013 at 2:55 am (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/comment-page-1/#comment-16700)

Hi GMng
1)How to excute 2 interfaces paralley in Oracle Data Integrator? 2)Can we use SCD in SnowFlake Schema in Oracle Data Integrator?

3)How to client finds the error records in production(Is there any programmers in project to track the errors records)?

4)How to write sub_query in ODI(Procedure,Yellow Interface) 5)How to truncate the target table before loading into target?

6)How to import objects from one environment to another environment?

Manohar Reddy.S (http://www.business-intelligence) says:
January 16, 2013 at 2:57 am (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/comment-page-1/#comment-16701)

Hi Sir..
Q)Group By only one column in Oracle Data Integrator? Q)Any pre_requisites to catch the error records in Oracle Data Integrator?

Q)Main use of Yellow Interface? Q)What is the difference between yellow interface and normal interface?

Q)How to find error records without error table? Q)Difference between Context And optimized Context?

Q)How to use equal operator in filter? Q)How to loop 10 xml files in a directory? Q)Can we use same IKM for SCD1 and SCD2?

Thanks&Regards
Manohar

pedro duran says:

February 11, 2013 at 2:23 pm (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/comment-page-1/#comment-16721)

DEV(development)
UAT ( execution) populated with ODI tools, and only the stuff we need on production.
Why not perform a full sinc between work reps? PRD (truncate select insert between UAT PRD)

PS: Changing of course the Work repository ID.

tom says:
June 3, 2013 at 7:25 pm (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/comment-page-1/#comment-16845)

Uli,
As we are in development and adding variables as we go, periodically regenerating the scenario (and selecting new vars to be included
in the scen) would this solution stop and allow you to check the new variables to be included?
If not, what happens? Does it just create/replace the scenario without the new vars?
Apprec the feedback. Thanks!

Richard Lyders says:

August 21, 2013 at 6:48 pm (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/comment-page-1/#comment-16882)

Hi Uli,
RE: This post is really old and I would not recommend to have an Execution Work Repository for TST or PRD at all unless your projects
are very small (non-enterprise stuff).

Considering that this post is from 2009, would you be so kind as to clarify whether you still recommend this promotion procedure
detailed in this blog post when using the latest ODI 11g (11.1.1.7)?

We are in the planning stages of our current project and I am trying to nail down a trusted, automated promotion procedure for our
ODI work when moving from DEV to INT and then PROD. We are trying to avoid manual steps where possible.

Regards,
Rich
 Arju (http://arjun@lagisetty.com) says:
December 17, 2013 at 8:16 pm (http://sonra.io/odi-automating-deployment-of-scenarios-in-oracle-data-integrator/comment-page-1/#comment-17846)

Hi Uli,
I was wondering why would you not recommend having execution work rep for test and production?
It seems counter intuitive for me?
Solutions are nice but they force us to export/import at a folder/ project level. Instead of package level.

What's your Opinion?

Full Name

Email

Site

Message...

Send Message

(http://feed.feedcat.net/857727) (https://twitter.com/ubethke)

BigDataMadeEasy
Big Data can be hard. Businesses need to acquire expertise in a lot of areas to be successful. See how the Sonra Hadoop Quick Start Appliance
(http://sonra.io/wp-content/uploads/2015/03/The%20Sonra%20Hadoop%20Quick%20Start%20Appliance.pdf) can help you get started in days or
weeks.

SonraIconplcSuccessStory
Read how Sonra has helped Icon PLC (http://www.oracle.com/us/corporate/customers/customersearch/icon-1-odi-s-2035228.html ) to reload their
data warehouse in three hours, deliver near-real-time management information, and provide daily drug-trial updates to sponsors. Read the full
story (http://www.oracle.com/us/corporate/customers/customersearch/icon-1-odi-s-2035228.html ).

ODITrainingfromTHEExperts
Value-priced. Customised. Onsite-Offsite. Online.

More information on our ODI Training Courses! (http://www.odi-training.com)

SearchourBlog
 Search

Tags

analytic functions (http://sonra.io/tag/analytic-functions/) Big Data (http://sonra.io/tag/big-data/) MapR (http://sonra.io/tag/mapr/)

OBIEE (http://sonra.io/tag/obiee/) odi (http://sonra.io/tag/odi/) ODI 11G (http://sonra.io/tag/odi-11g/)

odi best practice (http://sonra.io/tag/odi-best-practice/) Oracle (http://sonra.io/tag/oracle/)

oracle data integrator (http://sonra.io/tag/oracle-data-integrator/)

Oracle Warehouse Builder (http://sonra.io/tag/oracle-warehouse-builder/)

(http://feed.feedcat.net/857727)
(https://twitter.com/ubethke)