Problem with embedded jobs

We have a main Job called Overall_process_flow.dmc

There are 11 embedded jobs in the main job Overall_process_flow.dmc

The 11 main embedded jobs are :

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. Communications_Process1.dmc Organisation_Process1.dmc Hierarchies Process1.dmc Address Purpose Process1.dmc Customer Facts Process1.dmc FAccounts Process1.dmc Finformation Process1.dmc Orelationships Process1.dmc PRelationships Process1.dmc Person Process1.dmc Address Process1.dmc

Example of an embedded job-Communications Process1.dmc

The typical main embedded job called Communications Process1.dmc is calling more embedded jobs ,which are encircled.

Now the error .....

When we try to run the main job as seen on slide 1 ,with all the 11 embedded nodes we get TNS error messages. The job is using 2 saved connections to two different databases.

The error mostly looks like ,its coming from the database , but each time the
Overall_Process_Jobl.dmc is run ,it throws different error messages. Such as
Communications: D:\DataFlux\MDM-Customer\DF_Jobs\Communications Process1.dmc: Architect Engine - Embedded job: [08001] [DataFlux][ODBC Oracle Wire Protocol driver][Oracle]TNS-12519: TNS:no appropriate service handler found (-1). Hierarchies:D:\DataFlux\MDM-Customer\DF_Jobs\Hierarchies Process1.dmc: Architect Engine Embedded job: D:\DataFlux\MDM-Customer\DF_Jobs\BR001 Remove Spaces.dmc: Architect Engine Embedded job: [08001] [DataFlux][ODBC Oracle Wire Protocol driver][Oracle]TNS-12519: TNS:no appropriate service handler found (-1)..

As we can see the error message is pertaining to TNS 12519 .As such we have no TNS errors while accessing the database via sql queries and the ODBC connections are all saved in Dataflux.

Now for the tricky part....

When I tried to isolated the main embedded jobs (slide 2) ,by reducing the number of main embedded jobs to 5 ,the same errors dont appear and the Overall_Process_flow.dmc runs fine. Below is how we modified this job to check this behaviour. We are only having 5 embedded jobs in this modified job. And it runs without the same error messages. Here Hierarchies and Communica tions are throwing errors

Steps we took to try and resolve the problem

1. Changed the Processing order of of the 11 Embedded jobs and serialized them, by assigning numbers from 0 to 10 - for all 11 jobs.

Contd.

2. Changed the memory cache size of the Branch node

These steps have not resolved the problem

We continue to get the TNS error when we run the 11 embedded jobs Verified that we dont get the TNS error when we run 5 embedded jobs If we run more than 5 embedded jobs , we see the error for different jobs at different times when we are running the Overall Process Flow job . There is no single embedded job we could isolate reliably which could have been causing the issue.Thus there is no consistent pattern of error messages.

Steps suggested by SAS Techsupport 8/30/2012

In words of SAS Techsupport

If you get a TNS - 12519 TNS: "no appropriate service handler found" the very first thing to check is the max session/processes. Just increase it an restart your database.

Steps taken by DBA 8/30/2012

These are the steps taken by the DBA team

DBA team checked that whenever the job is run the database is reaching max no of process (600) causing the job failure we have to increase the max no of process to 800 .