2011 10 EBS Concurrent Processing Best Practices

1
Copyright 2011, Oracle and/or its affiliates. All rights reserved.
My Oracle Support Advisor Webcast Program EBS Concurrent Manager Performance - Best Practices
Tuesday, October 11, 2011 10:00 AM CET (GMT +2) Teleconference Access:
Upcoming Webcasts in EBS Technology area :

o November 2011 : None planned at this stage
North America: 1866 966 9439

International (UK): +44 (0) 1452 562 665 Conference ID : 95984161
Do you have any requests for future EBS - Technology Webcast Events? Please email your suggestions to ruediger.ziegler@oracle.com ,subject: Topics of Interest.
AGENDA
Presentation and Demo approximately 60 - 75 minutes
Q&A Session maximum 15 minutes

Web attendees can ask questions via Q&A panel Phone attendees can ask questions via Q&A panel or phone
3
Who to ask?
3 Send
your question
Q&A panel
type your question here
your question pops-up here
ATTENTION AUDIO INFORMATION

Voice streaming/Audio broadcast is available. For full audio access, please join the telephone conference.
Teleconference Connect details: 1. Conference ID: 95984161 2. International dial in: +44 (0) 1452 555 566 (UK) / 1866 966 9439 (US) 3. List with national toll free numbers is available in Note 1354461.1 Note: You can view this info anytime using WebEx menu from your WebEx-Session : Select Communicate --> Join Teleconference.
6
Safe Harbor Statement

The following is intended to outline our general product direction.
It is intended for information purposes only, and may not be

incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making
purchasing decisions. The development, release, and timing of any

features or functionality described for Oracles products remains at the sole discretion of Oracle.
EBS Concurrent Manager Performance - Best Practices

Presenters: Kandasamy Nainamalai (India) and Kalaivani Murugesan (India)
Q&A Panel Experts : Amira Abdelkader (Egypt), Rakesh Chithuluri (India), Michael Costa (US), Brian Kerr (US), Mario Bagnara (Chile), David Heisler (US)
8
Program Agenda
Overview of Concurrent Manager Enhancing Performance - Best Practices
Tuning Concurrent Queue Parameters
Specialized Concurrent manager

Output Post Processor OPP Tuning PCP/RAC Transaction Manager (TM)
Common / Typical issues and resolution

Purging Concurrent records Running cmclean.sql OPP and TM related issues.
Resolving concurrent requests
Trouble-shooting Concurrent Manager

9
Program Agenda



10
EBS Release 12 Architecture
11
Concurrent Manager - Introduction

Batch Processing Sub-System Programs are run as operating system background processes. These programs may be written using a variety of Oracle tools, programming languages for executables, or OS scripts. The Internal Concurrent Manager (ICM) controls all other concurrent managers. It administers the startup and shutdown of managers as defined by their work shift, monitors for process failure, and cleans up if a failure occurs. The Conflict Resolution Manager (CRM) enforces rules designed to ensure that incompatible concurrent requests do not run in the same conflict domain. The Standard Manager as shipped with Oracle Applications will accept and run any concurrent requests.
12

Concurrent > Manager > Administer
Are the Managers up and running ?
13
14
Concurrent Requests, Programs, and Processes
User requests Concurrent Program
System maintains list of requests to start Concurrent Programs Queue Table Run program ... Run program X Run program ...
Concurrent Managers Read applicable Requests and start Concurrent programs
Concurrent manager
Program X Started
15
Concurrent Manager - Tables

Table FND_CONCURRENT_REQUESTS FND_CONCURRENT_PROGRAMS Content Details of user requests, including status, start date, and completion date. Details of concurrent programs, including execution method, whether the program is constrained, and whether it must be run alone.
FND_CONCURRENT_PROCESSES Cross-references between concurrent requests and queues, and a history of concurrent manager processes. FND_CONCURRENT_QUEUES Information about each of the concurrent manager queues.
16
Concurrent Request - Lifecycle

Phase Pending / Normal Activity The request is waiting to be run
Pending / Standby
Running / Normal Completed / Normal Inactive / No Manager
The request is waiting at CRM

The request is running The request has finished The request cannot be run. No manager available to process it.
17
Program Agenda



18
Enhancing Performance - Best Practices

Sleep Seconds
Cache Size Purging Gather Schema Statistics Workload Management Specialized Concurrent managers Output Post Processor OPP
Java Heap Size

Tuning PCP/RAC Transaction Managers
19

Sleep Seconds - is the number of seconds your Concurrent manager waits between checking the list of pending concurrent requests (concurrent requests waiting to be started). Tips: Set the sleep time to be very brief, during periods when the number of requests submitted is expected to be high. Otherwise set the sleep time to a high number (e.g. 2 minutes). This avoids constant polls to check for new requests. Concurrent Manager will not sleep at all if there are pending requests.
20
Tuning Sleep - Parameters

Concurrent manager > define > Workshifts
Reduce the Sleep Time
21

Cache Size - Increase it (number of requests cached) to at least twice the number of target processes. For example, if a manager's work shift has 1 target process and a cache value of 3. Then it will read three requests, and try to run those three requests before reading any new requests. Tips (Assume you defined one Standard Manager process)
Set a value of 1 if the managers run long running jobs, Set 3 or 4 if the managers run short running jobs.
22
Tuning Parameters Cache

Concurrent > Manager > Define
Increase Cache Size
23

Process Increase it if the pending/normal request are high. For example, if the Standard Manager has a high number of pending request in the queue, then we can explore whether we can increase the process. Tip: Need to increase after examine the server capacity. If the server is not capable of handing such number of concurrent requests in parallel, then it can even downgrade the performance.
24
Tuning Parameters Processes

Concurrent > Manager > Define > Workshifts
Increase the Number of Processes
25
Purging Log Files and Tables

To conserve space you should periodically delete log and output files.
Use the Purge Concurrent Request and/or Manager Data program to purge Request Log files, Concurrent Manager Log Files, and report output files. Schedule as needed for each instance.
26
Purging Concurrent Requests

At regular intervals, Purge backend tables that are holding Concurrent Request related information and delete log/out files from concurrent manager tier. To purge, It is recommended to run the request the "Purge Concurrent Request and/or Manager Data (FNDCPPUR) with below Parameters at regular interval. Entity =All Mode=Age Mode Value=10 (Number of days to keep) Tip Discuss with Functional team for to fix the retention policy)
27

Purging Log/out files Ensure that the log/out files are removed from the below directory as you run "Purge Concurrent Request and/or Manager Data program". $APPLCSF/$APPLLOG $APPLCSF/$APPLOUT Apply Patch 7834670 (11i) 7530490 (R12). Refer to Note 822368.1 for more details. Truncate the reports.log file in log directory. Refer to Note 844976.1 for more details.
28
Concurrent Processing Performance

Defragment the tables to improve Performance As purging is done periodically, it deletes rows from the FND_CONCURRENT tables . Hence tables get fragmented over a period. It degrades the performance when full table scan is performed De-fragment the below tables periodically to reclaim unused space. It improve performance. (Refer How to defragment) FND_CONCURRENT_REQUESTS FND_CONCURRENT_PROCESSES FND_CRM_HISTORY FND_ENV_CONTEXT FND_TEMP_FILES
29

Defragment the tables to improve Performance
Table defragmentation alter table <owner>.<table_name> move; Index check select owner, index_name, status from dba_indexes
where table_owner = upper('&OWNER') and

table_name = upper('&SEGMENT_NAME'); As some Indexes might become unusable for the table move command, it is needed to check the Indexes. Rebuild Index if needed alter index <owner>.<index_name>
rebuild online;
Note: Ensure the tablespace in which the object currently exists has got sufficient space before you move or defragment. Always make a backup before moving the data. It is recommended to test these steps before going into PRODUCTION !
You will need to collect the statistics for the tables. For example: exec fnd_stats.gather_table_stats ('APPLSYS','FND_CONCURRENT_REQUESTS',PERCENT=>99);
30

Gather Schema Statistics to improve performance overall
At regular interval, run Gather Schema Statistics - use below parameter to speed up the request. Schema Name ALL Estimate Percent 10-40 Backup Flag NOBACKUP History Mode NONE Gather Options GATHER AUTO
31

Specialized / Dedicated Concurrent Managers
Create specialized concurrent managers which is dedicated to some specific concurrent requests which need to be processed in a different way.
Benefits 1. Speed up Critical Short Running Requests which has long wait time 2. To avoid Long Running Request during Peak time
Tip Avoid creating an excessive specialized managers. It can degrade the performance due polling on queue tables. You need to create specialized managers only if there is a real need.
32

1. Speed up Critical Short Running Requests
Assume you have a Business Critical "Short Running" (ex. Invoice Printing) Concurrent Request, but it is waiting for the Standard Manager for a long time before the Processing is getting picked up. You can then define a new Specialized Concurrent Manager. And include the specific critical short run Concurrent Request to the new Manager that you created. This will ensure all your critical short run Concurrent Requests are not waiting in the queue for longer time.
33
34
35

2. Avoid Long Running Request during Peak time
This helps you to prevents users from executing such Long Running Request during Peak time. This improves overall performance of the system.
Create a Specialized Manager to process the specific 'Long Running' requests.

Assign the Specialized Manager with a work shift with Off peak time. Assume, the Users submit the Concurrent Requests during peak time, then it will get assigned to the Specialized Manager. But Specialized Manager will not process it during peak time.
36

Output Post Processor - OPP
Concurrent Processing uses the Output Post Processor (OPP) to enforce Post Processing
actions for concurrent requests. Tip : Ensure that enough OPP process existing to serve the incoming request load. Otherwise increase process. Thread Vs Process You can either increase process (or) threads. It depends on the load . Concurrent -> Manager -> Define > "Output Post Processor oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=5 Each PCP node should have one OPP defined. Refer to Note 563233.1 for more details.
37

Increase Number of OPP processes
OR
Increase the number of Threads
38

Heap Size
To prevent the error "java.lang.OutOfMemoryError: Java heap space check the current value :
SELECT service_id, service_handle, developer_parameters FROM fnd_cp_services WHERE service_id = (SELECT manager_type FROM fnd_concurrent_queues WHERE concurrent_queue_name = 'FNDCPOPP');
Increase the maximum Java heap size for the OPP to 1024MB (1GB):
UPDATE fnd_cp_services SET developer_parameters = 'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m' WHERE service_id = (SELECT manager_type FROM fnd_concurrent_queues WHERE concurrent_queue_name = 'FNDCPOPP');
39

Bi Publisher Enhancement Parameters Use XML Publisher's XSLT processor set to True Enable XSLT runtime optimization set to True Enable scalable feature of XSLT processor set to False. Small Report - It is still recommended this remain false at Site level. This has the advantage of faster performance by not using temporary files for XSLT processing for small to medium reports. Large Reports - with Patch 7599031,can be safely set to True at the data definition level, enabling improved JAVA heap utilization along with ScalableFlag parameter and 'Options' field with the value to -Xmx1024M (or higher). Refer to Note 737311.1 How to Configure the Account Analysis Report in Release 12 for Large Reports
40
41
42
43
RAC & Parallel Concurrent Processing

Set profile option 'Concurrent: PCP Instance Check' to 'OFF' if Instancesensitive failover is not required. If the failover of Managers is taking too long, then refer Note:551895.1 Failover of Concurrent Manager Processes takes more than 30 Minutes.
Run a concurrent program against a specific RAC Instance with PCP/RAC setup. (Instance Affinity)
Note 1129203.1 - How to run a concurrent program against a specific RAC Instance with PCP/RAC setup ?
44
RAC & Parallel Concurrent Processing
45
Transaction Manager (TM)

Ensure enough number of TMs exist to serve the incoming request load. Profile Concurrent: Wait for Available TM - Total time to wait for a TM before switchover to next available TM. To improve performance , try setting this to 1- 2. In case all TM are busy, then increase number of TM. Use Advance Queue(AQ) with ATG RUP3 by setting System Profile Concurrent: TM Transport Type' to QUEUE'
Pipes Vs Queue Pipes are more efficient than Queue. To use Pipes, it requires Transaction Manager defined for every node. However you might want to use "Queue" for easy maintenance.
46
Program Agenda
Specialized Concurrent Manager


Purging Concurrent Records Running cmclean.sql OPP and TM related issues.
Resolving Concurrent Requests

47
Script cmclean.sql
Why to run CMCLEAN.sql ? Cleaning out the tables is a useful method of making sure that there are no invalid statuses that can prevent the managers from starting. See Note 134007.1 Concurrent Processing - CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables. It will make sure the relevant status codes are valid without deleting any information. Tip Doc ID 152209.1 What is the Meaning of the Codes in the STATUS_CODE and PHASE_CODE Columns of FND_CONCURRENT_REQUESTS Table? Doc ID 155925.1 How to Clean Out Concurrent Manager Tables?
48

Typical issues during Purging Concurrent Requests and resolution Log/out files are not removed from OS file system
Performance issue Apply Patch to fix above issues.

Purge Concurrent Request Hangs post cloning. Update FND Table.
49

Release 11i Apply Patch 7834670 (on top of 11.5.10.2) Ensure afppur.lpc is in Release 115.36 available Set the system profile "Concurrent: Force Local Output File Mode Refer to Note 822368.1 for more details. Most of your issues should get fixed with Patch 7834670. To set the Profile Option "Concurrent: Force Local Output File Mode to "Yes, It is needed that ALL Concurrent Manager nodes must be able to access the output file locationvia the local filesystem.
50

Release 12 - Patch 7530490 Ensure afppur.lpc is higher than in Release 120.4.12000000.4 available. Release 12.1 - Patch 10106197 Ensure afppur.lpc is in Release 120.6.12010000.8 available. Version details can be identified using this command
strings -a $FND_TOP/bin/FNDCPPUR|grep 'Header'|egrep 'afppur.lpc'
51
Purging Requests hangs after Cloning

Cloning is not updating the host- and Instance specific information used by Oracle Workflow Review the following tables of the Target Instance: FND_CONCURRENT_REQUESTS
Update LOGFILE_NAME with the correct path to the logfile directory.
FND_CONCURRENT_REQUESTS
Update OUTFILE_NAME with the new directory path on the Target System.
For more Information, please see the Cloning documentation :

Release 12 : Note 406982.1 Cloning Oracle Applications Release 12 with Rapid Clone Release 11i : Note 230672.1 Cloning Oracle Applications Release 11i with Rapid Clone
52
Pending Requests
Pending / Normal The request is waiting for the next available concurrent manager to be picked and it will get processed. Increase number processes of the Concurrent Manager.
Pending / Standby
The Request is waiting at CRM due to Conflict Request which is already being run. Check the Incompatibilities defined for the request. Check any request is running with Run Alone flag.
53
Concurrent Request Lifecycle
54
Incompatible Programs
Program A Program B Program A Program B
Domain 1
Domain 2
Logical database
Logical database
Conflict Domains is to prevent two programs from concurrently
accessing or updating the same data.

55
56

Running / Normal If the request is taking longer time, enable Database Trace and tkprof and find out expensive DML. Steps given below. Completed / Error Check the request log file and take action accordingly Inactive / No Manager If Standard Manager has Specialization Rules defined which has rows with Include. You must only define exclude. If you add any program with Include options, other programs get excluded automatically.
57
58
Program Agenda
Specialized Concurrent Manager


Purging Concurrent Records Running cmclean.sql OPP and TM related issues.
Resolving Concurrent Requests

59
Troubleshooting Concurrent Manager
How To Enable Tracing ( Request Running/Normal)
Output Post Processor -OPP Timeout

How to Report Performance Issues
60
How to Enable the Tracing

Enable Tracing On Program Level (Not suggested) System Administrator Responsibility Navigation: Concurrent -> Program -> Define
Click the Enable Trace button

Note:- This generates trace for all the request submitted by all end- users. Hence this could generate lots of trace files which is an additional overhead and degrade performance.
61
Enable the trace on Program level
62
How to Enable the Tracing

Enable Tracing On Request Level (Recommended)
Responsibility: System Administrator Navigate: Profiles > System Query Profile Option Concurrent: Allow Debugging
Set profile to Yes

Logon to the Responsibility that runs the Concurrent Program In the Submit Request Screen click on Debug Options (B) Select the Checkbox for SQL Trace and select desired Trace level.
Submit the Request.
63
Enable the trace on Request level
64
How to enable a trace (continued)

Once you have obtained the Raw trace file you need to format the file using TKPROF :
tkprof raw_trace_file.trc output_file explain=apps/<passwd> \ sort=(exeela,fchela, prsela)
If you do not know the path for those directories, you can run a command like the following query in SQL : select name, value from v$parameter where name = 'user_dump_dest';
65

Error
The Output Post-processor is running but has not picked up this request. No further
attempts will be made to post-process this request, and the request will be marked with Warning status.
Action
Setting the Profile Option Concurrent: OPP Response Timeout to a higher value may be necessary.
If the value of the Profile Option Concurrent:OPP Response Timeout is 120 then
increase it to 240. (Value * 2). Increase the number of processes or threads (or both) .
66

Error
The Concurrent Manager has timed out waiting for the Output Post Processor
to finish this Request. Check that there are enough Output Post Processor Service Processes running. More information may be found in the Service Process logfile.
Action Check the Profile Option "Concurrent:OPP Process Timeout". Increase the setting to a higher value. For example: Set the value to 10800 sec (3 hours). After this change, the Concurrent Request will wait maximum 3 hours, for the Output Post Processor to finish the Request.
67
Logging a SR for Concurrent Processing Issues

Please let Support know following details : Was there any recent change(s) Describe if requests stay in Pending (or) Running phase for longer time Is it specific to some Concurrent Programs (or) all ? Was CMCLEAN.sql script run ? (Note 134007.1 ) Run CCM.sql and upload the output (Note 171855.1 ) Provide total number of records in the below tables FND_CONCURRENT_REQUESTS FND_CONCURRENT_PROCESSES FND_CRM_HISTORY FND_ENV_CONTEXT FND_TEMP_FILES
68
Logging a SR for Concurrent Processing Issues

Upload Concurrent Request log, if its specific to a program Upload Database Trace with tkprof output Upload ICM (or) Standard manager log relevant Backend queue concurrent manager details
Select rpad(user_concurrent_queue_name ,50,'-') "Queue Name",TARGET_NODE, MAX_PROCESSES,running_processes, nvl(to_char(SLEEP_SECONDS), 'Not Set') Sleep_Second, nvl(to_char(cache_size),'Not Set')Cache_size from FND_CONCURRENT_QUEUES_VL fcq order by 1;
69
For More Information

1. Best Practices for Performance for Concurrent Managers in E-Business Suite (Doc ID 1057802.1) 2. Product Information Center (PIC) (Doc ID 1304305.1) 3. CCM.sql Diagnostic Script to Diagnose Common Concurrent Manager Issues (Doc ID 171855.1 ) 4. CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables (Doc ID 134007.1) 5. E-Business Suite Diagnostics References for R12 (Doc ID 421245.1)
70
Applications Technology Group Webcasts

November 2011 : None planned at this stage
Additional Webcasts are planned they will be promoted soon.

For complete details on all upcoming Oracle Advisor Webcast Events, please see Note 740966.1, Oracle Advisor Webcast Schedule. For EBS Technology Specific Webcasts please check Note 1186338.1. Do you have any requests for future ATG Advisor Webcast Events ? Please email your suggestions to me : ruediger.ziegler@oracle.com, subject: Topics of Interest.
71
Communities in My Oracle Support
72
ATG Communities in MOS available

Following Communities are available in My Oracle Support
Oracle E-Business Suite ... BI Publisher - Business Intelligence Products (not only EBS) Core Concurrent Processing - Anything around Concurrent Processing and Concurrent Managers Core Workflow - Any Workflow isse not only E-Business Suite Diagnostic Tools - Anything around EBS Diagnostics E-Business Customizations - Your Customizations Installation - Fresh Install of the E-Business Suite Patch Review EBS - Review of Patches around the E-Business Suite Performance - EBS Performance Upgrade - EBS Upgrade User Produktivity Kit - User Productivity Kit (UPK) available for the E-Business Suite Utilities - Utilities / generic EBS DBA issues ... This is the current list for the E-Business Suite Applications Technology Group
73
EBS ATG Product Information Center Note 1160285.1
74
Q&A
75
THANK YOU
for attending our Advisor Webcast !
76
77
78

2011 10 EBS Concurrent Processing Best Practices

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

2011 10 EBS Concurrent Processing Best Practices

Transféré par

Droits d'auteur :

Formats disponibles

1

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Upcoming Webcasts in EBS Technology area :

North America: 1866 966 9439

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Q&A Session maximum 15 minutes

type your question here

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

your question pops-up here

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

ATTENTION AUDIO INFORMATION

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Safe Harbor Statement

It is intended for information purposes only, and may not be

purchasing decisions. The development, release, and timing of any

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

EBS Concurrent Manager Performance - Best Practices

Specialized Concurrent manager

Common / Typical issues and resolution

Resolving concurrent requests

Trouble-shooting Concurrent Manager

Specialized Concurrent manager

Common / Typical issues and resolution

Resolving concurrent requests

Trouble-shooting Concurrent Manager

EBS Release 12 Architecture

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager - Introduction

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager - Introduction

Are the Managers up and running ?

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager - Introduction

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Concurrent Requests, Programs, and Processes

User requests Concurrent Program

Concurrent Managers Read applicable Requests and start Concurrent programs

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager - Tables

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Concurrent Request - Lifecycle

The request is waiting at CRM

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Specialized Concurrent manager

Common / Typical issues and resolution

Resolving concurrent requests

Trouble-shooting Concurrent Manager

Enhancing Performance - Best Practices

Java Heap Size

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Tuning Concurrent Queue Parameters

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Tuning Sleep - Parameters

Reduce the Sleep Time

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Tuning Concurrent Queue Parameters

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Tuning Parameters Cache

Increase Cache Size

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Tuning Concurrent Queue Parameters

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Tuning Parameters Processes

Increase the Number of Processes