Gather stats fails after upgrade to Oracle 10G

Gather Schema Statistics or Gather Table Statistics or running FND_STATS fails with "ORA-20005: object
statistics are locked".

Example:

ERROR: While GATHER_TABLE_STATS:

object_name=AR.AR_REV_REC_QT***ORA-20005: object statistics are locked (stattype = ALL)***

This can happen with Advance Queue tables.

In 10g, if a queue is created or recreated during the upgrade, automatic statistics gather gets locked (disabled) on
these queue.
The following statement can be used to check the tables which have statistics locked:

select owner, table_name, stattype_locked

from dba_tab_statistics
where stattype_locked is not null;

SOLUTION:

Unlock gather stats running on those queues :

exec dbms_stats.unlock_schema_stats('schema_owner');
exec dbms_stats.unlock_table_stats('table_owner','table_name');

Example:
SQL> exec dbms_stats.unlock_schema_stats ('AR');
SQL> exec dbms_stats.unlock_table_stats('AR', 'AR_REV_REC_QT');

How to resolve cloning issue: BEGIN failed--compilation

aborted at adcfgclone.pl line 26.
While running adcfgclone.pl as a part of database Tier clone, the below mentioned error occurs:

-bash-3.00$ perl adcfgclone.pl dbTechStack

Can't locate strict.pm in @INC (@INC contains:

/ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/5.8.3/i686-linux-thread-multi
/ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/5.8.3
/ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/site_perl/5.8.3/i686-linux-thread-multi
/ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/site_perl/5.8.3
/ade/smayer_perl58_main_linux/perl58/bin/Linux/Opt/lib/site_perl .) at adcfgclone.pl line 26.
BEGIN failed--compilation aborted at adcfgclone.pl line 26.

As indicated in the above error message, this is due to the PERL setting. We dint have the PERL5LIB set :

-bash-3.00$ echo $PERL5LIB

-bash-3.00$ which perl

/d02/oradev/devdb/10.2.0/perl/bin/perl

Resolution: $PERL5LIB was set to resolve the issue:

Contents under $ORACLE_HOME/perl :

-bash-3.00$ ls -ltr /d02/oradev/devdb/10.2.0/perl

total 16
drwxr-x--- 2 oradev dba 4096 Feb 9 04:33 include
drwxr-x--- 2 oradev dba 4096 Feb 9 04:33 bin
drwxr-x--- 4 oradev dba 4096 Feb 9 04:33 lib
drwxr-x--- 4 oradev dba 4096 Feb 9 04:33 man

Contents under $ORACLE_HOME/perl/lib:

-bash-3.00$ ls -ltr /d02/oradev/devdb/10.2.0/perl/lib

total 8
drwxr-x--- 3 oradev dba 4096 Feb 9 04:33 site_perl
drwxr-x--- 44 oradev dba 4096 Feb 9 04:33 5.8.3

-bash-3.00$ PERL5LIB=/d02/oradev/devdb/10.2.0/perl/lib/5.8.3
-bash-3.00$ export PERL5LIB
-bash-3.00$ id
uid=500(oradev) gid=502(dba) groups=502(dba)

-bash-3.00$ echo $PERL5LIB

/d02/oradev/devdb/10.2.0/perl/lib/5.8.3

Issue running adcfgclone.pl on Database Oracle home.

While running adcfgclone.pl as oracle user, the session just hangs without giving any error in the log
file. The issue was that it wasn’t able to connect to the database as:
sqlplus "/as sysdba"

Deep dive:

Most generally this can occur due to improper shutdown of the database. While in the course of a database
shutdown, if any of the oracle process is not completed killed and it holds a large shared memory segment,
such issues can occur which would prevent from normal database operation.

So next time when an instance tries to start it is not able to grab a large chunk of memory to get started.

This can be checked as follows:

After the database has been shut down , there shouldn’t be any processes running as oracle on the server level:

ipcs -m | grep <oracle user>

eg:

ipcs -m | grep oracle

key shmid owner perms bytes nattch status

0x185ee230 884763 oracle 640 789785472 0

Here in the above example, a single oracle process holds 789785472 bytes of shared memory, even after
shutdown. If the database instance is started, then we would expect abnormal behaviors on the database side,
like the login issue that we are discussing now.
This can cleaned as mentioned below:
ipcrm -m < shmid>

eg:

ipcrm -m 884763

This would free up the memory held.

This resolved the issue.

Issue starting 10g Forms and reports server

Error:

Ping URL for proc: 887947305 failed while pinging url:/reports/rwservlet/pingserver?start=auto

12/11/01 08:18:46 [4] [libopmnoc4j] Process Ping Failed: OC4J~OC4J_BI_Forms~default_island~1

(887947305:8511)

12/11/01 08:19:10 [2] [libopmnoc4j] AJP13: Receiving response code: 500 while configured success code is:
200

12/11/01 08:19:10 [2] [libopmnoc4j] Ping URL for proc: 887947305 failed while
pingingurl:/reports/rwservlet/pingserver?start=auto

Search String:

REP-56105

tail –f OC4J~OC4J_BI_Forms~default_island~1
ConfigFileName: /u01/oracle/product/ais10g/forms90/server/forms/server/formsweb.cfg

REP-56105: Engine rwEng-0 died with error

SOLUTION:

Bug no: 3557723

Work Around:

Disable pingserver command in $ORACLE_HOME/opmn/conf/opmn.xml

- Take the backup of opmn.xml file

@ cd $ORACLE_HOME/opmn/conf

@ cp opmn.xml opmn.xml.PINGAUTO

@ - Open file opmn.xml

@ - Take out the following lines from opmn.xml

@ < category id="urlping-parameters">

@ < data id="/reports/rwservlet/pingserver?start=auto" value="200"/>

@ < /category>

@ - Bounce the opmn

@ cd $ORACLE_HOME/opmn/bin

@ ./opmnctl stopall ; ./opmnctl startall

Error: CACHE mismatching

SOLUTION:
Also Check for the domain name in /u01/oracle/product/ais10g/webcache

Webcache.xml ----change the domain name.

Issue: Concurrent request running from a very long time

.
Application version: 11.5.10.2 multi-node (4 nodes)
Database: 11g RAC

Scenario:

Business critical concurrent request is running from more than 2 hours which was supposed
to complete within 1 hour.
The standard manager and the manager responsible to execute the critical concurrent
request (custom manager) were overloaded with many requests. The server on which these
managers run had very high server load and all CPU’s were utilized.

After diagnosis, we had to kill few requests which were running from more than 2 hours and
were creating load on the server. These requests were in fact not performing anything and
had gone zombie.

The server load came down and CPU’s available were set free to execute new
requests/processes.

However the business critical request was still running and wasn’t moving ahead at all. The
database session associated was also not executing anything.

Challenge:

Since this being a critical business request, we couldn’t terminate the request coz we had
already lost few hours running it. At the same time the database session also wasn’t
performing any execution and the whole request was not going anywhere even though it
showed running normal status.

Trick:

Not recommended, however can be a life saver at times.

---- Forcibly change the status of the request to pending normal.

---- Kill the database session of the request.

Use the below scripts:

update applsys.fnd_concurrent_requests set phase_code = 'P',
status_code = 'I',
actual_start_date = null,
crm_release_date = null,
controlling_manager=null,
logfile_name = null,
logfile_node_name = null,
outfile_name = null,
outfile_node_name = null,
crm_tstmp = null
where request_id in ('request-id');

alter system kill session 'sid, serial#, @instance-id';

Hence a new database session will get created as soon as the old one is killed and the
request starts running. Since the server on which the request is supposed to get processed
also had no issues now, the request went through very fast without any issues ;)

Login issues in Oracle Apps R12

Application version: 12.0.4 running on 11g RAC database.

None of the users are able to login to the Oracle Apps R12 URL. The URL just hangs without giving any
information.

All the application services were up. Couldn’t find any significant load on the CPU and the number of concurrent
requests running at the time was normal.

The http error_log had the below error:

oc4j_socket_recvfull timed out

The Application log has the below error:

12/04/26 03:15:13.511 html: Servlet error

java.lang.OutOfMemoryError: Java heap space
12/04/26 03:50:35.503 html: Servlet error

The opmn log had the below error:

12/04/26 03:15:00 java.lang.OutOfMemoryError: Java heap space

12/04/26 03:15:02 java.lang.OutOfMemoryError: Java heap space
12/04/26 03:15:06 java.lang.OutOfMemoryError: Java heap space
12/04/26 03:15:13 Exception in thread "Timer-1" java.lang.OutOfMemoryError: Java heap space
oracle.jbo.DMLException: JBO-26066: Error during rollback

Solution:

We would need to do changes to -Xms ( for start heap memory ) and -Xmx ( for maximum heap memory
) of the opmn.xml. Changing the Xmx to 512 MB and Xms to 128 MB will help.

>>>> Increase Java Heap Size for the OC4J instance as per the available memory (RAM) & restart the
instance.
You can do it with AS Control -> OC4J Instance -> Administration Tab -> Server Properties -> Java options.
Update -Xms -Xmx java options. Once updated, stop and start the OC4J instance to pick up the changes.

Login issues in Oracle EBS 11i after Clone ( from

hotbackup)

Error in login page:

ORA-01110: data file 98: '/disk09_250/oradata/mine/a_txn_data14.dbf'

ORA-26040: Data block was loaded using the NOLOGGING option

SOLUTION:

1. Compile jsp:
cd $FND_TOP/patch/115/bin
perl ojspCompile.pl --compile --flush

2. Access the form directly using the URL:

http://sanop.mine.com:8020/forms/frmservlet?form_params=config=PRODMINE

3. Run the request:

"Synchronize WF LOCAL tables" - Parameters: ALL
"Synchronize WF LOCAL tables" - Parameters: FND_USER
OACORE fails to start after increasing the JVM processes
in Oracle R12

Issue: OACORE processes are not starting.

Recent changes: Increased the JVM processes in the context file.

Error:

adoacorectl.sh: exiting with status 206

Solution:

1) Stop the middle tier processes:

adopmnctl.sh stop

2) Kill any runaway/zombie processes.

3) Remove the following jms lock files:

$ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence/forms_default_group_1/jms.state.lock,

$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence/oacore_default_group_1/jms.state.lock

$ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence/oafm_default_group_1jms.state.lock

4) Restart middle tier:

adopmnctl.sh start

Restore FNDLIBR executable

Restore FNDLIBR executable which has been deleted accidentally:

1. Invoke adadmin on concurrent manager node

2. Select option 2. Maintain Applications Files menu
3. Select 1. Relink Applications programs
4. Enter FND when it prompts for
Enter list of products to link ('all' for all products) [all]
5. When prompted for
Generate specific executable for each selected product [No] ? YES
select YES
& from list of executable select FNDLIBR.

This will create new FNDLIBR executable.

Move Concurrent Processing Server from one node to

another in Oracle Apps 11i
Example:

Oracle E-Business Suite (11.5.10.2) with 2 nodes:

Node 1: Database and the Concurrent Processing

Node 2: Forms, Reports and Web Server.

Goal: To Move the concurrent processing from Node1 to node 2.

Requirement: Shared appl_top / Shared application tier file system. If not follow the Step 2 below.

How to perform:

1. This is recommended to be performed on a NON-PRODUCTION system first with necessary backups.

2. Create a shared APPL_TOP by following Metalink Note 233428.1 for sharing the application tier files
system and allow the node 2 to share for concurrent manager node ( Node 1).

3. Edit context file on the Forms and Web node2 and change following context variables:

s_isConc from "NO" to " YES "

s_concstatus from "disabled" to " enabled "

On Node1: make the below changes

s_isConc from "YES" to "NO"

s_concstatus from "enabled" to "disabled"

4. Run autoconfig on all the APPS nodes.

 Concurrent managers are down with status: “System Hold,
Fix Manager”

Issue:

All the concurrent managers are down with a status message: system hold, fix manager.

The administer Concurrent manager screen has Actual: 0 and Target: 1 for all managers.

Solution: Ensure all application services are down.

- Please check the system profile: Concurrent: GSM Enabled. This should be set to ‘Y’

- Execute: afdcm037.sql Path : $FND_TOP/patch/115/sql

Description of the script:

PL/SQL script to create libraries for FNDSM and FNDSVC and create Managers for Preexisting Nodes. Future
nodes will be handled by db trigger.

- Relink the below mentioned executables:

adrelink.sh force=y “fnd FNDLIBR”

adrelink.sh force=y “fnd FNDSM”
adrelink.sh force=y “fnd FNDFS”
adrelink.sh force=y “fnd FNDCRM”

- Execute cmclean.sql to clean out the concurrent manager tables.

- Start all the application services and restest the issue.

ORA-00257: archiver error. Connect internal only, until

freed
While trying to connect to the database, we get the below mentioned error:

ERROR:
 ORA-00257: archiver error. Connect internal only, until freed.

Cause:

- The error is caused by the fact that the database archiver process is unable to archive the current online redolog
due to lack of space in the destination for the archivelogs.

- The flash_recovery_area is full.

Applicable to 9i, 10g and 11g database versions.

Solution:

The basic idea is to free space so as to make room for new archiving to happen. This can be addressed in
multiple ways.

- Allow more space in the DB_RECOVERY_FILE_DEST with the DB_RECOVERY_FILE_DEST_SIZE parameter

:

alter system set db_recovery_file_dest_size=5G ;

- Imagine a situation where the space increased in step 1 gets filled, ie, 5 GB. Set an alternative archiving
destination, so that when the primary destination is full, archiving is automatically performed to the alternate
destination:

Following changes in parameter file can enable the same:

log_archive_dest_1='LOCATION=use_db_recovery_file_dest NOREOPEN
ALTERNATE=LOG_ARCHIVE_DEST_2'
log_archive_dest_2='LOCATION=/other_dest_for_archiving'
log_archive_dest_state_1='enable'
log_archive_dest_state_2='alternate'
db_recovery_file_dest='/db01/mine/oracle/product/10.1.0/db_1/flash_recovery_area'
db_recovery_file_dest_size=5G

- Check the archive_log_dest and verify if the path mentioned exists and is valid.

- Take a backup of the existing archive files (recommended) OR it can be deleted to make more free archiving
space.

This can be done at the server level where files can be manually deleted. However incase of ASM storage with
rman, below mentioned way is recommended (which would perform backup and then deleting):
 Example :

rman target sys/sys

RMAN> backup archive log all ;
RMAN> delete archive until time 'trunc(sysdate)';

Unable to authenticate session in Oracle apps R12

(12.0.4): Login Issue
Unable to authenticate session in Oracle apps R12 (12.0.4): Login Issue

The Release 12.0.4 instance was cloned from another identical instance.

When we try to login, the login page does not display

Have tried clearing the cache (_pages). Have tried recompiling jsps and bouncing however the problem still persists.

Cleared all the Apache and related logs and found an interesting error.

It is in the following log file:

$INST_TOP/logs/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log

08/08/21 15:17:00.601 10.1.3.0.0 Started

08/08/21 15:17:02.769 html: 10.1.3.0.0 Started
08/08/21 15:17:33.132 html: oracle.apps.fnd.security.LeakDetectionFilter.initialized:1219349853132
08/08/21 15:17:33.132 html: LeakSetting:
global:true,session:false,aggressive:false,stderr:false,appslog:false,corelog:false
ignoring:null
08/08/21 15:17:34.859 html: chain failed
javax.servlet.ServletException: java.lang.RuntimeException: Unable to create user session. Please contact
your System Administrator.
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0)
].server.http.EvermindPageContext.handlePageThrowable(EvermindPageContext.java:780)

Solution:

Enable FND debug

The FND debug log had the below error:

Oracle error -1654: ORA-01654: unable to extend index APPLSYS.FND_LOGINS_N2 by 16 in tablespace
APPS_TS_ARCHIVE has been detected in FND_SIGNON.AUDIT_USER.

Check if the APPS_TS_ARCHIVE Tablespace is full.

Increase the size of tablespace APPS_TS_ARCHIVE / add a datafile to the tablespace.

This will resolve the issue.

ICX: Session Timeout and autoconfig variable

s_oc4j_sesstimeout in 12.0.4

Recently we came across a forms time out issue. All the user forms sessions were getting disconnected after an
Hour. The FORMS_TIMEOUT variable was set to 120 mins though. This should practically disconnect the forms
session after 2 hours.

Here the reason for the issue is because of the mismatch of the ICX and OC4J timeout session parameters.
OC4J session timeout, if not set, defaults to 20 minutes. This will cause the oc4j session to timeout
before the Apps/ Forms session timeout which is defined by profile option "ICX: Session Timeout"

Profile option "ICX: Session Timeout" and autoconfig variable "s_oc4j_sesstimeout" in context file should
match.

Example:
In case you need the session to timeout after 2 hrs then the profile option "ICX: Session Timeout" and
autoconfig variable "s_oc4j_sesstimeout" should be set to 120.

Unable to apply RDBMS patch (Opatch) on 10G ENV after

postclone and upgrade from 9i.
UNABLE TO APPLY RDBMS PATCH (Opatch) ON 10G ENV after postclone and upgrade from 9i.

SOLUTION:
Some options of opatch:
Opatch apply –invPtrLoc
Opatch apply –no_inventory
Which opatch
~/Oraclemine/10.2.0/OPatch/opatch
(Oracle06) 5005469 - -bash $ opatch apply -invPtrLoc $ORACLE_HOME/oraInst.loc
Invoking OPatch 10.2.0.2.0
. Oracle interim Patch Installer version 10.2.0.2.0
@ Copyright (c) 2005, Oracle Corporation. All rights reserved..
. Oracle Home: /mine/oracle/oraclemine/10.2.0
Central Inventory: /mine/oracle/oraclemine/10.2.0/oraInventory
from : /mine/oracle/oraclemine/10.2.0/oraInst.loc
OPatch version: 10.2.0.2.0
OUI version: 10.2.0.2.0
OUI location: /mine/oracle/oraclemine/10.2.0/oui
Log file location:
/mine/oracle/oraclemine/10.2.0/cfgtoollogs/opatch/opatch-2006_Apr_25_21-58-34-PDT_Tue.log
. ApplySession applying interim patch '5005469' to OH/mine/oracle/oraclemine/10.2.0'
List of Homes on this system:
OPatchSession cannot load inventory for the given Oracle Home
/mine/oracle/oraclemine/10.2.0. Possible causes are:
No read or write permission to ORACLE_HOME/.patch_storage
Central Inventory is locked by another OUI instance
No read permission to Central Inventory
The lock file exists in ORACLE_HOME/.patch_storage
The Oracle Home does not exist in Central Inventory
.
Apply Session failed: ApplySession failed to prepare the system. No detail available.
System intact, OPatch will not attempt to restore the system
.
OPatch failed with error code 73
===========================================================================
I have checked that patch_lock file is not available on $ORACLE_HOME/.patch_storage and no lock on
Inventory. Also required permissions are exisitng for .patch_storage and oraInventory. .
lsinventory gives the following info
===========================================================================
(oracle06) 5005469 - -bash $ opatch lsinventory -invPtrLoc
$ORACLE_HOME/oraInst.loc
Invoking OPatch 10.2.0.2.0
. Oracle interim Patch Installer version 10.2.0.2.0
@ Copyright (c) 2005, Oracle Corporation. All rights reserved..
.Oracle Home : /mine/oracle/oraclemine/10.2.0
Central Inventory : / mine /oracle/ oraclemine /10.2.0/oraInventory
from : / mine /oracle/ oraclemine /10.2.0/oraInst.loc
OPatch version : 10.2.0.2.0
OUI version : 10.2.0.2.0
OUI location : / mine /oracle/ oraclemine /10.2.0/oui
Log file location :
/ mine /oracle/ oraclemine /10.2.0/cfgtoollogs/opatch/opatch-2006_Apr_25_22-00-16-PDT_Tue.log .
List of Homes on this system:
Inventory load failed... OPatch cannot load inventory for the given Oracle
Home.
Possible causes are:
 Oracle Home dir. path does not exist in Central Inventory
Oracle Home is a symbolic link
Oracle Home inventory is corrupted
LsInventory Session failed: OracleHomeInventory::load() gets null oracleHomeInfo .OPatch failed with
error code 72
===========================================================================
Looks like $ORACLE_HOME is not attached to the local inventory. Attaching the ORACLE_HOME to the
inventory.
Run runClone.sh as mentioned under "Run 10g Cloning" step . This will resolve the issue

How to recover a cloned database.

Source Instance:

oracle@mine01$ sqlplus "/as sysdba"

SQL*Plus: Release 9.2.0.7.0 - Production on Sat Apr 21 03:26:33 2007

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
Connected to:
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 – Production

SQL> !pwd
/d02/oracle/proddb/9.2.0/admin/PROD_mine01/udump

SQL> alter session set tracefile_identifier=mine; ( this would set the trace file name as mine)

Session altered.

SQL> alter database backup controlfile to trace;

Database altered.

oracle@mine01$ ls -ltr *MINE*

-rw-rw-r-- 1 oracle dba 67001 Apr 21 03:28 prod_ora_23305_MINE.trc

uuencode prod_ora_23305_MINE.trc prod_ora_23305_MINE.trc|mailxsandesh.achar@mine.com ( This would send
an email to your mailbox with the trace file attached)

Get this controlfile to target instance.

Target Instance:

oradev@ mine02# ls -ltr *MINE*

-rwxrwxrwx 1 oradev dba 67001 Apr 21 03:29 prod_ora_23305_MINE.trc

oradev@mdsuaor09# cp prod_ora_23305_MINE.trc mine.sql

This would copy the trace file to a new file named mine.sql and all the required changes can be done to mine.sql as
mentioned below:

vi mine.sql

esc:se nu

check where the create controlfile line starts ( double copy of create control file line)and delete till there.

esc:1,60d

esc shift+g -------will take to last line

esc:?CREATE --------press n and check if there are 2 sets of create controlfile

if yes then bring the cursor to the last line of the wanted set

escd shift+g

change like this :

CREATE CONTROLFILE SET DATABASE "DEV" RESETLOGS NOARCHIVELOG

Now change the path of data files and save the file .
esc:%s,oracle/proddata,oradev/devdata,g

The entire practice above is to create a new set of controlfile and below mentioned steps would check if all datafiles
have been copied properly:

oradev@ mine02# grep / mine.sql > mine.sh

vi mine.sh

esc:%s,',,g

esc:%s/,//g

esc:%s,^,ls -ltr ,g

save the file .

chmod +x mine.sh
./mine.sh 2> mine.log

check the log file and see if there any data file missing.( mine.log file size shud be zero )

At this point we are ready with the control file and we are also sure that all data files have been copied to the target
instance.

oradev@ mine02# sqlplus "/as sysdba"

SQL*Plus: Release 9.2.0.7.0 - Production on Sat Apr 21 03:59:48 2007

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
Connected to an idle instance.

SQL> startup nomount;

Now open another session and keep monitoring the alert log file.

SQL> @mine.sql
Control file created.

SQL>recover database using backup controlfile;

Meanwhile go to the source instance and go to alert log file and do a tail -500 and check for the path of the log file
using the sequence number asked for recovery.

Change the path accordingly and give it for the recovery.

SQL> recover database using backup controlfile;

ORA-00279: change 7254772161098 generated at 04/20/2007 14:08:25 needed for
thread 1
ORA-00289: suggestion : /d02/oradev/devdb/9.2.0/dbs/arch1_88922.dbf
ORA-00280: change 7254772161098 for thread 1 is in sequence #88922

Specify log: {<RET>=suggested | filename | AUTO | CANCEL}

/d04/oradev/devdata/log4a.dbf
Log applied.
Media recovery complete.

( In this case the sequence number is 88922, hence check in source instance for the log file path for this
sequence number . This will be there in alertlog file. Then change the path of log file according to the target
instance and give it. Sometimes this will ask for multiple files with different sequence number)

SQL> select * from v$recover_file;

no rows selected

SQL> alter database open resetlogs;

Database altered.

SQL> select * from v$tempfile;

no rows selected

Check the alert log file while performing the below task.
There should be no errors.

SQL> shutdown immediate;

Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
oradev@ mine02# sqlplus "/as sysdba"

SQL*Plus: Release 9.2.0.7.0 - Production on Sat Apr 21 04:26:54 2007

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 1487899976 bytes

Fixed Size 732488 bytes
Variable Size 973078528 bytes
Database Buffers 503316480 bytes
Redo Buffers 10772480 bytes
Database mounted.
Database opened.
SQL> exit

At this point there shouldn't be any errors in the alertlog.

Start the listener:

oradev@ mine02# lsnrctl start DEV

Then Drop temp tablespace and create new temp tablespace. Then add temp files into it.

Change sys and system passwords if required.

OPMN and process monitoring in 10G Application server

 OPMN and process monitoring

Oracle Process Manager (PM) is the centralized process management mechanism in Oracle Application Server and
is used to manage Oracle AS processes. It starts, stops, restarts and detects death of these processes. The Oracle
AS processes that PM is configured to manage are specified in the opmn.xml file. The start and stop functions are
executed by the OPMN as per the user commands.

OPMN (Oracle Process monitoring and Notification services ) are a set of processes that manage mid-tier
Application Server components like Oracle HTTP Server (Apache) and OC4J containers.
OPMN consists of the Process Manager and the Notification Server.

Oracle Notification Server (ONS) is the transport mechanism for failure, recovery, startup and other related
notifications between components in Oracle Application Server. It operates according to a publish-subscribe model:
an Oracle AS component receives a notification of a certain type as per its subscription to ONS. When such a
notification is published, ONS sends it to the appropriate subscribers.

Four parameters determine the behavior of the Oracle Process Manager and Notification services process in
managing the iAS middle tier comprising of OC4J instances and the Apache HTTP server. They are

a) restart-on-death
b) ping timeout
c) ping interval
d) reverse-ping timeout

The settings for these parameters need to be governed by the heap sizing of the OC4J container JVMs, the latencies
involved with garbage collection algorithms and the response times of the HTTP server.

The way in which the parameters affect the functioning of OPMN is as follows:

- OPMN pings the process managed by it,ie, OC4J instances and http server periodically and expects a response
within a certain time out period. OPMN would try this 3 times before it declares the process as dead (due to no
response) and then it would kill and restart the particular process. OPMN manages the failover and availability of the
processes it manages.

Reasons for Oracle HTTP Server not responding to OPMN ping could be:
- due to a high load of concurrent requests which requires many http server attention.
- because of timeouts between the various modules of Apache and the servers servicing client requests
- bulk data received from any of the modules.
- all new connections are not getting processed due to the server being fully occupied.
- it can spawn, with many connections remaining in the CLOSE_WAIT state
- due to thrashing when NFS hiccups cause files that need to be served to be unavailable to it
- because of synchronization issues with the various mutexes that it needs to support for the proper
functioning of its modules.
Reasons for OC4J containers not responding to OPMN ping could be:
While the container is processing servlet or EJB logic within the JVM within which it runs, new objects get created in
its heap memory area all the while.

When the garbage collection thread starts to run, it looks for objects that it can release to the heap memory pool
based upon several algorithms that depend upon the kind of references the objects have to themselves. Since the
collection is "generational", i.e., objects having references are promoted to an older generation and presumed to
have a longer lifetime, objects with weak references are candidates for "cleaning" up, and their occupied memory
gets released to the global heap of the JVM.

In this way, memory is reclaimed back into the the heap memory pool and made available for use in the creation of
newer objects. The forays made by the garbage collector to reclaim memory in the heap are governed by several
algorithms and every such collection takes a finite amount of time during which no other work of application
processing is possible. And when the collection is over the entire heap, the full GC consists of a
mark/sweep/compact cycles that "mark" the memory to be reclaimed, "sweep" the memory into the corresponding
generations and "compact" the holes created when the memory is reclaimed, in order as to create contiguous
memory for future object creation.

These strategies consume more time, as to be expected, and can result in a delay in the container responding to an
OPMN ping cycle. During such full GC scans, OPMN can and will kill and restart the container, causing it to lose the
state of the application or request it was processing at that time.

Since full GC scans (referred to to as stop-the-world scans) can happen at any time during the lifetime of a request or
an application, there is always the danger of OPMN killing a perfectly functioning container on the assumption that it
was "hung" since it was "unresponsive".
Heap memory settings for OC4J containers
Changes are made in the opmn.xml file in the <java-option> sections for each OC4J instance:

-Xms ( for start heap memory ) and -Xmx ( for maximum heap memory ).

The recommended settings for the -Xmx value are 512MB as typical applications need that much of memory to
avoid java.lang.OutOfMemoryError exceptions as seen from experience.

Start with an -Xms value of 128MB to prevent side effects of "Too many files open" errors when this setting is
higher, as garbage collection kicks in only later for higher -Xms values, resulting in open file handles not getting
released by the GC.
<java-option>-server -Xms128M -Xmx512M

Thread pool sizing

In the server.xml file, set the thread-pool sizes as follows for optimum operation of the thread pool:
<global-thread-pool min="40" max="40" queue="80" keepAlive="-1"/>

This sets the min and max thread-pool sizes to the same value and the keepAlive parameter to "-1" -
recommended for production environments, this will ensure that idle threads are never destroyed to allow for
 thread reuse without the overhead for new thread creation. The min, max and queue values can be left at the
default as specified here.

Redundancy and load balancing

More than one OC4J instance can be started to accommodate the higher volume of concurrent requests that the
container may need to handle. This is set through the "numProcs" parameter in the opmn.xml file and this
parameter takes the value of 1 by default, to start a single OC4J instance. For multiple instances, the "numProcs"
parameter can be adjusted to different values ( 2 for two instances, and so on ) and PM needs to be restarted with
this value for the modules under its control. Very often, the applications that are being run may be process or
memory intensive and may require one to adjust the value of the "numProcs" parameter to effect load-balancing via
multiple instances.

DMZ setup on Oracle EBS R12

EXTERNAL SERVER SETUP

Environment Name: Machine:

CRP3 Database/Conc/Admin Node: minez

Internal web/Form Node: minej

External Web Node: mine8

Pre-Clone Steps on Internal Server

 Run adpreclone.pl on MINEJand MINEZ with applcrp3 and oracrp3 users.
As oracrp3 user:

cd $ORACLE_HOME/appsutil/scripts/CRP3_MINEZ

perl adpreclone.pl dbTier

As applcrp3 user:

cd $ADMIN_SCRIPTS_HOME

perl adpreclone.pl appsTier

 Take a backup of /d21/applcrp/CRP3 folder on MINEJ

 Restore the Backup into MINE8 server under /d21 mount point.
 Setup Host File:

Put the following entries in the Hosts File (/etc/hosts)

10.218.17.36 minez.sonapglobal.com minez

10.221.15.167 minej.bn.sonap.net minej.bn

 Change the owner Ship of /d21/oracrp3 folder and the file under it to applcrp3

cd /d21

chown –R oracrp3:dba oracrp3

Creating External Web Tier

Note: Your steps should be more specific than the examples shown.

 Create XML file for External Server:

su – applcrp3

cd /d21/applcrp3/CRP3/apps/apps_st/comn/clone/bin

perl adclonectx.pl contextfile=/d21/applcrp3/CRP3/inst/apps/CRP3_minej/appl/admin/CRP3_minej.xml

Enter the APPS password: sonapcrp3

Target System Hostname (virtual or normal) [mine8]:

Do you want the inputs to be validated (y/n) [n]? :

Target System Database SID: CRP3

Target System Database Server Node [mine8]: minez

Target System Base Directory: /d21/oracrp3/CRP3

Target System Forms ORACLE_HOME Directory [/d21/oracrp3/CRP3/apps/tech_st/10.1.2]:

Target System Web ORACLE_HOME Directory [/d21/oracrp3/CRP3/apps/tech_st/10.1.3]:

Target System APPL_TOP Mountpoint [/d21/oracrp3/CRP3/apps/apps_st/appl]:

Target System COMMON_TOP Directory [/d21/oracrp3/CRP3/apps/apps_st/comn]:

Target System Instance Home Directory [/d21/oracrp3/CRP3/inst]:

 Username for the Applications File System Owner [applcrp3]:

Group for the Applications File System Owner [dba]:

Target System Root Service [enabled]:

Target System Web Entry Point Services [enabled]:

Target System Web Application Services [enabled]:

Target System Batch Processing Services [disabled]:

Target System Other Services [enabled]:

Do you want to preserve the Display [minej:0.0] (y/n)? : n

Target System Display [mine8:0.0]:

Do you want the the target system to have the same port values as the source system (y/n) [y]? : n

Target System Port Pool [0-99]: 1

Choose a value which will be set as APPLPTMP value on the target node [1]: 2

New context path and file name [/d21/oracrp3/CRP3/inst/apps/CRP3_mine8/appl/admin/CRP3_mine8.xml]:

Cross Check if the Context File generated is correct or not, check with following command if the respective
components are enabled on MINE8:

grep –i status $CONTEXT_FILE

 Run adpreclone.pl to add the MINE8 server as a node to CRP3:

su - applcrp3

cd /d21/applcrp3/CRP3/apps/apps_st/comn/clone/bin

perl adcfgclone.pl appsTier /d21/applcrp3/CRP3/inst/apps/CRP3_mine8/appl/admin/CRP3_mine8.xml

Enter the APPS password:

Check the logfile for any error.

 CONTEXT_FILE configuration:

Modify the following CONTEXT_FILE parameters:

Context Existing Value New Value

File
Variabl
 e

s_applc /d21/applcrp3/CRP3/inst/apps/CRP3_mine8/l /d21/applcrp3/CRP3/conc

sf ogs/appl/conc

s_applt /d21/applcrp3/CRP3/inst/apps/CRP3_mine8/t /crp3_appltmp

mp emp

s_appl /d21/applcrp3/CRP3/inst/apps/CRP3_mine8/p /crp3_applptmp

ptmp temp

s_forms mine8 crp3ext

host

s_chron http://mine8.sonapglobal.com:8001/oracle_s http://crp3ext.sonapglobal.com:8001/oracle_s

osURL mp_chronos/oracle_smp_chronos_sdk.gif mp_chronos/oracle_smp_chronos_sdk.gif

s_exter http://mine8.sonapglobal.com:8001 http://crp3ext.sonapglobal.com:8001

nal_url

s_webe mine8 crp3ext

ntryhost

s_login_ http://mine8.sonapglobal.com:8001/OA_HTM http://crp3ext.sonapglobal.com:8001/OA_HTM

page L/AppsLogin L/AppsLogin

 Run AutoConfig on al l the Nodes.

Configuring MINE8 for DMZ


Run the script txkChangeProfH.sql for the Profile option setup:

# sonapssh minez

$ su – applcrp3

$ sqlplus appUpdate Hierarchy Types/sonapcrp3 @$FND_TOP/patch/115/sql/txkChangeProfH.sql SERVRESP

Run AutoConfig on all nodes.

 Update Node Trust Level

 To change the value of the Node Trust Level profile option value to External for a particular node, perform the
following steps:

1. Login to Oracle E-Bsonapness Suite as sysadmin user sonapng the internal URL

2. Select the System Administrator Responsibility

3. Select Profile / System

4. From the 'Find system profile option Values' window, select the server and get the valie MINE8 into it.

5. Query for %NODE%TRUST%. You will see a profile option named 'Node Trust Level'. The value for this profile
option at the site level will be Normal. Leave this setting unchanged.

6. Set the value of this profile option to External at the server level. The site level value should remain set to Normal

 Update List of Responsibility

To change the value of the Responsibility Trust Level profile option at the responsibility level for a particular
responsibility, perform the following steps:

7. Login to Oracle E-Bsonapness Suite as sysadmin user sonapng the internal URL

8. Select System Administrator Responsibility

9. Select Profile / System

10. From the 'Find system profile option Values' window, select the responsibility that you want to make available to
users logging in via the external web tier

11. Query for %RESP%TRUST%. You will see a profile option named 'Responsibility trust level'. The value for this
profile option at site level will be Normal. Leave this setting unchanged.

12. Set the value of this profile option for the chosen responsibility to External at the responsibility level. The site-level
value should remain Normal.

13. Repeat for all responsibilities that you want to make available from the external web tier.

List of Responsibilities which can be enabled on External Server is as followed:

Product Name Externally Accessible Responsibilites Additional Profile Options

iSupplier POS: External URL

POS Supplier Guest User POS: Internal URL

 Plan to Pay Supplier View

Plan, Source, Pay Supplier View

Source to Pay Supplier View

Supplier Profile Manager

Procure to Pay Supplier View

Oracle Sourcing Sourcing Supplier PON: External Applications

Framework Agent

PON: External login URL

Oracle Self Registered Employee Default

iProcurement Responsibility

Self Registered New User Default

Responsibility

 Enable Oracle E-Business Suite Application Server Security

1. Set the value of Application Server Security Authentication (s_appserverid_authentication) to SECURE, in the
CONTEXT_FILE on all the nodes.

2. Run AutoConfig on each Applications middle tier to complete the configuration.

3. After AutoConfig completes successfully, restart Oracle HTTP Server and OC4J processe

 Increase JVM Size

Change the following JVM parameter in the CONTEXT_FILE as mentioned in the below table:

Note: Take a backup of Context File before Changing.

Variable Exisiting Value New Value

s_oacore_jvm_start_options -server -verbose:gc -Xmx512M - -server -verbose:gc –Xmx1024M

Xms128M - –Xms521M -
XX:MaxPermSize=160M XX:MaxPermSize=256M

 Run Autoconfig
Run AutoConfig from ADMIN_SCRIPTS_HOME sonapng adautocfg.sh.

Enable SSL Login

Note:Include a subset of test steps that will confirm that the customization has been installed properly.

REFERENCE

 Metalink Document: “Oracle E-Bsonapness Suite R12 Configuration in a DMZ” Document ID: 380490.1

Oracle EBS R12 with servlet forms mode and more

Forms Servlet mode in Oracle EBS R12:

By default, Oracle EBS R12 has forms 10g Stack in servlet mode.
Now the question is how is the client server connections handled?
In Servlet mode we have a java servlet called the forms listener servlet which is responsible for the
communication between the Forms java client and the oracle application serverforms services.

Unlike Socket mode, the Forms Servlet mode doesn’t require an additional forms port to handle communication
between client and the application server. The forms servlet architecture operates through the http server port.
All forms connections happen through the http port. Hence desktop clients can’t access forms directly in
servlet mode.

The servlet architecture is designed to work with industry standards, and also to be fully supported for advanced
network configurations with no additional ports or SSL configuration.

In Oracle Applications R12 customers who want load balancing must use the servlet implementation in which
forms servers can be balanced across multiple application servers by using a hardware load balancer along with
the 10g Application Server.

The way to find if the servlet mode/socket mode is implemented:

Check the forms configuration file : appsweb.cfg pointed by environment

variable"FORMS_WEB_CONFIG_FILE".

The entries:
 serverURL=
 connectMode=
Would be as mentioned below:

In servlet mode In Socket mode

 serverURL=/forms/lservlet serverURL=(should be blank)
connectMode=servlet connectMode=Socket

This can also be verified in the context file in 11i and R12:

context variable value for Servlet value for Socket

s_forms_servlet_serverurl 11i: /forms/formservlet (no value, blank)
r12: /forms/lservlet
s_forms_servlet_comment (no value, blank) #
s_frmConnectMode r12: servlet r12: socket

We can change the forms servlet mode to socket and how we do it is:

1. Source the environment on the application tier.

2. Stop all the application tier services.
3. Execute :
$FND_TOP/bin/txkrun.pl -script=ChangeFormsMode \
-contextfile=$INST_TOP/appl/admin/mycontext.xml \
-mode=socket \
-port=9095 \
-runautoconfig=No

Note: context file is the full path of the context file of the instance. Mode is the mode to which the form has to be
changed. Port number used to run Forms in socket mode. The default port number is 9095.A port number is not
needed if servlet mode is used. Recommended value for autoconfig is NO.

4. Run autoconfig on apps tier.

5. Start all the application tier services.
6. Check whether Forms Server is running:
$INST_TOP/admin/scripts/adformsrvctl.sh status
7. Log in to Oracle Applications and launch a Forms-based application.
8. Open the Sun Java Console (from Tools Menu in Internet Explorer).
9. Check whether the "mode" directive displayed in Sun Java Console when launching forms-
based applications is set to socket.
10. The direct launch URL for Forms Socket Mode is:
<web_protocol>://<web_host>.<web_domain>:<web_port>/OA_HTML/frmservlet

Note: If we need to change the forms mode from socket to servlet mode then all steps remain same except
for the script which we run and the verification step:
1. $FND_TOP/bin/txkrun.pl -script=ChangeFormsMode \
-contextfile=$INST_TOP/appl/admin/mycontext.xml \
-mode=servlet \
-runautoconfig=No
No Port number is required as it would use the http port.
2. Verification of servlet mode should reflect:

The "mode" directive displayed in Sun Java Console when launching forms- based applications, is set
to http,native
Direct Forms Servlet Launch is:
web_protocol>://<web_host>.<web_domain>:<web_port>/forms/frmservlet

How to Find version

1. How to find Apps Version (11i/R12/12i)
>> Connect to database as user apps

SQL> select release_name from apps.fnd_product_groups;

Output would be like 12.0.4 or 11.5.10.2

2. Web Server/Apache or Application Server in Apps 11i/R12

>> Log in as Application user, set environment variable and run below:

$IAS_ORACLE_HOME/Apache/Apache/bin/httpd –version

Output for 11i should be like :

Server version: Oracle HTTP Server Powered by Apache/1.3.19 (Unix)

Server built: Jan 26 2005 11:06:44 (iAS 1.0.2.2.2 rollup 5)

Output for R12 should be like :

Server version: Oracle-Application-Server-10g/10.1.3.0.0Oracle-HTTP-Server

Server built: Dec 4 2006 14:44:38

3. Forms & Report version (aka developer 6i) in 11i

>> Log in as Application user, set environment variable and run below query:

$ORACLE_HOME/bin/f60run | grep Version | grep Forms

Output would be like :

Forms 6.0 (Forms Runtime) Version 6.0.8.25.2 (Production)
Check fourth character in version 25 which means Forms 6i patchset 16 (25-9)
4. Forms & Report version in R12/12i

>> Log in as Application user, set environment variable and run below query:

$ORACLE_HOME/bin/rwrun | grep Release

Output should be like:

Report Builder: Release 10.1.2.2.0

You can safely ignore warnings

5. Oracle Jinitiator in 11i/R12/12i

>> Log in as Application user, set environment variable and run below query:

grep jinit_ver_comma $CONTEXT_FILE

(Default is Java Plug-In for R12/12i )

6. Oracle Java Plug-in in 11i/R12/12i

>>Log in as Application user, set environment variable and run below query:

grep plugin $CONTEXT_FILE

7. File Version on file system

>>adident Header <filename>

Or

strings <file_name> | grep Header

Here adident is AD Utility (Oracle Apps) and strings is Unix utility

8. Version of pld file

>> *.pld are source code of *.pll which are inturn source of *.plx. *.pll is in $AU_TOP/resource and to find its
version check

adident Header $AU_TOP/resource/<filename>.pll

IGSAU012.pll:
$Header IGSAU012.pld 115.1.115100.1 2004/04/01 05:40:18 appldev ship $

Or

strings $AU_TOP/resource/<filename>.pll | grep -i header

FDRCSID(’$Header: IGSAU012.pld 115.1.115100.1 2004/04/01 05:40:18 appldev ship $’);

9. OA Framework Version

>> http:// hostname.domainName:port/OA_HTML/OAInfo.jsp (Only for 11i); Log in as Application user, set
environment variable and run below query:

adident Header $FND_TOP/html/OA.jsp

adident Header $OA_HTML/OA.jsp

output for both should look like

$Header OA.jsp 115.60 2006/03/31 00:47:28 atgops1 noship $

120.21 means OA Framework Version (coming soon..)

115.60 means OA Framework Version (coming soon..)
115.56 means OA Framework Version (coming soon..)
115.36 means OA Framework Version 5.7
115.27 means OA Framework Version 5.6E
115.26 means OA Framework Version 5.5.2E

10. Discoverer Version for 11i (3i or 4i)

>> Log in as Application user, set environment variable and run below query:

$ORACLE_HOME/bin/disc4ws | grep -i Version

Discoverer Version for 11i or R12 (10g AS) >> Check under Application Server Section as 10g AS Discoverer
is on standalone.

11. Workflow Version with Apps

>> Connect to Database as apps user

SQL> select TEXT Version from WF_RESOURCES where NAME = ‘WF_VERSION’;

Output like 2.6.0 means workflow version 2.6.0

12. Oracle Single Sign On

>> Connect to database which holds SSO repository:

SQL>select version from orasso.wwc_version$;

13. Oracle Internet Directory

>> There are two components in OID (Software/binaries & Schema/database)

>>> To find software/binary version

$ORACLE_HOME/bin/oidldapd –version

output should look like

oidldapd: Release 10.1.4.0.1 - Production on thu sep 11 11:08:12 2008

Copyright (c) 1982, 2006 Oracle. All rights reserved.

>>> To find Schema Version/ database use

ldapsearch -h <hostname> -p <port> -D “cn=orcladmin” -w “<password>” -b “” \

-s base “objectclass=*” orcldirectoryversion

and output should be like

version: 1
dn:
orcldirectoryversion: OID 10.1.4.0.1

or

Run following query in database

SQL> select attrval from ods.ds_attrstore where entryid = 1 and attrname = ‘orcldirectoryversion’;

Output should be like OID 10.1.4.0.1

14. Application Server

>> Oracle Application Server (Prior to Oracle WebLogic Server)

If application server is registered in database (Portal, Discoverer) check from database:

SQL> select * from ias_versions;

Or

SQL>select * from INTERNET_APPSERVER_REGISTRY.SCHEMA_VERSIONS;

AOC4J (Oracle Container for J2EE)

Set ORACLE_HOME
cd $ORACLE_HOME/j2ee/home
java -jar oc4j.jar -version

15. Oracle Portal

>> SQL> select version from portal.wwc_version$;

16. Database Component

I) Oracle Database

To find database version:

SQL> select * from v$version;

Or

All component version in database:

$ORACLE_HOME/OPatch/opatch lsinventory -detail

17. Oracle Enterprise Manager

>> Metalink Note 605398.1: How to find the version of the main EM components

18. Unix Operating System

Solaris -> cat /etc/release

Red Hat Linux -> cat /etc/redhat-release