Process Scheduler Issues

Issue 1: Generating Integration Broker .trc file with messages like (Publication Manager::Publish()), even though IB services
are down Resolution: I find this as a bug with PeopleSoft. This wont do any harm ,but will result in unnecesssary generation
of trace files. To avoid this change the parameter LogFence=0 in psprcs,cfg file and reboot the process scheduler

Issue2: PeopleSoft Process scheduler-Processes getting queued.

Resolution There are three tables in PeopleSoft which should be in sync always PSPRCSRQST, PSPRCSQUE and
PSPRCSPARMS
Execute the below query to check the tables are in synch
select count(*) from PSPRCSRQSTunion allselect count(*) from PSPRCSQUEunion allselect count(*) from
PSPRCSPARMS;
If the counts are different execute the delete queries to synch them and restart the scheduler. This will resolve the issue
DELETE FROM PSPRCSQUE QUE WHERE NOT EXISTS (SELECT 'X' FROM PSPRCSRQST RQST WHERE
RQST.PRCSINSTANCE = QUE.PRCSINSTANCE);
DELETE FROM PSPRCSPARMS PARMS WHERE NOT EXISTS (SELECT 'X' FROM PSPRCSQUE QUE WHERE
QUE.PRCSINSTANCE = PARMS.PRCSINSTANCE);

You can also check when the process is scheduled to run if it remains queued after doing the above synch up. Also check
whether there is any timezone difference between your DB,App and PRcs server machines. This can also cause the
processes to get queued.

Issue:3

Database error encountered: SELECT DAYSBEFOREPURGE FROM PS_PRCSPURGELIST WHERE RUNSTATUS = '9'.

When starting the Process Scheduler, this error message appears in the SCHDLR_mmdd.log. This is preventing the purge
process from running

Resolution

This is a potential issue for customers upgrading to 8.4 or new customers. The applications upgrade path or installation
instructions should include a step to run the prcspurgelist.dms. Some products like CRM and EPM don't include it. Most
customers reporting this issue are using CRM or EPM. The prcspurgelist.dms populates the PS_PRCSPURGELIST with the
information seen in the System Settings System Purge Options. The PS_PRCSPURGELIST table must contain this
information before the purge process can run.

Issue4
The Process Scheduler purge process is not running. The following messages were found in the scheduler log file,
schdlr_mmdd.log. "PSMSTPRC.1278194 (0) [03/31/05 01:00:29 PSAPP@machinename](0) Purging
expiredrequest(s)...PSMSTPRC.1278194 (0) [03/31/05 01:00:30 PSAPP@machinename](-1) Debuggingdisabled. Comm unication failure
during connect(): 79. (143,9)PSMSTPRC.1278194 (0) [03/31/05 01:00:30 PSAPP@machinename](1) Debugging has beenforcibly disabled
do to a Configuration or Communication problem.PSMSTPRC.1278194 [03/31/05 01:00:32 PSAPP@machinename](1)
(Handler):PSMSTPRC.1278194 (0) [03/31/05 01:00:32 PSAPP@machinename](3) Unable toinitiate purge due to unsuccessful schedule
ofthe request using PT_PRCS Application Class"

Resolution The PTPT1200 and PTPT1000 permission lists are required for the Process Scheduler user. These can be made
available to the user by adding the roles PeopleTools and PeopleSoft User.For additional information on the security needed
for the purge, refer to metalink article: E-PRCS: What Are The Permissions For The Purge Process, Prcsyspurge? (Doc ID
1110701.1).

Issue 5
Peoplesoft nvision programs continues to be in processing status

Resolution
This normally happens when Excel process in windows Hangs. Check the task manager and restart excel if needd.

Issue5

AE programs stays in initiated state

If you check PSAESRV log you see an error message like
'RunAeAsync1 parameter data validation failed'

Resolution:
Check the trace settings associated with the program
setting trace options on an application engine process but using override instead of append in the
parameter list can be the issue

ISSUE6:

Process Scheduler(s) are running fine, then processes remain in a queued state.

Resolution
Things to check:
1) MAX API Aware setting <--- Are all active slots filled?
This setting controls the total amount of concurrently running processes that the Process
Scheduler can handle at any time. This value controls all Process Category Types and Process
Types. Once the maximum value has been reached, the Process Scheduler cannot start
processing new requests.

Note: PSJobs are not considered in this value. PSJobs are a container for other processes.
 2) MAX Concurrent setting <--- Are all active slots filled for problem process type?
This setting controls the amount of processes assigned to the corresponding process category
that the Process Scheduler can handle at any time. For example, if the Process Category is
defined as 5, then a total of 5 processes assigned to that Process Category can be run
concurrently. This is limited by the value defined in the Max Api Aware. Once the maximum value
has been reached, the Process Scheduler cannot start processing new requests.
3) Is the Process Scheduler successfully updating it's timestamp and heartbeat?
Shutdown the process scheduler, delete it's CACHE directory to clear any possibly corrupt cache,
set TraceSQL=31and LogFence=5 in psprcs.cfg, reconfigure the process scheduler server via the
psadmin
utility and then boot up the process scheduler. Gather all log files from the process scheduler
LOGS
directory with the most current date for review to see the behavior of the process scheduler on
startup for
further clues as to what the problem may be.
4) Check the system Time Zone settings. For UNIX, this may be in different places, for example
AIX, the /etc/environment file. Make sure the parameter is correctly stated, e.g. PST8PDT,
EST5EDT. The middle figure is the UTC offset. Consult UNIX man pages for details on the
particular OS.
2)
CASE 2: Recurrence causes the Process Scheduler to hang

Process Schedulers hang on a recurrence and stop initiating all processes.

SOLUTION:
In this case after reviewing the trace file it was found that the Process Scheduler could not move past a
certain process instance and it was found that the instance was a bad recurrence. Once the problem rows
were removed from the PSPRCSRQST and PSPRCSQUE tables, the Process Scheduler was able to
start processing all other processes. This issue ended up being that the recurrence had overlapping date
ranges in the "Run On Specific Dates" section vs the "Exception Dates".

CASE 3: Processes submitted with a blank servername remain queued

When submitting a process online without specifying a Process Scheduler, the process will often stay in
queued status in the Process Monitor even though there is a Process Scheduler active. Should you
hardcode the process definition to a particular Process Scheduler or select a Process Scheduler from the
process request submission page, then the process does run immediately.
SOLUTION:

There are two possible solutions for this:

1) Decrease the heartbeat on the affected Process Scheduler.

OR

2) Configure a Master Scheduler to take over the job of routing the processes.
CASE 4: All processes are queued
All Process requests remain queued.

SOLUTION:

The Process Scheduler tables: PSPRCSRQST and PSPRCSQUE were out of sync. After syncing up the
tables to ensure the same process instance numbers were in both tables and recycling the Process
Schedulers, the processes started running.

CASE 5: Appengine processes are queued, but other processes are running normally

Appengine processes remain queued, however other processes like Sqrs and Crystals run.

This issue was caused by a code failure as outlined in Bug:11326740 - "psft 1381611000-master
scheduler: ae processes stay queued as sqr to e-mail fail".

SOLUTION:
1) A resolution is available in the following releases:
PeopleTools 8.46.18 Update ID [701898] / Bug ID [11407475].
PeopleTools 8.47.11 Update ID [686615] / Bug ID [11401062].
PeopleTools 8.48.05 Update ID [686170] / Bug ID [11401063].
Please review Knowledge Document How to Download PeopleSoft Bundles and Fixes from My Oracle
Support (Doc ID 1432368.1) for further details.

OR

2) Manually remove the offending Sqr processes.

a) Locate the offending processes by querying the database:
- SELECT PRCSINSTANCE, JOBINSTANCE, RUNDTTM, PRCSNAME FROM PSPRCSRQST WHERE
RUNSTATUS=3 AND PRCSTYPE IN ('SQR Report', 'SQR Process')
- SELECT PRCSINSTANCE, JOBINSTANCE, RUNDTTM, PRCSNAME FROM PSPRCSQUE WHERE
RUNSTATUS=3 AND PRCSTYPE IN ('SQR Report', 'SQR Process')

b) Of the returned values, run the following statements:

- UPDATE PSPRCSRQST SET RUNSTATUS = 2 WHERE RUNSTATUS=3 AND PRCSTYPE IN ('SQR
Report', 'SQR Process')
- UPDATE PSPRCSQUE SET RUNSTATUS = 2 WHERE RUNSTATUS=3 AND PRCSTYPE IN ('SQR
Report', 'SQR Process')

c) Commit the changes and bounce all Schedulers in the environment to force the Master Scheduler to
pull the updated list from the database.
CASE 6: Processes are queued on a Solaris Process Scheduler

On Solaris Process Schedulers, all Processes intermittenly remain queued.

SOLUTION:
This issue was caused because the Solaris server time kept drifting to the past, which is a known issue
on Sun Fire 3800/4800/4810/6800, V1280, and Netra 1280 domains.

To resolve this issue, apply one of the following workarounds:

Setting the variable "tod_broken" to 1 in the domain kernel (see below), or

Reboot the SCs before 528 days of SC continuous uptime (recommended at 500 days), or
Install Patch 112884-04 (ScApp 5.15.3)

CASE 7: Processes remain queued if a non-Master Scheduler is brought down

In a multi-Process Scheduler environment running with a Master Scheduler, if one of the non-Master
Schedulers is brought down, processes will remain queued.

SOLUTION:

For exampe, there are two Schedulers: PSNT1 and PSNT2 - this one is configured with "Do Not
Redistribute". PSNT1 can run 3 processes, while PSNT2 can run 5. This gives 8 total processes that can
be run at any time. 5 processes are submitted to PSNT2 and 3 processes are submitted to PSNT1. Then,
PSNT2 is brought down.

The Master Scheduler will try to run the first 5 submitted to PSNT2, but it cannot because it is down and it
cannot reassign the processes because of the "Do Not Redistribute" option. As a result, all subsequent
processes will remain queued until PSNT2 comes back online and takes care of the first 5 processes.

CASE 8: Processes defined to run on one OS are run on another, leading to queuing behavior

In an environment containing different OS types - ie NT, Unix, certain processes that are defined to run
only on a specific OS type, such as NT, are assigned to run on Unix. This results in those processes
remaining queued.

SOLUTION:
This issue was caused by a code failure in the Master Scheduler code that caused it to ignore the
"Redistribute Workload" Option defined on NT to keep its processes on NT type OS only. This issue was
fixed in PeopleTools 8.50.