How to troubleshoot long running concurrent request in R12.2

Normally we are getting concurrent program running longer and concurrent request not picking jobs, etc. those are very frequent issue we expected from development team.

Collect below basis level of information to development team

1 Oracle seeded program/Custom program?

2 How much time it used earlier?

3 Is there any recent code change done in concurrent program?

4 Is this program fetching higher data compare to last run?

5 Does this job running any specific time/ It can be run any time?

6 Does this job fetching data using DB link?

7 Does the problem happen on both the test and production instance?

Once asked above questions to development team in meanwhile we need to start basic sanity check in our environment.

1 Verify the status of the concurrent program

select REQUEST_ID,phase_code,status_code,ORACLE_SESSION_ID from apps.fnd_concurrent_requests where request_id=’1234567’;

2 Find which concurrent manger ran a specific concurrent request

select b.USER_CONCURRENT_QUEUE_NAME from fnd_concurrent_processes a,fnd_concurrent_queues_vl b, fnd_concurrent_requests c where a.CONCURRENT_QUEUE_ID = b.CONCURRENT_QUEUE_ID and a.CONCURRENT_PROCESS_ID = c.controlling_manager and c.request_id ='856272604';

3 Verify the actual and target of the CM

4 Verify the concurrent manager status

5 Find SID for the concurrent request use below query

SELECT A.REQUEST_ID, D.SID, D.sql_id,D.SERIAL# , C.SPID
FROM APPS.FND_CONCURRENT_REQUESTS A,
APPS.FND_CONCURRENT_PROCESSES B,
gV$PROCESS C,
gV$SESSION D
WHERE A.CONTROLLING_MANAGER = B.CONCURRENT_PROCESS_ID
AND C.PID = B.ORACLE_PROCESS_ID
AND B.SESSION_ID=D.AUDSID
AND A.REQUEST_ID = &REQUEST_ID
AND A.PHASE_CODE = 'R';

6 Find any blocking session for concurrent request use below query and verify concurrent program SID

select (select username from v$session where sid=a.sid) blocker, a.sid, ' is blocking ' "IS BLOCKING", (select username from v$session where sid=b.sid) blockee,b.sid from v$lock a, v$lock b where a.block = 1 and b.request > 0 and a.id1 = b.id1 and a.id2 = b.id2;

If any blocking session there use below query to find the concurrent request id 

SELECT s.inst_id,a.request_id,s.sid,s.serial#,c.spid FROM apps.fnd_concurrent_requests a, gv$process c, gv$session s WHERE s.sid in ('&sid') AND s.paddr = c.addr
AND a.oracle_process_id = c.spid AND a.phase_code = UPPER ('R');

7 Find client identifier and module, action for concurrent request

8 Then check with module owner with the concurrent request

select sid,serial#,USERNAME,CLIENT_IDENTIFIER,status,LAST_CALL_ET/60,SQL_HASH_VALUE,module,action,program from v$session where sid='&n';

9 Find which sql query is running 

select sql_text from v$sqltext where sql_id = 'a1x81t4jwn3jh' order by piece;

10 Find stale value of the tables which is used by the concurrent program

select table_name, stale_stats, last_analyzed from dba_tab_statistics where stale_stats='YES';

11 You may have to run the gather stats against those tables which are having stale value.

12 Use OEM and monitor session ID

13 If development team ask tkprof,AWR,ASH report please generate and analysis.  

Reference:

R12.2 forms performance Tunings

Concurrent Manager Performance tuning R12.2