How to clear weblogic stuck threads in R12.2

Web logic stuck threads Overview

 

What is stuck threads in weblogic server?

Stuck threads are  long running threads and It means thread has been running longer predefined timeout.

 

What stuck threads will do?

A thread might get stuck for various reasons.

 

Example:

 When an SQL query is running and the time it takes to complete is say 800 seconds, then, as the default stuck thread timing is 600 seconds in WebLogic Server, the thread allocated for that query waits for 600 seconds and goes to stuck state.

Then managed server status showing waning state

 

How to troubleshoot the stuck threads in EBS instance?

Stuck threads will reduce the ebs performance and it’s very important activity to DBA’s

Step 1: Go to the weblogic console and check for managed server status like oacore

Step 2: if It will show the oacore server in “warning state” then we need to check

 

Scenario 1:

Every alert we need to check the log file and investigate based on the log file what caused this.

 

Scenario 2:

Need to check execution context id (ECID) 

How to find ECID ?

There are two way to find the ECID

 one  is need to check weblogic console 

 And another is need to check log file

 

Go to the weblogic console

Click OA-Core server->Monitoring->Threads

OA-Core health showing “Warning”

Click àwarning

Scroll down click next->next page show the stuck threads

Find ECID

 

Example below value we can see

 

ECID-Context: 1.005ZMgd6^_005ej6ub5B97q2OayhmZMG000FoR00015X;

 

Need to check this value only ECID   005ZMgd6^_005ej6ub5B97q2OayhmZMG000FoR00015X

 

Find the ecid in logfile

 grep -e STUCK -e ECID $FMW_HOME/user_projects/domains/EBS_domain/servers/oacore_server9/logs/oacore_server9.out|grep ECID|awk {'print $2'}|sed 's/^.\{2\}//'|sed 's/.\{5\}$//' 

 

Find the ecid details using v$session using below queries

 

select 'alter system kill session '''||sid||','||serial#||''' immediate;', program,trunc(last_call_et/60) last_call_et_minutes,module,

action,ecid,username,schemaname,client_identifier, process, machine, sql_id, status,blocking_session_status,event,state

from v$session where program like 'oacore_server6'

--and ecid like '%%'

-- and action like '%%'

order by program,last_call_et desc;

 

 

 

 

'ALTERSYSTEMKILLSESSION'''||SID||','||SERIAL#||'''IMMEDIATE;'

--------------------------------------------------------------------------------

alter system kill session1234,56789 immediate;

 

 

After killing SID, verify if "WARNING" message is gone and health is showing "OK" now

 

 

Reference:

EBS 12.2 Oacore JVMs Getting Into Warning State Because Of STUCK Threads (Doc ID 2061045.1)