Hang Analysis in Exadata

In half rack of Exadata machine, from node got the following error:


PMON failed to acquire latch, see PMON dump



First I checked that whether the database was hung or not.

Database was up and all the four instance were up too along with ASM instances.

Database was also accepting the logins.

The above error occurred just once, so that was a relief and seemed a temporary

glitch.



Checked the PMON dump, and it showed that a process temporarily blocked PMON

as it held the lock on child library cache.

PMON unable to acquire latch 60106398 Child shared pool level=7 child#=2



But wanted to make sure that things are fine, so used the blocking was temporary

and the database is not hung or having serious problem, used the oradebug utility

to do the hang analysis.

In summary, the hang analysis suggested that there were no cycles (true blocking

at the internal kernel level). There were also not any session which was blocking

many (10) sessions. There were also no Open Chains (many sessions in waiting).

There have been mention of waiting of sessions on the Mutex.



Did the hang analysis as follows:

Opened two putty sessions and fired SQL Plus on both of them with the node:



In Session 1: (gathering hang analysis)

SQL> oradebug setmypid
SQL> oradebug unlimit;
SQL> oradebug hanganalyze 3



Hang Analysis in /u01/app/oracle/diag/rdbms/test/test4/trace/test4_ora_20189.trc

In Session 2: (gathering system state)

SQL> oradebug setmypid
SQL> oradebug unlimit;
SQL> oradebug dump systemstate 266

Then wait for at least 1 minute, and then again do following for comparison:

In Session 1: (gathering hang analysis again)

SQL> oradebug hanganalyze 3

Hang Analysis in /u01/app/oracle/diag/rdbms/test/test4/trace/test4_ora_20189.trc


In Session 2: (gathering system state again)



SQL> oradebug setmypid
SQL> oradebug unlimit;
SQL> oradebug dump systemstate 266

/u01/app/oracle/diag/rdbms/test/test4/trace/test4_ora_20189.trc
contains the hang analysis and you should check for cycles, blockers, open chains.



Reference: MOS [ID 452358.1]