18c RAT Replay Failed with ora-3137 and ora-18 Due to High "Row Cache Mutex" (Doc ID 2620875.1)

Last updated on MAY 01, 2020

Applies to:

Symptoms

• RAT replay failing half way through the run with following error:
  Sessions failing with ORA-00018 : maximum number of sessions exceeded.
  WRC clients failing with ORA-03137
   
• AWR and ASH show 90% of time on "row cache mutex":
  Event Waits Total Wait Time (sec) Avg Wait % DB time Wait Class
  row cache mutex 6,478,228 809.5K 124.96ms 92.1 Concurrency <<~~~~~ 92%
  Mutex Sleep Summary
  ordered by number of sleeps desc
  Mutex Type Location Sleeps Wait Time (ms)
  Row Cache [17] kqrCreateUsingSecondaryKey 77,626,639 515,410,723
   
• Hanganalyze output trace:
  Waiter session,

  ..
  is waiting for 'row cache mutex' with wait info:
  {
  p1: 'cache id'=0xa
  p2: 'where requested'=0x11
  time in wait: 1.835975 sec
  heur. time in wait: 2.729759 sec
  timeout after: never
  wait id: 2514
  blocking: 0 sessions
  current sql_id: 0
  current sql: <none>
  short stack:
  ksedsts()+418<-ksdxfstk()+71<-ksdxcb()+896<-sspuser()+217<-__sighandler()<-sem
  timedop()+10<-skgpwwait()+200<-ksliwat()+2229<-kslwaitctx()+220<-kgxWait()+130
  9<-kgxExclusive()+758<-kqrGetAllHashMutexes()+733<-kqrCreateUsingSecondaryKey(
  )+593<-kqrpre2()+5915<-kqrpre1()+40<-kkdl_get_adjunct_user()+608<-koknmpn()+29
  2<-opikod()+702<-opiodr()+1244<-ttcpip()+1239<-opitsk()+1943<-opiino()+957<-op
  iodr()+1244<-opidrv()+1091<-sou2o()+191<-opimai_real()+455<-ssthrdmain()+423<-
  main()+275<-__libc_start_main()+253

  Blocker session

  and is blocked by
  => Oracle session identified by:
  {
  instance: 1 (rat9n99.rat9n99-1)
  os id: 69486
  process id: 253, oracle@db-db2-1-yul.ops.sfdc.net
  session id: 5828
  session serial #: 50621
  module name: 0 (JDBC Thin Client.ops.sfdc.net (TNS V1-V3))
  }
  which is waiting for 'row cache mutex' with wait info: <<~~~~~~
  {
  p1: 'cache id'=0xa
  p2: 'where requested'=0x11 <<~~~~~~
  time in wait: 0.000000 sec
  heur. time in wait: 1.067597 sec
  timeout after: never
  wait id: 34233
  blocking: 2223 sessions <<~~~~~~
  current sql_id: 0
  current sql: <none>
  short stack: <none: error encountered - ORA-00072: process "Unix process pid: 69486, image: oracle@db-db2-1-yul.ops.sfdc.net" is not active> <<~~~~~ no short stack

    
  

   

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.