GEN0 Terminating The Instance Due To Error 15082 Since ASMB Is Not Moving For More Than 4 Minutes (Doc ID 2450132.1)

Starting background process RSM0
Wed Sep 12 06:27:32 2018
RSM0 started with pid=53, OS id=345670
Wed Sep 12 06:27:42 2018
NOTE: ASMB registering with ASM instance as Flex client 0x1002f (reg:1825550931) (reconnect)
Wed Sep 12 06:27:59 2018
ERROR: terminating instance because ASMB is stuck for 243 seconds
GEN0 (ospid: 311815): terminating the instance due to error 15082
Wed Sep 12 06:27:59 2018
System state dump requested by (instance=1, osid=311815 (GEN0)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/RACDB/RACDB/trace/RACDB_diag_311821_20180912062759.trc
Wed Sep 12 06:28:00 2018
Dumping diagnostic data in directory=[cdmp_20180912062759], requested by (instance=1, osid=311815 (GEN0)), summary=[abnormal instance termination].
Wed Sep 12 06:28:00 2018
Instance terminated by GEN0, pid = 311815
Wed Sep 12 06:28:00 2018
License high water mark = 18
Wed Sep 12 06:28:01 2018
USER (ospid: 346215): terminating the instance
Wed Sep 12 06:28:01 2018
Instance terminated by USER, pid = 34621

PROCESS 6: GEN0
  ----------------------------------------
  SO: 0x11bc9c7e0, type: 2, owner: (nil), flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
   proc=0x11bc9c7e0, name=process, file=ksu.h LINE:15729, pg=0, conuid=0
  (process) Oracle pid:6, ser:1, calls cur/top: 0x11949aa60/0x11949aa60
            flags : (0x6) SYSTEM  icon_uid:0 logon_pdbid=0
            flags2: (0x0),  flags3: (0x10)
            call error: 0, sess error: 0, txn error 0
            intr queue: empty
    (post info) last post received: 0 0 182
                last post received-location: kjxgn1.h LINE:262 ID:kjxgnaef
                last process to post me: 0x11bca6d80 1 6
                last post sent: 0 0 175
                last post sent-location: kjc.h LINE:2308 ID:KJCS Post snd proxy to flush msg
                last process posted by me: 0x11bca7e10 1 6
                waiter on post event: 0
    (latch info) hold_bits=0x0 ud_influx=0x0
    Process Group: DEFAULT, pseudo proc: 0x11a2eea20
    O/S info: user: grid, term: UNKNOWN, ospid: 30404
    OSD pid info:
    Short stack dump:
ksedsts()+346<-ksdxfstk()+71<-ksdxcb()+912<-sspuser()+217<-__sighandler()<-semtimedop()+10<-skgpwwait()+200<-ksliwat()+2292<-kslwaitctx()+197<-kjoer_rplt_wait_for_acks()+778<-kjoer_rplt_operation()+2291<-kjoer_deregister()+3865<-kfnrclDelete()+4261<-kss_del_cb()+242<-kssdel()+208<-kfnrclKill()+2083<-kfnrcl_gc()+1305<-kfnTimeout()+450<-ksbcti()+247<-ksbabs()+11524<-ksbrdp()+1079<-opirip()+609<-opidrv()+602<-sou2o()+145<-opimai_real()+202<-ssthrdmain()+417<-main()+262<-__libc_start_main()+253

......
........

PROCESS 29: ASMB
  ----------------------------------------
  SO: 0x45cce9948, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
   proc=0x45cce9948, name=process, file=ksu.h LINE:13949, pg=0 conuid=0
  (process) Oracle pid:29, ser:1, calls cur/top: 0x45d4d2120/0x45d4d2120
            flags : (0x6) SYSTEM  icon_uid:0
            flags2: (0x0),  flags3: (0x10)
            intr error: 0, call error: 0, sess error: 0, txn error 0
            intr queue: empty
    ksudlp FALSE at location: 0
    (post info) last post received: 1092 256 12
                last post received-location: ksu.h LINE:15435 ID:ksusig
                last process to post me: 0x45ccdf568 1 6
                last post sent: 0 0 439
                last post sent-location: kffm2.h LINE:516 ID:kffmLoad
                last process posted by me: 0x45ccf2f80 3 0
                waiter on post event: 0
    (latch info) hold_bits=0x0
    Process Group: DEFAULT, pseudo proc: 0x45f0e41d8
    O/S info: user: oracle, term: UNKNOWN, ospid: 345186 (DEAD)

..........
..........



Current Wait Stack:
     0: waiting for 'ASM background timer'
        =0x0, =0x0, =0x0
        wait_id=168634755 seq_num=13285 snap_id=1
        wait times: snap=4 min 2 sec, exc=4 min 2 sec, total=4 min 2 sec               <<<<<<<<< 
        wait times: max=infinite, heur=4 min 2 sec
        wait counts: calls=0 os=0
        in_wait=1 iflags=0x520

.........
.........

PROCESS 34: ASMB
  ----------------------------------------
  SO: 0x11bcb97a0, type: 2, owner: (nil), flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
   proc=0x11bcb97a0, name=process, file=ksu.h LINE:15729, pg=0, conuid=0
  (process) Oracle pid:34, ser:2, calls cur/top: 0x119499b48/0x119499b48
            flags : (0x6) SYSTEM  icon_uid:0 logon_pdbid=0
            flags2: (0x0),  flags3: (0x10)
            call error: 0, sess error: 0, txn error 0
            intr queue: empty
    (post info) last post received: 0 0 0
                last post received-location: No post
                last process to post me: none
                last post sent: 1681 0 3
                last post sent-location: ksl2.h LINE:4497 ID:kslpsr
                last process posted by me: 0x11bcbb8c0 4 0
                waiter on post event: 0
    (latch info) hold_bits=0x0 ud_influx=0x0
    Process Group: DEFAULT, pseudo proc: 0x11a2eea20
    O/S info: user: grid, term: UNKNOWN, ospid: 30650
    OSD pid info:
    Short stack dump:
ksedsts()+346<-ksdxfstk()+71<-ksdxcb()+912<-sspuser()+217<-__sighandler()<-read()+14<-ntpfprd()+141<-nsbasic_brc()+432<-nioqrc()+6340<-ttcdrv()+25337<-nioqwa()+73<-upirtrc()+1424<-upirtr()+160<-OCIKDispatch()+189<-kfnOpExecuteWithWaitInt()+4452<-kfnOpExecuteWithWait()+2931<-kfnbRun()+5324<-ksbrdp()+1079<-opirip()+609<-opidrv()+602<-sou2o()+145<-opimai_real()+202<-ssthrdmain()+417<-main()+262<-__libc_start_main()+253


........

PROCESS 39:
------------------------------
SO: 0x11bcbea70, type: 2, owner: (nil), flag: INIT/-/-/-/0x00 if: 0x3 c:
0x3
proc=0x11bcbea70, name=process, file=ksu.h LINE:15729, pg=0, conuid=0
(process) Oracle pid:39, ser:43, calls cur/top: 0x10308c3b8/0x10308c3b8
flags : (0x0) - icon_uid:0 logon_pdbid=0
flags2: (0x0), flags3: (0x510)
call error: 0, sess error: 0, txn error 0
intr queue: empty
(post info) last post received: 0 0 47
last post received-location: ksr2.h LINE:695 ID:ksrpublish
last process to post me: 0x11bcf9a80 170 0
last post sent: 0 0 48
last post sent-location: ksr2.h LINE:699 ID:ksrmdone
last process posted by me: 0x11bd0a380 186 0
waiter on post event: 0
(latch info) hold_bits=0x0 ud_influx=0x0
Process Group: DEFAULT, pseudo proc: 0x11a2eea20
O/S info: user: oracle, term: UNKNOWN, ospid: 287335
OSD pid info:

Short stack dump:
ksedsts()+346<-ksdxfstk()+71<-ksdxcb()+912<-sspuser()+217<-__sighandler()<-sem
timedop()+10<-skgpwwait()+200<-ksliwat()+2277<-kslwaitctx()+197<-kjoeq_rplt_wa
it_for_acks()+818<-kjoeq_rplt_operation()+2862<-kjoeq_get()+5173<-kffGetFAExt(
)+1353<-kfnsFillMapMsg()+1730<-kfnsUFG()+2118<-kfnsBackground()+5317<-kfnDispa
tchInt()+634<-opiodr()+1229<-ttcpip()+1257<-opitsk()+1940<-opiino()+941<-opiod
r()+1229<-opidrv()+1021<-sou2o()+145<-opimai_real()+455<-ssthrdmain()+417<-mai
n()+262<-__libc_start_main()+253
.
.

client details:
O/S info: user: oracle, term: , ospid: 286899 <=== DWM DB instance ASMB
^^^^^^^^^^^^^
machine: node1 program:
oracle@RACDB (TNS V1-V3)
application name: oracle@RACDB (TNS V1-V3), hash
value=1273305315

Current Wait Stack:
2: waiting for 'DLM enqueue copy completion' <<<<<<<<<<<<<<<<< foreground processes wait.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Owner ID=0x10006, op|restype=0x14641, id1|id2=0x10000014f
wait_id=98446701 seq_num=26776 snap_id=1

wait times: snap=2 min 14 sec, exc=2 min 14 sec, total=2 min 14 sec <====
wait times: max=infinite, heur=2 min 14 sec
wait counts: calls=4494 os=4494
in_wait=1 iflags=0x1520

1: waiting for 'ASM file metadata operation'
msgop=0xd, locn=0x2, =0x0
wait_id=98446700 seq_num=26775 snap_id=1
wait times: snap=0.000000 sec, exc=0.000169 sec, total=2 min 14 sec
wait times: max=infinite, heur=2 min 14 sec
wait counts: calls=0 os=0
in_wait=1 iflags=0x1520

.........


*** 2018-09-11T23:14:27.796153+02:00
PROCESS 51:
  ----------------------------------------
  SO: 0x11bccb130, type: 2, owner: (nil), flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
   proc=0x11bccb130, name=process, file=ksu.h LINE:15729, pg=0, conuid=0
  (process) Oracle pid:51, ser:116, calls cur/top: 0x105117318/0x105117318
            flags : (0x0) -  icon_uid:0 logon_pdbid=0
            flags2: (0x0),  flags3: (0x510)
            call error: 0, sess error: 0, txn error 0
            intr queue: empty
    (post info) last post received: 0 0 47
                last post received-location: ksr2.h LINE:695 ID:ksrpublish
                last process to post me: 0x11bce7060 22 0
                last post sent: 0 0 48
                last post sent-location: ksr2.h LINE:699 ID:ksrmdone
                last process posted by me: 0x11bce9180 18 0
                waiter on post event: 0
    (latch info) hold_bits=0x0 ud_influx=0x0
    Process Group: DEFAULT, pseudo proc: 0x11a2eea20
    O/S info: user: oracle, term: UNKNOWN, ospid: 345196
    OSD pid info:
    Short stack dump:
ksedsts()+346<-ksdxfstk()+71<-ksdxcb()+912<-sspuser()+217<-__sighandler()<-semtimedop()+10<-skgpwwait()+200<-ksliwat()+2292<-kslwaitctx()+197<-kjoeq_rplt_wait_for_acks()+818<-kjoeq_rplt_operation()+2862<-kjoeq_rls()+4644<-kffaDelete()+3179<-kss_del_cb()+242<-kssdel()+208<-kffilDelete()+3170<-kss_del_cb()+242<-kssdel()+208<-kfnsUFG()+3297<-kfnsBackground()+5317<-kfnDispatchInt()+740<-opiodr()+1229<-ttcpip()+1257<-opitsk()+1940<-opiino()+941<-opiodr()+1229<-opidrv()+1021<-sou2o()+145<-opimai_real()+455<-ssthrdmain()+417<-main()+262<-__libc_start_main()+253