My Oracle Support Banner

GEN0 Terminating The Instance Due To Error 15082 Since ASMB Is Not Moving For More Than 4 Minutes (Doc ID 2450132.1)

Last updated on OCTOBER 05, 2018

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Information in this document applies to any platform.

Symptoms

The issue may start with one instance which might have some issues and then eventually it leads to database instances getting terminated on the remaining nodes.

Gen0 terminates database instance due to error 15082

 1. Database alertlog

Starting background process RSM0
Wed Sep 12 06:27:32 2018
RSM0 started with pid=53, OS id=345670
Wed Sep 12 06:27:42 2018
NOTE: ASMB registering with ASM instance as Flex client 0x1002f (reg:1825550931) (reconnect)
Wed Sep 12 06:27:59 2018
ERROR: terminating instance because ASMB is stuck for 243 seconds
GEN0 (ospid: 311815): terminating the instance due to error 15082
Wed Sep 12 06:27:59 2018
System state dump requested by (instance=1, osid=311815 (GEN0)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/ORCLsty/ORCL/trace/ORCL_diag_311821_20180912062759.trc
Wed Sep 12 06:28:00 2018
Dumping diagnostic data in directory=[cdmp_20180912062759], requested by (instance=1, osid=311815 (GEN0)), summary=[abnormal instance termination].
Wed Sep 12 06:28:00 2018
Instance terminated by GEN0, pid = 311815
Wed Sep 12 06:28:00 2018
License high water mark = 18
Wed Sep 12 06:28:01 2018
USER (ospid: 346215): terminating the instance
Wed Sep 12 06:28:01 2018
Instance terminated by USER, pid = 34621

2. systemstate Dump

PROCESS 6: GEN0
  ----------------------------------------
  SO: 0x11bc9c7e0, type: 2, owner: (nil), flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
   proc=0x11bc9c7e0, name=process, file=ksu.h LINE:15729, pg=0, conuid=0
  (process) Oracle pid:6, ser:1, calls cur/top: 0x11949aa60/0x11949aa60
            flags : (0x6) SYSTEM  icon_uid:0 logon_pdbid=0
            flags2: (0x0),  flags3: (0x10)
            call error: 0, sess error: 0, txn error 0
            intr queue: empty
    (post info) last post received: 0 0 182
                last post received-location: kjxgn1.h LINE:262 ID:kjxgnaef
                last process to post me: 0x11bca6d80 1 6
                last post sent: 0 0 175
                last post sent-location: kjc.h LINE:2308 ID:KJCS Post snd proxy to flush msg
                last process posted by me: 0x11bca7e10 1 6
                waiter on post event: 0
    (latch info) hold_bits=0x0 ud_influx=0x0
    Process Group: DEFAULT, pseudo proc: 0x11a2eea20
    O/S info: user: grid, term: UNKNOWN, ospid: 30404
    OSD pid info:
    Short stack dump:
ksedsts()+346<-ksdxfstk()+71<-ksdxcb()+912<-sspuser()+217<-__sighandler()<-semtimedop()+10<-skgpwwait()+200<-ksliwat()+2292<-kslwaitctx()+197<-kjoer_rplt_wait_for_acks()+778<-kjoer_rplt_operation()+2291<-kjoer_deregister()+3865<-kfnrclDelete()+4261<-kss_del_cb()+242<-kssdel()+208<-kfnrclKill()+2083<-kfnrcl_gc()+1305<-kfnTimeout()+450<-ksbcti()+247<-ksbabs()+11524<-ksbrdp()+1079<-opirip()+609<-opidrv()+602<-sou2o()+145<-opimai_real()+202<-ssthrdmain()+417<-main()+262<-__libc_start_main()+253

......
........

PROCESS 29: ASMB
  ----------------------------------------
  SO: 0x45cce9948, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
   proc=0x45cce9948, name=process, file=ksu.h LINE:13949, pg=0 conuid=0
  (process) Oracle pid:29, ser:1, calls cur/top: 0x45d4d2120/0x45d4d2120
            flags : (0x6) SYSTEM  icon_uid:0
            flags2: (0x0),  flags3: (0x10)
            intr error: 0, call error: 0, sess error: 0, txn error 0
            intr queue: empty
    ksudlp FALSE at location: 0
    (post info) last post received: 1092 256 12
                last post received-location: ksu.h LINE:15435 ID:ksusig
                last process to post me: 0x45ccdf568 1 6
                last post sent: 0 0 439
                last post sent-location: kffm2.h LINE:516 ID:kffmLoad
                last process posted by me: 0x45ccf2f80 3 0
                waiter on post event: 0
    (latch info) hold_bits=0x0
    Process Group: DEFAULT, pseudo proc: 0x45f0e41d8
    O/S info: user: oracle, term: UNKNOWN, ospid: 345186 (DEAD)

..........
..........



Current Wait Stack:
     0: waiting for 'ASM background timer'
        =0x0, =0x0, =0x0
        wait_id=168634755 seq_num=13285 snap_id=1
        wait times: snap=4 min 2 sec, exc=4 min 2 sec, total=4 min 2 sec               <<<<<<<<< 
        wait times: max=infinite, heur=4 min 2 sec
        wait counts: calls=0 os=0
        in_wait=1 iflags=0x520


.........
.........

PROCESS 34: ASMB
  ----------------------------------------
  SO: 0x11bcb97a0, type: 2, owner: (nil), flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
   proc=0x11bcb97a0, name=process, file=ksu.h LINE:15729, pg=0, conuid=0
  (process) Oracle pid:34, ser:2, calls cur/top: 0x119499b48/0x119499b48
            flags : (0x6) SYSTEM  icon_uid:0 logon_pdbid=0
            flags2: (0x0),  flags3: (0x10)
            call error: 0, sess error: 0, txn error 0
            intr queue: empty
    (post info) last post received: 0 0 0
                last post received-location: No post
                last process to post me: none
                last post sent: 1681 0 3
                last post sent-location: ksl2.h LINE:4497 ID:kslpsr
                last process posted by me: 0x11bcbb8c0 4 0
                waiter on post event: 0
    (latch info) hold_bits=0x0 ud_influx=0x0
    Process Group: DEFAULT, pseudo proc: 0x11a2eea20
    O/S info: user: grid, term: UNKNOWN, ospid: 30650
    OSD pid info:
    Short stack dump:
ksedsts()+346<-ksdxfstk()+71<-ksdxcb()+912<-sspuser()+217<-__sighandler()<-read()+14<-ntpfprd()+141<-nsbasic_brc()+432<-nioqrc()+6340<-ttcdrv()+25337<-nioqwa()+73<-upirtrc()+1424<-upirtr()+160<-OCIKDispatch()+189<-kfnOpExecuteWithWaitInt()+4452<-kfnOpExecuteWithWait()+2931<-kfnbRun()+5324<-ksbrdp()+1079<-opirip()+609<-opidrv()+602<-sou2o()+145<-opimai_real()+202<-ssthrdmain()+417<-main()+262<-__libc_start_main()+253


........

PROCESS 39:
------------------------------
SO: 0x11bcbea70, type: 2, owner: (nil), flag: INIT/-/-/-/0x00 if: 0x3 c:
0x3
proc=0x11bcbea70, name=process, file=ksu.h LINE:15729, pg=0, conuid=0
(process) Oracle pid:39, ser:43, calls cur/top: 0x10308c3b8/0x10308c3b8
flags : (0x0) - icon_uid:0 logon_pdbid=0
flags2: (0x0), flags3: (0x510)
call error: 0, sess error: 0, txn error 0
intr queue: empty
(post info) last post received: 0 0 47
last post received-location: ksr2.h LINE:695 ID:ksrpublish
last process to post me: 0x11bcf9a80 170 0
last post sent: 0 0 48
last post sent-location: ksr2.h LINE:699 ID:ksrmdone
last process posted by me: 0x11bd0a380 186 0
waiter on post event: 0
(latch info) hold_bits=0x0 ud_influx=0x0
Process Group: DEFAULT, pseudo proc: 0x11a2eea20
O/S info: user: oracle, term: UNKNOWN, ospid: 287335
OSD pid info:

Short stack dump:
ksedsts()+346<-ksdxfstk()+71<-ksdxcb()+912<-sspuser()+217<-__sighandler()<-sem
timedop()+10<-skgpwwait()+200<-ksliwat()+2277<-kslwaitctx()+197<-kjoeq_rplt_wa
it_for_acks()+818<-kjoeq_rplt_operation()+2862<-kjoeq_get()+5173<-kffGetFAExt(
)+1353<-kfnsFillMapMsg()+1730<-kfnsUFG()+2118<-kfnsBackground()+5317<-kfnDispa
tchInt()+634<-opiodr()+1229<-ttcpip()+1257<-opitsk()+1940<-opiino()+941<-opiod
r()+1229<-opidrv()+1021<-sou2o()+145<-opimai_real()+455<-ssthrdmain()+417<-mai
n()+262<-__libc_start_main()+253
.
.

client details:
O/S info: user: oracle, term: , ospid: 286899 <=== DWM DB instance ASMB
^^^^^^^^^^^^^
machine: rac1 program:
oracle@erac1e (TNS V1-V3)
application name: oracle@erac1 (TNS V1-V3), hash
value=1273305315

Current Wait Stack:
2: waiting for 'DLM enqueue copy completion' <<<<<<<<<<<<<<<<< foreground processes wait.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Owner ID=0x10006, op|restype=0x14641, id1|id2=0x10000014f
wait_id=98446701 seq_num=26776 snap_id=1

wait times: snap=2 min 14 sec, exc=2 min 14 sec, total=2 min 14 sec <====
wait times: max=infinite, heur=2 min 14 sec
wait counts: calls=4494 os=4494
in_wait=1 iflags=0x1520

1: waiting for 'ASM file metadata operation'
msgop=0xd, locn=0x2, =0x0
wait_id=98446700 seq_num=26775 snap_id=1
wait times: snap=0.000000 sec, exc=0.000169 sec, total=2 min 14 sec
wait times: max=infinite, heur=2 min 14 sec
wait counts: calls=0 os=0
in_wait=1 iflags=0x1520


.........


*** 2018-09-11T23:14:27.796153+02:00
PROCESS 51:
  ----------------------------------------
  SO: 0x11bccb130, type: 2, owner: (nil), flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
   proc=0x11bccb130, name=process, file=ksu.h LINE:15729, pg=0, conuid=0
  (process) Oracle pid:51, ser:116, calls cur/top: 0x105117318/0x105117318
            flags : (0x0) -  icon_uid:0 logon_pdbid=0
            flags2: (0x0),  flags3: (0x510)
            call error: 0, sess error: 0, txn error 0
            intr queue: empty
    (post info) last post received: 0 0 47
                last post received-location: ksr2.h LINE:695 ID:ksrpublish
                last process to post me: 0x11bce7060 22 0
                last post sent: 0 0 48
                last post sent-location: ksr2.h LINE:699 ID:ksrmdone
                last process posted by me: 0x11bce9180 18 0
                waiter on post event: 0
    (latch info) hold_bits=0x0 ud_influx=0x0
    Process Group: DEFAULT, pseudo proc: 0x11a2eea20
    O/S info: user: oracle, term: UNKNOWN, ospid: 345196
    OSD pid info:
    Short stack dump:
ksedsts()+346<-ksdxfstk()+71<-ksdxcb()+912<-sspuser()+217<-__sighandler()<-semtimedop()+10<-skgpwwait()+200<-ksliwat()+2292<-kslwaitctx()+197<-kjoeq_rplt_wait_for_acks()+818<-kjoeq_rplt_operation()+2862<-kjoeq_rls()+4644<-kffaDelete()+3179<-kss_del_cb()+242<-kssdel()+208<-kffilDelete()+3170<-kss_del_cb()+242<-kssdel()+208<-kfnsUFG()+3297<-kfnsBackground()+5317<-kfnDispatchInt()+740<-opiodr()+1229<-ttcpip()+1257<-opitsk()+1940<-opiino()+941<-opiodr()+1229<-opidrv()+1021<-sou2o()+145<-opimai_real()+455<-ssthrdmain()+417<-main()+262<-__libc_start_main()+253

 

Changes

 None

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.