RAC DB creation fails with error - LMON (ospid: XXXX): terminating the instance due to ORA error 481 (Doc ID 2626738.1)

LCK0 started with pid=9, OS id=24205

2020-01-06T11:11:06.444791+00:00
Using default pga_aggregate_limit of 8192 MB
2020-01-06T11:11:07.992683+00:00
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libafd19.so
2020-01-06T11:11:08.076400+00:00
NOTE: ASMB (index:0) (24168) connected to ASM instance +ASM1, osid: 24176 (Flex mode; client id 0x3ea40495851bb0d6)
NOTE: initiating MARK startup
Starting background process MARK
2020-01-06T11:11:08.103547+00:00
MARK started with pid=69, OS id=24240
2020-01-06T11:11:08.106891+00:00
NOTE: MARK has subscribed
2020-01-06T11:11:08.491491+00:00
KSXPPING: KSXP selected for Ping
2020-01-06T11:15:42.007558+00:00
LMS2 (ospid: 24033_24046) has detected no messaging activity from instance 2
LMS2 (ospid: 24033_24046) issues an IMR to resolve the situation
2020-01-06T11:15:42.017926+00:00
LMS1 (ospid: 24031_24045) has detected no messaging activity from instance 2================>No message from Instance 2
2020-01-06T11:15:42.018048+00:00
Please check LMS2 trace file for more detail.
2020-01-06T11:15:42.018189+00:00
Communications reconfiguration: instance_number 2 from LMS2 (ospid: 24033_24046)
2020-01-06T11:15:42.028538+00:00
LMS1 (ospid: 24031_24045) issues an IMR to resolve the situation
Please check LMS1 trace file for more detail.=====================>LMS1 trace files.
2020-01-06T11:15:42.030533+00:00
LMON (ospid: 24023) drops the IMR request from LMS1 (ospid: 24031_24045) because IMR is in progress and inst 2 is marked bad.
2020-01-06T11:15:42.230597+00:00
Dumping diagnostic data in directory=[cdmp_20200106111542], requested by (instance=2, osid=46790 (LMON)), summary=[abnormal instance termination].
2020-01-06T11:15:44.440937+00:00
Increasing priority of 3 RS
Reconfiguration started (old inc 3, new inc 7)
Dumping diagnostic data in directory=[cdmp_20200106111542], requested by (instance=2, osid=46790 (LMON)), summary=[abnormal instance termination].
2020-01-06T11:15:44.440937+00:00
Increasing priority of 3 RS
Reconfiguration started (old inc 3, new inc 7)
List of instances (total 1) :
1
Dead instances (total 1) :
2
My inst 1
publish big name space - dead or down/up instance detected, invalidate domain 0 ========>Dead instance detected.
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
 

2020-01-06T11:12:44.427118+00:00
DIA0 Critical Database Process As Root: Hang ID 1 blocks 1 sessions
Final blocker is session ID 316 serial# 61251 OSPID 46974 on Instance 2
If resolvable, instance eviction will be attempted by Hang Manager
2020-01-06T11:12:48.454801+00:00
DIA0 Critical Database Process As Root: Hang ID 2 blocks 1 sessions
Final blocker is session ID 316 serial# 36699 OSPID 24205 on Instance 1
If resolvable, instance eviction will be attempted by Hang Manager
2020-01-06T11:15:42.020293+00:00
LMS2 (ospid: 46800_46815) has detected no messaging activity from instance 1
2020-01-06T11:15:42.029248+00:00
Instance termination initiated by instance 1 with reason 1.===================================>Instance termination initiated by instance 1
Instance 1 received a reconfig event from its cluster manager indicating that this instance is supposed to be down
2020-01-06T11:15:42.029653+00:00
Please check instance 1's alert log and LMON trace file for more details.
2020-01-06T11:15:42.029746+00:00
LMS1 (ospid: 46798_46814) has detected no messaging activity from instance 1
Please also examine the CSS log files.
2020-01-06T11:15:42.030602+00:00
LMS2 (ospid: 46800_46815) issues an IMR to resolve the situation
Please check LMS2 trace file for more detail.
2020-01-06T11:15:42.039588+00:00
LMS1 (ospid: 46798_46814) issues an IMR to resolve the situation
Please check LMS1 trace file for more detail.
LMON (ospid: 46790): terminating the instance due to ORA error 481===========>Lmon termination with ORA -481
2020-01-06T11:15:42.076243+00:00
Cause - 'Instance is being terminated by instance 1 (reason 1 inst 2 uid 0x21dfd42)'=========>Instance 1 terminated
2020-01-06T11:15:42.230649+00:00
System state dump requested by (instance=2, osid=46790 (LMON)), summary=[abnormal instance termination]. error - 'Instance is terminating.
'
System State dumped to trace file /<ORACLE_BASE>diag/rdbms/<DB_UNIQUE_NAME>/<DB_INSTANCE2>/trace/<DB_INSTANCE2>_diag_xxxx.trc
2020-01-06T11:15:42.871608+00:00
Dumping diagnostic data in directory=[cdmp_20200106111542], requested by (instance=2, osid=46790 (LMON)), summary=[abnormal instance termination].
2020-01-06T11:15:43.691383+00:00
License high water mark = 1
2020-01-06T11:15:44.184861+00:00
Instance terminated by LMON, pid = 46790
2020-01-06T11:15:44.248917+00:00
Warning: 2 processes are still attacheded to shmid 8454149:
(size: 36864 bytes, creator pid: 46568, last attach/detach pid: 46798)
2020-01-06T11:15:44.692388+00:00
USER(prelim) (ospid: 50939): terminating the instance
2020-01-06T11:15:44.693247+00:00
Instance terminated by USER(prelim), pid = 50939

 

WAIT]:PROTO: [1578305485057750]ACNH 0x7f806c8f7308: 16 attempts to connect, timeout in 0:4:14.882.882820

IPCLW:[0.4]{-}[WAIT]:UTIL: [1578305485057750] ACNH 0x7f806c8f7308 State: 0 SMSN: 624517457 PKT(624517458.1802927115) # Pending: 1
IPCLW:[0.5]{-}[WAIT]:UTIL: [1578305485057750] Peer: [UNKNWN].0 AckSeq: 0
IPCLW:[0.6]{-}[WAIT]:UTIL: [1578305485057750] Flags: 0x00000000 IHint: 0x378fb02f0000001e THint: 0x0
IPCLW:[0.7]{-}[WAIT]:UTIL: [1578305485057750] Local Address: 169.254.xx.204:17022 Remote Address: 169.254.x.145:43201==============>Address ping
IPCLW:[0.8]{-}[WAIT]:UTIL: [1578305485057750] Remote PID: ver 0 flags 1 trans 2 tos 0 opts 0 xdata3 dfb7 xdata2 df07832a
IPCLW:[0.9]{-}[WAIT]:UTIL: [1578305485057750] : mmsz 8288 mmr 9200 mms 2 xdata 9b75db35
IPCLW:[0.10]{-}[WAIT]:UTIL: [1578305485057750] IVPort: 53181 TVPort: 56117 IMPT: 32322 RMPT: 57271 Pending Sends: Yes Unacked Sends: Yes ==================>LMS is having pending send and unacknowledged requests.
IPCLW:[0.11]{-}[WAIT]:UTIL: [1578305485057750] Send Engine Queued: No sshdl -1 ssts 0 rtts 0 snderrchk 0 creqcnt 16 credits 0/0
IPCLW:[0.12]{-}[WAIT]:UTIL: [1578305485057750] Unackd Messages 624517457 -> 624517457. SSEQ 1802927114 Send Time: INVALID TIME SMSN # Xmits: 0 EMSN INVALID TIME
IPCLW:[0.13]{-}[WAIT]:UTIL: [1578305485057750] Pending send queue:
IPCLW:[0.14]{-}[WAIT]:UTIL: [1578305485057750] [0] mbuf 0x7f806c6473b0 MSN 624517457 Seq 1802927114 -> 1802927115 # XMits: 0

*** 2020-01-06T10:12:13.162406+00:00 (CDB$ROOT(1))
IPCLW:[0.15]{E}[WAIT]:PROTO: [1578305533162356]ACNH 0x7f806c8f7308: 32 attempts to connect, timeout in 0:3:26.778.778214===========>timeout

*** 2020-01-06T10:12:54.012742+00:00 (CDB$ROOT(1))
2020-01-06 10:12:54.012 : GSIPC:PING: send PINGREQ[1] to 2.2 (seq 0.0) stm 0xc5cb00ea
2020-01-06 10:12:54.013 : GSIPC:PING: rcv'd PINGACK[1] from 2.2 elptm 695 usec

*** 2020-01-06T10:12:56.027396+00:00 (CDB$ROOT(1))
2020-01-06 10:12:56.027 : GSIPC:PING: rcv'd PINGREQ[1] from 2.2 (seq 0.0) stm 0xe388cd4c
2020-01-06 10:12:56.027 : GSIPC:PING: send PINGACK[1] to 2.2 (seq 0.0)

 

64 bytes from 169.254.x.145: icmp_seq=1 ttl=64 time=0.014 ms
64 bytes from 169.254.x.145: icmp_seq=2 ttl=64 time=0.054 ms
64 bytes from 169.254.x.145: icmp_seq=3 ttl=64 time=0.038 ms

^C
--- 169.254.5.145 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 6999ms
rtt min/avg/max/mdev = 0.014/0.025/0.054/0.013 ms
<HOSTNAME2>/root # ping 169.254.xx.228
PING 169.254.22.228 (169.254.22.228) 56(84) bytes of data.
64 bytes from 169.254.xx.228: icmp_seq=1 ttl=64 time=0.014 ms
64 bytes from 169.254.xx.228: icmp_seq=2 ttl=64 time=0.012 ms
64 bytes from 169.254.xx.228: icmp_seq=3 ttl=64 time=0.023 ms

--- 169.254.xx.228 ping statistics ---
11 packets transmitted, 11 received, 0% packet loss, time 9999ms
rtt min/avg/max/mdev = 0.010/0.015/0.027/0.007 ms