OAK-10021:Failure while running storage setup on the system (Doc ID 2911616.1)

[root@oak1 ~]$ odacli describe-job -i <job id>

Job details
----------------------------------------------------------------
ID: <job id>
Description: Provisioning service creation
Status: Failure
Created: <timestamp>
Message: DCS-10001:Internal error encountered:OAK-10021:Failure while running storage setup on the system. Cause: OAK failed in storage discovery. Action: Run cleanup and retry again. If issues persist contact Oracle support..

2022-11-17 04:07:31,111 DEBUG [Thread-8575] [] c.o.d.c.u.CommonsUtils: Output :
OAKD still hasn't discovered storage
2022-11-17 04:07:31,111 DEBUG [Thread-8575] [] c.o.d.c.u.CommonsUtils: Output : EMPTY CONTENT
2022-11-17 04:07:31,111 DEBUG [Thread-8575] [] c.o.d.c.u.CommonsUtils: Output :
OAKD failed to discover storage
2022-11-17 04:07:31,111 DEBUG [Thread-8575] [] c.o.d.c.u.CommonsUtils: Output : EMPTY CONTENT
2022-11-17 04:07:31,111 DEBUG [Thread-8575] [] c.o.d.c.u.CommonsUtils: Output :
errMsg = OAK-10021:Failure while running storage setup on the system.
2022-11-17 04:07:31,111 DEBUG [Thread-8575] [] c.o.d.c.u.CommonsUtils: Output :
// *Cause: OAK failed in storage discovery.
2022-11-17 04:07:31,111 DEBUG [Thread-8575] [] c.o.d.c.u.CommonsUtils: Output :
// *Action: Run cleanup and retry again. If issues persist contact Oracle
2022-11-17 04:07:31,111 DEBUG [Thread-8575] [] c.o.d.c.u.CommonsUtils: Output :
// support.
2022-11-17 04:07:31,111 DEBUG [Thread-8575] [] c.o.d.c.u.CommonsUtils: Output : EMPTY CONTENT
2022-11-17 04:07:31,111 DEBUG [Thread-8575] [] c.o.d.c.u.CommonsUtils: Output : EMPTY CONTENT
2022-11-17 04:07:31,111 DEBUG [Thread-8575] [] c.o.d.c.u.CommonsUtils: Output :
Exiting....
2022-11-17 04:07:31,118 DEBUG [Discovering DiskGroup : JobId=efead330-5311-46c0-b5ea-afe74fb4e359] [] c.o.d.c.u.c.DCSProcessBuilder: Return code: 1
2022-11-17 04:07:31,118 DEBUG [Discovering DiskGroup : JobId=efead330-5311-46c0-b5ea-afe74fb4e359] [] c.o.d.c.u.c.CommandExecutor: Return code: 1
2022-11-17 04:07:31,126 DEBUG [Discovering DiskGroup : JobId=efead330-5311-46c0-b5ea-afe74fb4e359] [] c.o.d.c.u.CommonsUtils: Output :
Verifying storage topology,
it may take few minutes...
Verification of storage topology is successful...

..

Provisioning the system in 'default' mode.

Stopping OAKD

OAKD pid = 89116
OAKD still hasn't discovered storage
OAKD still hasn't discovered storage
OAKD still hasn't discovered storage
OAKD still hasn't discovered storage
OAKD still hasn't discovered storage
OAKD still hasn't discovered storage
OAKD still hasn't discovered storage
OAKD still hasn't discovered storage
OAKD still hasn't discovered storage
OAKD still hasn't discovered storage
OAKD still hasn't discovered storage
OAKD still hasn't discovered storage

OAKD failed to discover storage

errMsg = OAK-10021:Failure while running storage setup on the system.
// *Cause: OAK failed in storage discovery.
// *Action: Run cleanup and retry again. If issues persist contact Oracle
// support.

Node 1:

[root@oak1 ~]# ps -ef |grep oak
root 4643 1 0 Nov17 ? 00:00:00 /bin/sh
/etc/rc.d/init.d/init.oak run >/dev/null 2>&1 </dev/null
root 29530 69999 0 23:01 pts/2 00:00:00 grep --color=auto oak
root 89116 4643 0 Nov17 ? 00:00:44 /opt/oracle/oak/bin/oakd -non-cluster

[root@oak1 ~]# odaadmcli show ismaster
OAKD is in Slave Mode

Node 2:

[root@oak2 ~]# ps -ef |grep oak
root 4640 1 0 Nov17 ? 00:00:00 /bin/sh
/etc/rc.d/init.d/init.oak run >/dev/null 2>&1 </dev/null
root 5794 4640 0 Nov17 ? 00:00:44 /opt/oracle/oak/bin/oakd -non-cluster
root 14828 10243 0 23:01 pts/0 00:00:00 grep --color=auto oak

[root@oak2 ~]# odaadmcli show ismaster
OAKD is in Slave Mode

Node 1:

[root@oak1 ~]# time bnxtnvm listdev

Oracle dual 25Gb Ethernet Adapter or dual 10Gb Ethernet Adapter #1
Device Interface Name : p1p1
MACAddress : b0:26:28:c6:d0:80
PCI Device Name : 0000:af:00.0

real 2m23.738s                                        <<<<<< Here
user 2m23.449s
sys 0m0.051s

[root@oak1 ~]# date;fwr_exp_util list;date;
Sat Nov 19 13:05:13 HKT 2022
0 Controller [/SYS/MB/PCIE3/SAS2] => Expander[1949NMQ80P - Primary, wwn: 508002000252a17e, name: Oracle Database Appliance - E0 LED: OFF]
1 Controller [/SYS/MB/PCIE8/SAS3] => Expander[1949NMQ80P - Secondary, wwn: 508002000252b5fe, name: Oracle Database Appliance - E0 LED: OFF]
Sat Nov 19 13:12:27 HKT 2022                        <<<<<< Here

 

Node 2:

[root@oak2 ~]# time bnxtnvm listdev

Oracle dual 25Gb Ethernet Adapter or dual 10Gb Ethernet Adapter #1

Device Interface Name : p1p1
MACAddress : b0:26:28:c6:dc:50
PCI Device Name : 0000:af:00.0

real 0m0.133s
user 0m0.037s
sys 0m0.053s

[root@oak2 ~]# date;fwr_exp_util list;date;
Sat Nov 19 13:05:16 HKT 2022
0 Controller [/SYS/MB/PCIE3/SAS2] => Expander[1949NMQ80P - Primary, wwn: 508002000252a17e, name: Oracle Database Appliance - E0 LED: OFF]
1 Controller [/SYS/MB/PCIE8/SAS3] => Expander[1949NMQ80P - Secondary, wwn: 508002000252b5fe, name: Oracle Database Appliance - E0 LED: OFF]
Sat Nov 19 13:05:18 HKT 2022