My Oracle Support Banner

Exadata: ASM Diskgroups Cannot Be Mounted Due to ORA-56864 (LIBCELL network error) and ORA-56841 (Master Diskmon cannot connect to a CELL) After Adding New Cell Servers in the Exadata Cluster Configuration (Doc ID 2205555.1)

Last updated on AUGUST 04, 2018

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.1 to 12.1.0.2 [Release 12.1]
Oracle Exadata Storage Server Software - Version 12.1.1.1.0 to 12.1.2.2.3 [Release 12.1]
Information in this document applies to any platform.

Symptoms

1) ASM diskgroups cannot be mounted after adding new cell servers in the Exadata cluster configuration due to the ORA-56864 and ORA-56841 errors:

.
.
.
Wed Nov 16 14:12:55 2016
NOTE: Diskgroup used for Voting files is:
DBFSDG
Diskgroup with spfile:DBFSDG
NOTE: Diskgroup used for OCR is:DBFSDG
NOTE: Diskgroups listed in ASM_DISKGROUP are
DATADG

RECODG
.
.
.

Wed Nov 16 15:56:55 2016
Using default pga_aggregate_limit of 3072 MB
Wed Nov 16 15:57:05 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dskm_39756.trc:
ORA-56864: Master Diskmon "ADD CELL o/192.168.55.88;192.168.55.89" operation cannot complete because of a LIBCELL network error
ORA-56841: Master Diskmon cannot connect to a CELL
Wed Nov 16 15:57:16 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dskm_39756.trc:
ORA-56864: Master Diskmon "ADD CELL o/192.168.55.88;192.168.55.89" operation cannot complete because of a LIBCELL network error
ORA-56841: Master Diskmon cannot connect to a CELL
Wed Nov 16 15:57:16 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dskm_39756.trc:
ORA-56864: Master Diskmon "ADD CELL o/192.168.55.88;192.168.55.89" operation cannot complete because of a LIBCELL network error
ORA-56841: Master Diskmon cannot connect to a CELL
Wed Nov 16 15:57:16 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dskm_39756.trc:
ORA-56864: Master Diskmon "ADD CELL o/192.168.55.88;192.168.55.89" operation cannot complete because of a LIBCELL network error
ORA-56841: Master Diskmon cannot connect to a CELL
Wed Nov 16 15:57:16 2016
.
.
.
Wed Nov 16 15:57:17 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dskm_39756.trc:
ORA-56864: Master Diskmon "ADD CELL o/192.168.55.88;192.168.55.89" operation cannot complete because of a LIBCELL network error
ORA-56841: Master Diskmon cannot connect to a CELL
Wed Nov 16 15:57:17 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_39798.trc:
WARNING: Read Failed. group:0 disk:3 AU:0 offset:0 size:4096
path:Unknown disk
incarnation:0xe969f792 asynchronous result:'I/O error'
subsys:Unknown library krq:0x7f623c1b8ff8 bufp:0x7f623c1b7000 osderr1:0x0 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
Wed Nov 16 15:57:17 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dskm_39756.trc:
ORA-56864: Master Diskmon "ADD CELL o/192.168.55.88;192.168.55.89" operation cannot complete because of a LIBCELL network error
ORA-56841: Master Diskmon cannot connect to a CELL
Wed Nov 16 15:57:17 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_39798.trc:
WARNING: Read Failed. group:0 disk:4 AU:0 offset:0 size:4096
path:Unknown disk
incarnation:0xe969f793 asynchronous result:'I/O error'
subsys:Unknown library krq:0x7f623c1b6ba8 bufp:0x7f623b708000 osderr1:0x0 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
Wed Nov 16 15:57:17 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dskm_39756.trc:
ORA-56864: Master Diskmon "ADD CELL o/192.168.55.88;192.168.55.89" operation cannot complete because of a LIBCELL network error
ORA-56841: Master Diskmon cannot connect to a CELL
Wed Nov 16 15:57:17 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_39798.trc:
WARNING: Read Failed. group:0 disk:5 AU:0 offset:0 size:4096
path:Unknown disk
incarnation:0xe969f794 asynchronous result:'I/O error'
subsys:Unknown library krq:0x7f623c1b6788 bufp:0x7f623b706000 osderr1:0x0 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
.
.
.
Wed Nov 16 15:58:15 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_39798.trc:
ORA-15080: synchronous I/O operation failed to read block 0 of disk 3 in disk group
ORA-15080: synchronous I/O operation failed to read block 0 of disk 2 in disk group
ORA-15080: synchronous I/O operation failed to read block 0 of disk 1 in disk group
ORA-15080: synchronous I/O operation failed to read block 0 of disk 0 in disk group
Wed Nov 16 15:58:15 2016
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dskm_39756.trc:
ORA-56864: Master Diskmon "ADD CELL o/192.168.55.86;192.168.55.87" operation cannot complete because of a LIBCELL network error
ORA-56841: Master Diskmon cannot connect to a CELL
.
.
.
.
.

 

2) Error description:

ORA-56864, 00000, "Master Diskmon \"%s%s\" operation cannot complete because of a LIBCELL network error"
// *Document: YES
// *Cause: A LIBCELL operation between Master Diskmon and a CELL failed because of a network error.
// *Action: Look in the $ORA_CRS_HOME/log/<hostname>/diskmon/diskmon.log file for additional error information.

 

ORA-56841, 00000, "Master Diskmon cannot connect to a CELL"
// *Document: YES
// *Cause: A LIBCELL operation from Master Diskmon failed to create a network connection to the target CELL.
// *Action: Look in the $ORA_CRS_HOME/log/<hostname>/diskmon/diskmon.log file for additional error information.

 

3) Clusterware alert.log from the affected compute node(s) is reporting connectivity problems to the cell server(s):

.
.
Wed Nov 16 18:04:18 2016
ossnet_connect_to_box: Giving up on box 192.168.55.88 as retry limit (7) reached.
Wed Nov 16 18:04:30 2016
ossnet_connect_to_box: Giving up on box 192.168.55.88 as retry limit (7) reached.
ossnet_connect_to_box: Giving up on box 192.168.55.86 as retry limit (7) reached.
ossnet_connect_to_box: Giving up on box 192.168.55.88 as retry limit (7) reached.
ossnet_connect_to_box: Giving up on box 192.168.55.84 as retry limit (7) reached.
ossnet_connect_to_box: Giving up on box 192.168.55.86 as retry limit (7) reached.
ossnet_connect_to_box: Giving up on box 192.168.55.86 as retry limit (7) reached.
ossnet_connect_to_box: Giving up on box 192.168.55.88 as retry limit (7) reached.
ossnet_connect_to_box: Giving up on box 192.168.55.88 as retry limit (7) reached.
ossnet_connect_to_box: Giving up on box 192.168.55.84 as retry limit (7) reached.
ossnet_connect_to_box: Giving up on box 192.168.55.84 as retry limit (7) reached.
.
.

 

4) diskmon log is reporting the next connectivity errors as well:

======================================================

Trace file /u01/app/oracle/diag/crs/asmcloud1/crs/trace/diskmon.trc

======================================================

.
.
.
2016-11-16 18:06:02.022691 : DISKMON:3546650368: SKGXP:[7f5fd81cd390.4]{E}: skgxp_add_ip: normalant - Subnet mask for each subnet in the ipaddressX parameters in cellinit.ora must be the same on both this node as well as 192.168.55.84. (markup_portid)
2016-11-16 18:06:02.022694 : DISKMON:3546650368: SKGXP:[7f5fd81cd390.5]{E}: map_add_ip FAILED: nsm=0x7f5fd81d3504 ip=192.168.55.84 loc=markup_portid
2016-11-16 18:06:02.022697 : DISKMON:3546650368: dskm_ant_rsc_monitor_start8: : error 1 at location skgxpaddip - Subnet mask for each subnet in the ipaddressX parameters in cellinit.ora must be the same on both this node as well as 192.168.55.84. (markup_portid).
.
.
2016-11-16 18:06:14.051778 : DISKMON:7395072: dskm_send_nack1: ercode = 56841
2016-11-16 18:06:14.051783 : DISKMON:7395072: dskm_send_nack4: done
2016-11-16 18:06:14.051863 : DISKMON:3552564992: dskm_node_guids_are_offline: query SM done. retcode = 56891(REACHABLE)
CLSB:3552564992: Oracle Clusterware infrastructure error in DISKMON (OS PID 25794): Fatal signal 11 has occurred in program diskmon thread 3552564992; nested signal count is 1
2016-11-16 18:06:14.051986 : DISKMON:7395072: dskm_process_msg5: received msg type KGZM_GET_OSSID (0x0002)
2016-11-16 18:06:14.051990 : DISKMON:7395072: dskm_proc_get_ossid3: sage oss devname to add is o/192.168.55.84;192.168.55.85
2016-11-16 18:06:14.051992 : DISKMON:7395072: dskm_proc_get_ossidx13: sage oss devname (Exadata) to add is o/192.168.55.84;192.168.55.85
2016-11-16 18:06:14.051994 : DISKMON:7395072: dskm_get_ossb: dead cell: o/192.168.55.84;192.168.55.85
2016-11-16 18:06:14.052007 : DISKMON:7395072: dskm_proc_get_ossid4: dskm_get_ossb failed for device o/192.168.55.84;192.168.55.85 with error 56841
2016-11-16 18:06:14.052010 : DISKMON:7395072: dskm_process_msg7: processed msg 124 type KGZM_GET_OSSID (0x0002), retcode 56841
2016-11-16 18:06:14.052015 : DISKMON:7395072: dskm_process_msg8: error 56841 processing msg type KGZM_GET_OSSID (0x0002) from client +ASM1/ASM/39530, reid cid=c0dbd1dc6eaeef11bfb21eceec85a928,icin=374785424,nmn=1,lnid=374785424,gid=7,gin=1,gmn=0,umemid=1,opid=10,opsn=1,lvl=process hdr=0x 0
2016-11-16 18:06:14.052032 : DISKMON:7395072: dskm_send_nack1: ercode = 56841
.
.

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
 Community Discussions Exadata


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.