ODA:CRS/ASM Is Down an I/O error occurred for voting file ERROR: -9(Error 27072, OS Error (Linux-x86_64 Error: 5: Input/output error (Doc ID 2447658.1)

Last updated on APRIL 17, 2023

Applies to:

Symptoms

+An I/O error which is causing CSSD to fail while starting the cluster

 File Cluster Alert log: 

2018-08-24 18:54:33.349 [OCSSD(47149)]CRS-1649: An I/O error occurred for voting file: /dev/mapper/HDD_E0_S12_1xxxxxxxx8p1; details at (:CSSNM00059:) in <GRID_ORA_BASE>/diag/crs/<HOSTNAME>/crs/trace/ocssd.trc.

2018-08-24 18:55:03.352 [OCSSD(47149)]CRS-1649: An I/O error occurred for voting file: /dev/mapper/HDD_E0_S12_1xxxxxxxx8p1; details at (:CSSNM00059:) in <GRID_ORA_BASE>/diag/crs/<HOSTNAME>/crs/trace/ocssd.trc.

2018-08-24 18:55:33.355 [OCSSD(47149)]CRS-1649: An I/O error occurred for voting file: /dev/mapper/HDD_E0_S12_1xxxxxxxx8p1; details at (:CSSNM00059:) in <GRID_ORA_BASE>/diag/crs/<HOSTNAME>/crs/trace/ocssd.trc.

2018-08-24 18:55:58.717 [CSSDAGENT(47136)]CRS-5818: Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in <GRID_ORA_BASE>/diag/crs/<HOSTNAME>/crs/trace/ohasd_cssdagent_root.trc.

2018-08-24 18:55:58.806 [OHASD(3051)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00221:) {0:0:2} in <GRID_ORA_BASE>/diag/crs/<HOSTNAME>/crs/trace/ohasd.trc.

2018-08-24 18:55:59.717 [OCSSD(47149)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in <GRID_ORA_BASE>/diag/crs/<HOSTNAME>/crs/trace/ocssd.trc

2018-08-24 18:55:59.718 [OCSSD(47149)]CRS-1652: Starting clean up of CRSD resources.

2018-08-24 18:55:59.722 [OCSSD(47149)]CRS-1653: The clean up of the CRSD resources failed.

2018-08-24 18:55:59.722 [OCSSD(47149)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 47149 experienced fatal signal or exception code 6.

2018-08-24T18:55:59.727025-04:00

Errors in file <GRID_ORA_BASE>/diag/crs/<HOSTNAME>/crs/trace/ocssd.trc (incident=57):

CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []

Incident details in: <GRID_ORA_BASE>/diag/crs/<HOSTNAME>/crs/incident/incdir_57/ocssd_i57.trc

+From the ocssd trace OS errors triggered around that time :

File:: ocssd.trc

2018-08-24 13:47:02.026 : SKGFD:1145505536: ERROR: -9(Error 27072, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 131089
Additional information: 4294967295)
)
2018-08-24 13:47:02.026 : CSSD:1145505536: (:CSSNM00059:)clssnmvWriteBlocks: write failed at offset 17 of /dev/mapper/HDD_E0_S12_1xxxxxxxx8p1
2018-08-24 13:47:02.026 : CSSD:1145505536: clssnmlio_opthr: disk /dev/mapper/HDD_E0_S12_1xxxxxxxx8p1 write failure, status 0, top 0x7ff944462300
2018-08-24 13:47:02.041 : CSSD:1147082496: clssscthrdmain: Terminating thread clssnmvLeaseAqIoThread

+OS /var/log/messages:

Aug 24 13:42:31 <HOSTNAME> kernel: [ 3372.691700] F 4298047.349/180824174231 ocssd.bin[81818] oracleafd:18:0937:Write IO to ASM managed device: [249] [49]
Aug 24 13:52:28 <HOSTNAME> kernel: [ 3967.821518] F 4298643.877/180824175228 ocssd.bin[81818] oracleafd:18:0937:Write IO to ASM managed device [Repeat:23]: [249] [49]
Aug 24 17:32:11 <HOSTNAME> Oracle Clusterware: 2018-08-24 17:32:11.256#012[(68956)]CRS-8500:Oracle Clusterware OSYSMOND process is starting with operating system process ID xxxx
Aug 24 17:32:15 <HOSTNAME> kernel: [17124.364786] F 4311831.327/180824213215 ocssd.bin[69026] oracleafd:18:0937:Write IO to ASM managed device: [249] [49]
Aug 24 17:42:12 <HOSTNAME> kernel: [17719.815242] F 4312428.176/180824214212 ocssd.bin[69026] oracleafd:18:0937:Write IO to ASM managed device [Repeat:23]: [249] [49]
Aug 24 18:02:27 <HOSTNAME> Oracle Clusterware: 2018-08-24 18:02:27.742#012[(9640)]CRS-8500:Oracle Clusterware OSYSMOND process is starting with operating system process ID xxxx

[root@<HOSTNAME> ~]# crsctl query css votedisk

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. OFFLINE e1711103bxxxxxxxxfead443465be05f (/dev/mapper/HDD_E0_S08_1746055780p1) [DATA]

2. OFFLINE 40d07f2c3xxxxxxxxfd42f2f1231633f (/dev/mapper/HDD_E0_S11_1745993816p1) [DATA]

3. OFFLINE b5036c398xxxxxxxxfe6e68a7413fb42 (/dev/mapper/HDD_E0_S12_1xxxxxxxx8p1) [DATA]

But When cluster is started in exclusive mode using "crsctl start crs -excl -nocrs", we could see the ASM disks are online ,cached and member.

And asm_distring parameter is using AFD ( ASM filter driver)

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.