OCI: Linux instance hangs with iscsi connection timeout error "rejecting I/O to offline device" (Doc ID 2602573.1)

Last updated on FEBRUARY 26, 2020

Applies to:

Symptoms

Instance hangs and requires a reboot to bring it back online. In messages log following log entries were found before the reboot log entries.

[3615354.859794] sd 3:0:0:1: rejecting I/O to offline device
[3615354.862308] sd 3:0:0:1: rejecting I/O to offline device
[3615354.864738] sd 3:0:0:1: rejecting I/O to offline device
[3615354.867232] sd 3:0:0:1: rejecting I/O to offline device
[3615354.869706] sd 3:0:0:1: rejecting I/O to offline device
[3615354.872241] sd 3:0:0:1: rejecting I/O to offline device
[3615354.874702] sd 3:0:0:1: rejecting I/O to offline device
[3615354.877108] sd 3:0:0:1: rejecting I/O to offline device
[3615354.879534] sd 3:0:0:1: rejecting I/O to offline device
[3615354.881968] sd 3:0:0:1: rejecting I/O to offline device
[3615354.884393] sd 3:0:0:1: rejecting I/O to offline device
[3615354.886861] sd 3:0:0:1: rejecting I/O to offline device
[3615354.889262] XFS (dm-0): metadata I/O error: block 0x1f328010 ("xlog_iodone") error 5 numblks 64
[3615354.889304] sd 3:0:0:1: rejecting I/O to offline device
[3615354.889335] sd 3:0:0:1: rejecting I/O to offline device
[3615354.897866] XFS (dm-0): Log I/O Error Detected. Shutting down filesystem
[3615354.900602] XFS (dm-0): Please umount the filesystem and rectify the problem(s)
[3615354.903326] XFS (dm-0): metadata I/O error: block 0x1f328018 ("xlog_iodone") error 5 numblks 64
[3615354.906359] XFS (dm-0): metadata I/O error: block 0x1f328020 ("xlog_iodone") error 5 numblks 64
[3615354.909410] XFS (dm-0): metadata I/O error: block 0x1f328028 ("xlog_iodone") error 5 numblks 64
[3615354.912406] XFS (dm-0): metadata I/O error: block 0x1f328030 ("xlog_iodone") error 5 numblks 64
[3615354.915448] print_req_error: I/O error, dev sdb, sector 523407368
[3615354.917814] print_req_error: I/O error, dev sdb, sector 785083744
[3615354.920188] XFS (dm-0): metadata I/O error: block 0x1f328038 ("xlog_iodone") error 5 numblks 64
[3615354.923766] XFS (dm-0): metadata I/O error: block 0x1f328040 ("xlog_iodone") error 5 numblks 64
[3615354.927368] XFS (dm-0): metadata I/O error: block 0x1f328008 ("xlog_iodone") error 5 numblks 64
rhnsd[5798]: /etc/sysconfig/rhn/up2date does not contain valid systemIdPath entry
[3628602.763673] systemd-logind[1398]: Failed to execute operation: Connection timed out
[3628757.732200] sd 2:0:0:1: timing out command, waited 180s
[3628757.735806] print_req_error: I/O error, dev sda, sector 1962016
[3628757.738407] Read-error on swap-device (8:0:1962024)
[3628757.740443] Read-error on swap-device (8:0:1962032)
[3628757.743005] Read-error on swap-device (8:0:1962040)
[3628757.745002] Read-error on swap-device (8:0:1962048)

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.