OCI kernel: connectionX:0: detected conn error (1020) (Doc ID 2677308.1)

Last updated on JULY 15, 2022

Applies to:

Symptoms

On OL7 instances iscsid found not trying to re-establish iscsi connections after following error. 

Inspected logs;

Apr 29 03:22:41 <hostname> iscsid: iscsid: Kernel reported iSCSI connection 2:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Apr 29 03:22:41 <hostname> iscsid: iscsid: re-opening session 2 (reopen_cnt 0)
Apr 29 03:22:41 <hostname> iscsid: iscsid: disconnecting conn 0x55878c4c6ad8, fd 6
Apr 29 03:22:41 <hostname> kernel: connection2:0: detected conn error (1020)
Apr 29 03:22:43 <hostname> iscsid: iscsid: Poll was woken by an alarm
Apr 29 03:22:43 <hostname> iscsid: iscsid: re-opening session 2 (reopen_cnt 0)
Apr 29 03:22:43 <hostname> iscsid: iscsid: connecting to 169.254.2.3:3260
Apr 29 03:22:43 <hostname> iscsid: iscsid: connected local port 37550 to 169.254.2.3:3260
Apr 29 03:22:43 <hostname> iscsid: iscsid: login response status 0000
Apr 29 03:22:43 <hostname> iscsid: iscsid: deleting a scheduled/waiting thread!
Apr 29 03:22:43 <hostname> iscsid: iscsid: connection2:0 is operational after recovery (1 attempts)

There are no iscsid retrying connection entries.

May 18 21:50:58 <hostname> kernel: connection2:0: detected conn error (1020)
May 18 21:53:18 <hostname> kernel: connection1:0: detected conn error (1020)

Instead disk io errors are reported. Both disks went into read-only mode.

May 18 21:56:21 <hostname> kernel: sd 3:0:0:1: timing out command, waited 180s
May 18 21:56:21 <hostname> kernel: sd 3:0:0:1: [sdb] FAILED Result: hostbyte=DID_IMM_RETRY driverbyte=DRIVER_OK
May 18 21:56:21 <hostname> kernel: sd 3:0:0:1: [sdb] CDB: Write(10) 2a 00 03 98 63 20 00 00 10 00
May 18 21:56:21 <hostname> kernel: blk_update_request: I/O error, dev sdb, sector 60318496
May 18 21:56:21 <hostname> kernel: Buffer I/O error on dev sdb, logical block 7539812, lost async page write
May 18 21:56:21 <hostname> kernel: Buffer I/O error on dev sdb, logical block 7539813, lost async page write
May 18 21:56:21 <hostname> kernel: JBD: Detected IO errors while flushing file data on sdb

May 18 22:08:37 <hostname> kernel: EXT3-fs (sdb): error: remounting filesystem read-only
May 20 12:06:59 <hostname> kernel: EXT3-fs (sdc): error: remounting filesystem read-only

Changes

Many OS commands like, ps, netstat, df etc hangs.

No packets transfer between the initiator and the iSCSI target block storage is seen via tcpdump.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.