19C:Node crashing while cssd trying to access Voting disk with AFD
(Doc ID 2580966.1)
Last updated on APRIL 17, 2023
Applies to:
Oracle Database - Enterprise Edition - Version 19.3.0.0.0 and laterInformation in this document applies to any platform.
Symptoms
19C Grid cluster installation of two node on production system fails at root.sh script. No failures are displayed to the screen.
First node gets evicted and the installation wizard disappears. Nodes comes back up normally.
# AFD is in use
This issue can happen for upgrade to 19C with AFD is in use
[ 364.304848] ------------[ cut here ]------------
[ 364.304893] kernel BUG at include/linux/scatterlist.h:73!
[ 364.304932] invalid opcode: 0000 [#1] SMP
[ 364.304965] Modules linked in: oracleacfs(POE) oracleadvm(POE) oracleoks(POE) tcp_lp fuse nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache dm_service_time iscsi_tcp libiscsi_tcp oracleafd(POE) team_mode_activebackup team rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm sunrpc ib_uverbs ib_umad rdma_cm ib_cm iw_cm bnxt_re ib_core dell_smbios dell_wmi_descriptor dcdbas skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ipmi_ssif ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd wdat_wdt pcspkr joydev sg i2c_i801 mei_me mei lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad dm_multipath
[ 364.305626] binfmt_misc ip_tables xfs sr_mod sd_mod cdrom crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper bnx2x syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32c_intel tg3 megaraid_sas ahci bnxt_en drm libahci libata devlink mdio ptp pps_core libcrc32c drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
[ 364.305940] CPU: 11 PID: 3724 Comm: kdmwork-253:16 Kdump: loaded Tainted: P OE ------------ 3.10.0-1062.el7.x86_64 #1
[ 364.306014] Hardware name: Dell Inc. PowerEdge R740/0923K0, BIOS 2.2.11 06/13/2019
[ 364.306063] task: ffff8f94f7f9b150 ti: ffff8f94e9fc8000 task.ti: ffff8f94e9fc8000
[ 364.306112] RIP: 0010:[] [] sg_set_page.part.23+0x4/0x6
[ 364.306177] RSP: 0018:ffff8f94e9fcbb28 EFLAGS: 00010002
[ 364.306213] RAX: ffff8f94e6606060 RBX: ffff8f376c745010 RCX: 000012fd00000001
[ 364.306259] RDX: 0000000000000000 RSI: 000000007d203032 RDI: ffff8f94e6606040
[ 364.306304] RBP: ffff8f94e9fcbb28 R08: ffff8f94e9fcbba0 R09: 0000000000000000
[ 364.306353] R10: ffff8f94eeb65658 R11: 01fffffffffffffe R12: ffff8f94e9fcbba0
[ 364.306399] R13: ffff8f94e9fcbb98 R14: ffff8f94e9fcbb90 R15: 0000000000000000
[ 364.306446] FS: 0000000000000000(0000) GS:ffff8f94fd0c0000(0000) knlGS:0000000000000000
[ 364.306499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 364.308549] CR2: 0000000004446200 CR3: 00000051e8c10000 CR4: 00000000007607e0
[ 364.310588] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 364.312330] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 364.313815] PKRU: 00000000
[ 364.315286] Call Trace:
[ 364.316751] [] __blk_segment_map_sg+0x19c/0x1a0
[ 364.318220] [] ? sg_init_table+0x1a/0x40
[ 364.319677] [] blk_rq_map_sg+0x9b/0x220
[ 364.321132] [] scsi_init_sgtable+0x49/0x70
[ 364.321899] [] scsi_init_io+0x4e/0x200
[ 364.322530] [] sd_setup_read_write_cmnd+0x3d/0x950 [sd_mod]
[ 364.323159] [] ? __scsi_get_command+0x1a/0xf0
[ 364.323792] [] sd_init_command+0x2f/0xc0 [sd_mod]
[ 364.324420] [] scsi_setup_cmnd+0x111/0x1c0
[ 364.325041] [] scsi_prep_fn+0xdb/0x180
[ 364.325655] [] blk_peek_request+0x17a/0x2a0
[ 364.326268] [] scsi_request_fn+0x48/0x680
[ 364.326873] [] __blk_run_queue+0x39/0x50
[ 364.327466] [] __elv_add_request+0xd3/0x260
[ 364.328055] [] blk_insert_cloned_request+0xe0/0x1a0
[ 364.328647] [] map_request+0x15d/0x2a0 [dm_mod]
[ 364.329230] [] map_tio_request+0x16/0x40 [dm_mod]
[ 364.329806] [] kthread_worker_fn+0x85/0x180
[ 364.330377] [] ? kthread_stop+0xf0/0xf0
[ 364.330945] [] kthread+0xd1/0xe0
[ 364.331498] [] ? insert_kthread_work+0x40/0x40
[ 364.332043] [] ret_from_fork_nospec_begin+0x7/0x21
[ 364.332574] [] ? insert_kthread_work+0x40/0x40
[ 364.333090] Code: e1 f3 a4 74 05 e8 cd 13 92 ff 48 8d 65 e8 5b 41 5c 41 5d 5d c3 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 0f
[ 364.334195] RIP [] sg_set_page.part.23+0x4/0x6
[ 364.334703] RSP
[ 364.304893] kernel BUG at include/linux/scatterlist.h:73!
[ 364.304932] invalid opcode: 0000 [#1] SMP
[ 364.304965] Modules linked in: oracleacfs(POE) oracleadvm(POE) oracleoks(POE) tcp_lp fuse nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache dm_service_time iscsi_tcp libiscsi_tcp oracleafd(POE) team_mode_activebackup team rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm sunrpc ib_uverbs ib_umad rdma_cm ib_cm iw_cm bnxt_re ib_core dell_smbios dell_wmi_descriptor dcdbas skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ipmi_ssif ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd wdat_wdt pcspkr joydev sg i2c_i801 mei_me mei lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad dm_multipath
[ 364.305626] binfmt_misc ip_tables xfs sr_mod sd_mod cdrom crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper bnx2x syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32c_intel tg3 megaraid_sas ahci bnxt_en drm libahci libata devlink mdio ptp pps_core libcrc32c drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
[ 364.305940] CPU: 11 PID: 3724 Comm: kdmwork-253:16 Kdump: loaded Tainted: P OE ------------ 3.10.0-1062.el7.x86_64 #1
[ 364.306014] Hardware name: Dell Inc. PowerEdge R740/0923K0, BIOS 2.2.11 06/13/2019
[ 364.306063] task: ffff8f94f7f9b150 ti: ffff8f94e9fc8000 task.ti: ffff8f94e9fc8000
[ 364.306112] RIP: 0010:[] [] sg_set_page.part.23+0x4/0x6
[ 364.306177] RSP: 0018:ffff8f94e9fcbb28 EFLAGS: 00010002
[ 364.306213] RAX: ffff8f94e6606060 RBX: ffff8f376c745010 RCX: 000012fd00000001
[ 364.306259] RDX: 0000000000000000 RSI: 000000007d203032 RDI: ffff8f94e6606040
[ 364.306304] RBP: ffff8f94e9fcbb28 R08: ffff8f94e9fcbba0 R09: 0000000000000000
[ 364.306353] R10: ffff8f94eeb65658 R11: 01fffffffffffffe R12: ffff8f94e9fcbba0
[ 364.306399] R13: ffff8f94e9fcbb98 R14: ffff8f94e9fcbb90 R15: 0000000000000000
[ 364.306446] FS: 0000000000000000(0000) GS:ffff8f94fd0c0000(0000) knlGS:0000000000000000
[ 364.306499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 364.308549] CR2: 0000000004446200 CR3: 00000051e8c10000 CR4: 00000000007607e0
[ 364.310588] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 364.312330] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 364.313815] PKRU: 00000000
[ 364.315286] Call Trace:
[ 364.316751] [] __blk_segment_map_sg+0x19c/0x1a0
[ 364.318220] [] ? sg_init_table+0x1a/0x40
[ 364.319677] [] blk_rq_map_sg+0x9b/0x220
[ 364.321132] [] scsi_init_sgtable+0x49/0x70
[ 364.321899] [] scsi_init_io+0x4e/0x200
[ 364.322530] [] sd_setup_read_write_cmnd+0x3d/0x950 [sd_mod]
[ 364.323159] [] ? __scsi_get_command+0x1a/0xf0
[ 364.323792] [] sd_init_command+0x2f/0xc0 [sd_mod]
[ 364.324420] [] scsi_setup_cmnd+0x111/0x1c0
[ 364.325041] [] scsi_prep_fn+0xdb/0x180
[ 364.325655] [] blk_peek_request+0x17a/0x2a0
[ 364.326268] [] scsi_request_fn+0x48/0x680
[ 364.326873] [] __blk_run_queue+0x39/0x50
[ 364.327466] [] __elv_add_request+0xd3/0x260
[ 364.328055] [] blk_insert_cloned_request+0xe0/0x1a0
[ 364.328647] [] map_request+0x15d/0x2a0 [dm_mod]
[ 364.329230] [] map_tio_request+0x16/0x40 [dm_mod]
[ 364.329806] [] kthread_worker_fn+0x85/0x180
[ 364.330377] [] ? kthread_stop+0xf0/0xf0
[ 364.330945] [] kthread+0xd1/0xe0
[ 364.331498] [] ? insert_kthread_work+0x40/0x40
[ 364.332043] [] ret_from_fork_nospec_begin+0x7/0x21
[ 364.332574] [] ? insert_kthread_work+0x40/0x40
[ 364.333090] Code: e1 f3 a4 74 05 e8 cd 13 92 ff 48 8d 65 e8 5b 41 5c 41 5d 5d c3 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 0f
[ 364.334195] RIP [] sg_set_page.part.23+0x4/0x6
[ 364.334703] RSP
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |
| References |