Kernel Panic / Server hang after applying 19.10 GI RU (Patch 32226239 : GI RELEASE UPDATE 19.10.0.0.0) (Doc ID 2769194.1)

Last updated on JULY 20, 2024

Applies to:

Oracle Database - Enterprise Edition - Version 19.9.0.0.0 and later
Information in this document applies to any platform.

Symptoms

1) The issue is seen after applying GI 19.10 RU (Patch 32226239 : GI RELEASE UPDATE 19.10.0.0.0)

2) The analysis of OS crash dump /vmcore shows following AFD related functions:

[257558.460933] kernel BUG at drivers/scsi/vmw_pvscsi.c:338!

[257558.461140] invalid opcode: 0000 [#1] SMP

[257558.461169] Modules linked in: rds_tcp rds oracleacfs(PO) oracleadvm(PO) oracleoks(PO) udp_diag tcp_diag inet_diag nfsv3 rpcsec_gss_krb5 nfsv4 nfs fscache oracleafd(PO) cmac arc4 ecb md4 nls_utf8 cifs dns_resolver ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6_tables iptable_filter falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_11308(E) vmw_vsock_vmci_transport vsock vmw_balloon vmwgfx ttm pcspkr drm_kms_helper drm vmxnet3 sg i2c_piix4 vmw_vmci shpchp i2c_core nfsd nfs_acl lockd binfmt_misc grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic

[257558.469864] pata_acpi sd_mod serio_raw ata_piix vmw_pvscsi libata dm_mirror dm_region_hash dm_log dm_mod fuse

crash7latest> bt

PID: 19971 TASK: ffff88185a115400 CPU: 1 COMMAND: "oracle_19971_cl"

#0 [ffff88005ad8b230] machine_kexec at ffffffff8106418b

#1 [ffff88005ad8b2a0] crash_kexec at ffffffff8111e2e2

#2 [ffff88005ad8b370] oops_end at ffffffff8101b948

#3 [ffff88005ad8b3a0] die at ffffffff8101bf6b

#4 [ffff88005ad8b3d0] do_trap at ffffffff81017fcc

#5 [ffff88005ad8b430] do_error_trap at ffffffff81018507

#6 [ffff88005ad8b510] do_invalid_op at ffffffff81018d30

#7 [ffff88005ad8b520] invalid_op at ffffffff8176037d

[exception RIP: pvscsi_queue+1415]

RIP: ffffffffc00c2647 RSP: ffff88005ad8b5d8 RFLAGS: 00010002

RAX: 00000000000000b1 RBX: ffff881920c22a00 RCX: 0000000000001000

RDX: 00000000000000b1 RSI: 0000000000001000 RDI: ffff88191e40f620

RBP: ffff88005ad8b648 R8: 0000000000000000 R9: ffff8823959c7000

R10: ffff882394977c00 R11: ffff882391b1a000 R12: ffff8815121ef9c0

R13: ffff882390f50000 R14: ffff882390f50000 R15: ffff881920c33fd0

ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018

#8 [ffff88005ad8b5d0] pvscsi_queue at ffffffffc00c24e0 [vmw_pvscsi]

#9 [ffff88005ad8b650] scsi_dispatch_cmd at ffffffff814e587d

#10 [ffff88005ad8b680] scsi_request_fn at ffffffff814e8a66

#11 [ffff88005ad8b6f0] __blk_run_queue at ffffffff8131b7fd

#12 [ffff88005ad8b710] blk_queue_bio at ffffffff8132172b

#13 [ffff88005ad8b770] afd_mkrequest_fn at ffffffffc0682134 [oracleafd]

#14 [ffff88005ad8b850] generic_make_request at ffffffff8131f844

#15 [ffff88005ad8b8a0] submit_bio at ffffffff8131f997

#16 [ffff88005ad8b910] safd_bio_submit at ffffffffc0686019 [oracleafd]

#17 [ffff88005ad8b960] afdq_batch_submit at ffffffffc067a326 [oracleafd]

#18 [ffff88005ad8ba80] afdc_io at ffffffffc066b246 [oracleafd]

#19 [ffff88005ad8bb50] afdc_dispatch_ioctl at ffffffffc066ba6b [oracleafd]

#20 [ffff88005ad8bc30] afdc_execute_ioctl at ffffffffc066d31c [oracleafd]

#21 [ffff88005ad8bdb0] afd_ioctl at ffffffffc0681b23 [oracleafd]

#22 [ffff88005ad8be00] blkdev_ioctl at ffffffff8132fcda

#23 [ffff88005ad8be70] block_ioctl at ffffffff8125c3a1

#24 [ffff88005ad8be80] do_vfs_ioctl at ffffffff81235313

#25 [ffff88005ad8bf00] sys_ioctl at ffffffff812355a1

#26 [ffff88005ad8bf50] system_call_fastpath at ffffffff8175c2b6

RIP: 00007f26ac183397 RSP: 00007fffde6c69c8 RFLAGS: 00000246

RAX: ffffffffffffffda RBX: 0000000017b12f00 RCX: 00007f26ac183397

RDX: 0000000017b12f00 RSI: 0000000000507606 RDI: 0000000000000039

RBP: 00007fffde6c6a40 R8: 00007f26ab284498 R9: 0000000000000000

R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000

R13: 0000000000000001 R14: 0000000017b3fe30 R15: 00007fffde6c6c68

ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b

3) The OS messages log (/var/log/messages) may show following multipath related errors:

multipathd: oracleafd!admin: add path (uevent)

multipathd: oracleafd/admin: HDIO_GETGEO failed with 25

multipathd: oracleafd/admin: failed to get path uid

multipathd: uevent trigger error

kernel: Request for unknown module key 'Oracle Linux RHCK Module Signing Key: dd995b155c19b3a7c3ef7707b969e25f9639666e' err -11

kernel: OKSK-00028: In memory kernel log buffer address: 0xffff99c5c22f6340, size: 10485760

kernel: OKSK-00027: Oracle kernel distributed lock manager hash size is 2000000

kernel: Request for unknown module key 'OracleLinux RHCK Module Signing Key: dd995b155c19b3a7c3ef7707b969e25f9639666e' err-11

multipathd: asm!.asm_ctl_vdbg: add path (uevent)

multipathd: asm/.asm_ctl_vdbg: failed to get path uid

kernel: ADVMK-0001: Module load succeeded. Build information: (LOW DEBUG) - USM_19.10.0.0.0ACFSRU_LINUX.X64_201219 built on 2020/12/22 06:23:53.

kernel: asmInit: rval=0 mode=0 asm_request_based=1

multipathd: uevent trigger error ...

kernel: blk_cloned_rq_check_limits: over max segments limit.

Changes

Cause

	To view full details, sign in with your My Oracle Support account.
	Don't have a My Oracle Support account? Click to get started!

In this Document

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.

Kernel Panic / Server hang after applying 19.10 GI RU (Patch 32226239 : GI RELEASE UPDATE 19.10.0.0.0) (Doc ID 2769194.1)

Applies to:

Symptoms

Changes

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!