My Oracle Support Banner

Kernel Panic / Server hang after applying 19.10 GI RU (Patch 32226239 : GI RELEASE UPDATE 19.10.0.0.0) (Doc ID 2769194.1)

Last updated on APRIL 15, 2021

Applies to:

Oracle Database - Enterprise Edition - Version 19.9.0.0.0 and later
Information in this document applies to any platform.

Symptoms

1) The issue is seen after applying GI 19.10 RU (Patch 32226239 : GI RELEASE UPDATE 19.10.0.0.0)

2) The analysis of OS crash dump /vmcore shows following AFD related functions:

[257558.460933] kernel BUG at drivers/scsi/vmw_pvscsi.c:338!
[257558.461140] invalid opcode: 0000 [#1] SMP
[257558.461169] Modules linked in: rds_tcp rds oracleacfs(PO) oracleadvm(PO) oracleoks(PO) udp_diag tcp_diag inet_diag nfsv3 rpcsec_gss_krb5 nfsv4 nfs fscache oracleafd(PO) cmac arc4 ecb md4 nls_utf8 cifs dns_resolver ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6_tables iptable_filter falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_11308(E) vmw_vsock_vmci_transport vsock vmw_balloon vmwgfx ttm pcspkr drm_kms_helper drm vmxnet3 sg i2c_piix4 vmw_vmci shpchp i2c_core nfsd nfs_acl lockd binfmt_misc grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic
[257558.469864]  pata_acpi sd_mod serio_raw ata_piix vmw_pvscsi libata dm_mirror dm_region_hash dm_log dm_mod fuse
crash7latest> bt
PID: 19971  TASK: ffff88185a115400  CPU: 1   COMMAND: "oracle_19971_cl"
 #0 [ffff88005ad8b230] machine_kexec at ffffffff8106418b
 #1 [ffff88005ad8b2a0] crash_kexec at ffffffff8111e2e2
 #2 [ffff88005ad8b370] oops_end at ffffffff8101b948
 #3 [ffff88005ad8b3a0] die at ffffffff8101bf6b
 #4 [ffff88005ad8b3d0] do_trap at ffffffff81017fcc
 #5 [ffff88005ad8b430] do_error_trap at ffffffff81018507
 #6 [ffff88005ad8b510] do_invalid_op at ffffffff81018d30
 #7 [ffff88005ad8b520] invalid_op at ffffffff8176037d
    [exception RIP: pvscsi_queue+1415]
    RIP: ffffffffc00c2647  RSP: ffff88005ad8b5d8  RFLAGS: 00010002
    RAX: 00000000000000b1  RBX: ffff881920c22a00  RCX: 0000000000001000
    RDX: 00000000000000b1  RSI: 0000000000001000  RDI: ffff88191e40f620
    RBP: ffff88005ad8b648   R8: 0000000000000000   R9: ffff8823959c7000
    R10: ffff882394977c00  R11: ffff882391b1a000  R12: ffff8815121ef9c0
    R13: ffff882390f50000  R14: ffff882390f50000  R15: ffff881920c33fd0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffff88005ad8b5d0] pvscsi_queue at ffffffffc00c24e0 [vmw_pvscsi]
 #9 [ffff88005ad8b650] scsi_dispatch_cmd at ffffffff814e587d
#10 [ffff88005ad8b680] scsi_request_fn at ffffffff814e8a66
#11 [ffff88005ad8b6f0] __blk_run_queue at ffffffff8131b7fd
#12 [ffff88005ad8b710] blk_queue_bio at ffffffff8132172b
#13 [ffff88005ad8b770] afd_mkrequest_fn at ffffffffc0682134 [oracleafd]
#14 [ffff88005ad8b850] generic_make_request at ffffffff8131f844
#15 [ffff88005ad8b8a0] submit_bio at ffffffff8131f997
#16 [ffff88005ad8b910] safd_bio_submit at ffffffffc0686019 [oracleafd]
#17 [ffff88005ad8b960] afdq_batch_submit at ffffffffc067a326 [oracleafd]
#18 [ffff88005ad8ba80] afdc_io at ffffffffc066b246 [oracleafd]
#19 [ffff88005ad8bb50] afdc_dispatch_ioctl at ffffffffc066ba6b [oracleafd]
#20 [ffff88005ad8bc30] afdc_execute_ioctl at ffffffffc066d31c [oracleafd]
#21 [ffff88005ad8bdb0] afd_ioctl at ffffffffc0681b23 [oracleafd]
#22 [ffff88005ad8be00] blkdev_ioctl at ffffffff8132fcda
#23 [ffff88005ad8be70] block_ioctl at ffffffff8125c3a1
#24 [ffff88005ad8be80] do_vfs_ioctl at ffffffff81235313
#25 [ffff88005ad8bf00] sys_ioctl at ffffffff812355a1
#26 [ffff88005ad8bf50] system_call_fastpath at ffffffff8175c2b6
    RIP: 00007f26ac183397  RSP: 00007fffde6c69c8  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000017b12f00  RCX: 00007f26ac183397
    RDX: 0000000017b12f00  RSI: 0000000000507606  RDI: 0000000000000039
    RBP: 00007fffde6c6a40   R8: 00007f26ab284498   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000000000000
    R13: 0000000000000001  R14: 0000000017b3fe30  R15: 00007fffde6c6c68
    ORIG_RAX: 0000000000000010  CS: 0033  SS: 002b 

3) The OS messages log (/var/log/messages) may show following multipath related errors:

multipathd: oracleafd!admin: add path (uevent)
multipathd: oracleafd/admin: HDIO_GETGEO failed with 25
multipathd: oracleafd/admin: failed to get path uid
multipathd: uevent trigger error
kernel: Request for unknown module key 'Oracle Linux RHCK Module Signing Key: dd995b155c19b3a7c3ef7707b969e25f9639666e' err -11
kernel: OKSK-00028: In memory kernel log buffer address: 0xffff99c5c22f6340, size: 10485760
kernel: OKSK-00027: Oracle kernel distributed lock manager hash size is 2000000
kernel: Request for unknown module key 'OracleLinux RHCK Module Signing Key: dd995b155c19b3a7c3ef7707b969e25f9639666e' err-11
multipathd: asm!.asm_ctl_vdbg: add path (uevent)
multipathd: asm/.asm_ctl_vdbg: failed to get path uid
kernel: ADVMK-0001: Module load succeeded. Build information:  (LOW DEBUG) - USM_19.10.0.0.0ACFSRU_LINUX.X64_201219 built on 2020/12/22 06:23:53.
kernel: asmInit: rval=0 mode=0 asm_request_based=1
multipathd: uevent trigger error ...
kernel: blk_cloned_rq_check_limits: over max segments limit. 

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.