AFD causes stack trace during server reboot
(Doc ID 2633620.1)
Last updated on APRIL 17, 2023
Applies to:
Oracle Database - Enterprise Edition - Version 19.5.0.0.0 and laterInformation in this document applies to any platform.
Symptoms
On : 19.5.0.0.0 version, STORAGE
ACTUAL BEHAVIOR
---------------
When a system configured with Oracle Restart and AFD device is booting up, the kernel module for AFD is causing a stacktrace.
This is possibly causing shutdown issues.
Stack trace from OS log (/var/log/messages)
Dec 6 15:26:11 <NODE_NAME> kernel: Bad or missing usercopy whitelist? Kernel memory exposure attempt detected from SLUB object 'AFDkdevice' (offset 264, size 128)!
Dec 6 15:26:11 <NODE_NAME> kernel: ------------[ cut here ]------------
Dec 6 15:26:11 <NODE_NAME> kernel: WARNING: CPU: 41 PID: 35285 at mm/usercopy.c:81 usercopy_warn+0x8e/0xa6
Dec 6 15:26:11 <NODE_NAME> kernel: Modules linked in: oracleacfs(PO) oracleadvm(PO) oracleoks(PO) oracleafd(PO) fuse btrfs xor zstd_decompress zstd_compress xxhash raid6_pq vfat msdos fat ext2 rpcsec_gss_krb5 nfsv4 dns_resolver nfsv3 nfs_acl nfs lockd grace fscache binfmt_misc mptctl mptbase xt_CT iptable_raw xt_nat iptable_nat nf_nat_ipv4 nf_nat iptable_mangle nf_log_ipv4 nf_log_common xt_LOG xt_limit ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_owner xt_multiport xt_conntrack nf_conntrack iptable_filter dummy ext4 mbcache jbd2 fscrypto iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp ipmi_ssif kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc sg hpilo aesni_intel hpwdt crypto_simd glue_helper cryptd shpchp ipmi_si pcspkr ipmi_devintf ioatdma acpi_power_meter
Dec 6 15:26:11 <NODE_NAME> kernel: ipmi_msghandler lpc_ich pcc_cpufreq tcp_htcp auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod ata_generic pata_acpi i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm crc32c_intel ata_piix libata serio_raw hpsa nvme mdio ptp nvme_core scsi_transport_sas pps_core dca dm_mirror dm_region_hash dm_log dm_mod
Dec 6 15:26:11 <NODE_NAME> kernel: CPU: 41 PID: 35285 Comm: afdtool.bin Tainted: P O 4.14.35-1902.7.3.1.el7uek.x86_64 #2
Dec 6 15:26:11 <NODE_NAME> kernel: Hardware name: HP ProLiant DL360p Gen8, BIOS P71 05/21/2018
Dec 6 15:26:11 <NODE_NAME> kernel: task: ffff9e68bd342f80 task.stack: ffffc190a9570000
Dec 6 15:26:11 <NODE_NAME> kernel: RIP: 0010:usercopy_warn+0x8e/0xa6
Dec 6 15:26:11 <NODE_NAME> kernel: RSP: 0018:ffffc190a9573ab8 EFLAGS: 00010246
Dec 6 15:26:11 <NODE_NAME> kernel: RAX: 0000000000000080 RBX: ffffffffa51e8df5 RCX: 0000000000000000
Dec 6 15:26:11 <NODE_NAME> kernel: RDX: 0000000000000000 RSI: ffff9ea2bf2569c8 RDI: ffff9ea2bf2569c8
Dec 6 15:26:11 <NODE_NAME> kernel: RBP: ffffc190a9573ad8 R08: 0000000000000000 R09: 0000000000004476
Dec 6 15:26:11 <NODE_NAME> kernel: R10: 0000000000000004 R11: 0000000000004475 R12: 0000000000000080
Dec 6 15:26:11 <NODE_NAME> kernel: R13: ffff9e67a986c188 R14: 0000000000000001 R15: 0000000000a89070
Dec 6 15:26:11 <NODE_NAME> kernel: FS: 000014cfb9703cc0(0000) GS:ffff9ea2bf240000(0000) knlGS:0000000000000000
Dec 6 15:26:11 <NODE_NAME> kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 6 15:26:11 <NODE_NAME> kernel: CR2: 000014cfb9711000 CR3: 0000003f56e2c006 CR4: 00000000001606e0
Dec 6 15:26:11 <NODE_NAME> kernel: Call Trace:
Dec 6 15:26:11 <NODE_NAME> kernel: __check_heap_object+0xb8/0x11b
Dec 6 15:26:11 <NODE_NAME> kernel: __check_object_size+0xe3/0x1bc
Dec 6 15:26:11 <NODE_NAME> kernel: AfdgCopyout+0x2c/0x60 [oracleafd]
Dec 6 15:26:11 <NODE_NAME> kernel: afdk_fetch_all+0x110/0x1f0 [oracleafd]
Dec 6 15:26:11 <NODE_NAME> kernel: afdc_dispatch_ioctl+0x107f/0x1230 [oracleafd]
Dec 6 15:26:11 <NODE_NAME> kernel: ? __check_object_size+0x53/0x1bc
Dec 6 15:26:11 <NODE_NAME> kernel: ? _copy_from_user+0x33/0x62
Dec 6 15:26:11 <NODE_NAME> kernel: afdc_execute_ioctl+0x353/0x460 [oracleafd]
Dec 6 15:26:11 <NODE_NAME> kernel: ? mntput+0x24/0x33
Dec 6 15:26:11 <NODE_NAME> kernel: ? path_lookupat+0xed/0x203
Dec 6 15:26:11 <NODE_NAME> kernel: ? putname+0x53/0x57
Dec 6 15:26:11 <NODE_NAME> kernel: ? filename_lookup+0xef/0x183
Dec 6 15:26:11 <NODE_NAME> kernel: ? __check_object_size+0x53/0x1bc
Dec 6 15:26:11 <NODE_NAME> kernel: afd_ioctl+0x179/0x320 [oracleafd]
Dec 6 15:26:11 <NODE_NAME> kernel: blkdev_ioctl+0x878/0x912
Dec 6 15:26:11 <NODE_NAME> kernel: ? vfs_statx+0x77/0xd7
Dec 6 15:26:11 <NODE_NAME> kernel: block_ioctl+0x41/0x45
Dec 6 15:26:11 <NODE_NAME> kernel: do_vfs_ioctl+0xaa/0x602
Dec 6 15:26:11 <NODE_NAME> kernel: ? SYSC_newstat+0x3d/0x70
Dec 6 15:26:11 <NODE_NAME> kernel: SyS_ioctl+0x79/0x84
Dec 6 15:26:11 <NODE_NAME> kernel: do_syscall_64+0x79/0x1b9
Dec 6 15:26:11 <NODE_NAME> kernel: entry_SYSCALL_64_after_hwframe+0x169/0x0
Dec 6 15:26:11 <NODE_NAME> kernel: RIP: 0033:0x14cfb12802f7
Dec 6 15:26:11 <NODE_NAME> kernel: RSP: 002b:00007ffc35d28648 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Dec 6 15:26:11 <NODE_NAME> kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 000014cfb12802f7
Dec 6 15:26:11 <NODE_NAME> kernel: RDX: 0000000000a89490 RSI: 000000000020760e RDI: 0000000000000008
Dec 6 15:26:11 <NODE_NAME> kernel: RBP: 00007ffc35d28900 R08: 0000000000000800 R09: 0000000000000080
Dec 6 15:26:11 <NODE_NAME> kernel: R10: 0000000000000009 R11: 0000000000000246 R12: 0000000000000000
Dec 6 15:26:11 <NODE_NAME> kernel: R13: 0000000000a89490 R14: 0000000000a895c0 R15: 000014cfb968f220
Dec 6 15:26:11 <NODE_NAME> kernel: Code: 1d a5 48 0f 45 f2 4c 89 44 24 10 48 89 c2 48 89 4c 24 08 48 89 1c 24 4d 89 d8 4c 89 d1 48 c7 c7 58 8e 1e a5 31 c0 e8 75 e5 e7 ff <0f> 0b 48 83 c4 18 5b 5d c3 49 c7 c1 a1 7a 1f a5 4c 89 cb 4d 89
Dec 6 15:26:11 <NODE_NAME> kernel: ---[ end trace 20b55e3f6ea56c4d ]---
Dec 6 15:26:16 <NODE_NAME> bashhist: TS=1575642376.403255826|USER=oracle/4001|PID=48252|PWD=<GRID_HOME>|SSH_CLIENT=|SUDO=<USER_NAME>/5009|CMD=asmcmd afd_label <DG_NAME> <DEVICE_NAME>
Dec 6 15:26:17 <NODE_NAME> kernel: LogAndDROP INPUT IN=eth0 OUT= MAC=8c:dc:d4:b2:a6:20:2c:f8:9b:14:6e:07:08:00 SRC=10.1.130.72 DST=10.1.66.42 LEN=482 TOS=0x00 PREC=0x00 TTL=62 ID=14656 DF PROTO=UDP SPT=3772 DPT=1028 LEN=462
Dec 6 15:26:30 <NODE_NAME> bashhist: TS=1575642390.082757584|USER=oracle/4001|PID=48252|PWD=<GRID_HOME>|SSH_CLIENT=|SUDO=<USER_NAME>/5009|CMD=asmcmd afd_lslbl
Dec 6 15:26:52 <NODE_NAME> bashhist: TS=1575642412.883095020|USER=oracle/4001|PID=48252|PWD=<GRID_HOME>|SSH_CLIENT=|SUDO=<USER_NAME>/5009|CMD=asmcmd afd_lsdsk
Dec 6 15:27:04 <NODE_NAME> bashhist: TS=1575642424.027972577|USER=oracle/4001|PID=48252|PWD=<GRID_HOME>|SSH_CLIENT=|SUDO=<USER_NAME>/5009|CMD=crsctl start res ora.cssd -init
Dec 6 15:26:11 <NODE_NAME> kernel: ------------[ cut here ]------------
Dec 6 15:26:11 <NODE_NAME> kernel: WARNING: CPU: 41 PID: 35285 at mm/usercopy.c:81 usercopy_warn+0x8e/0xa6
Dec 6 15:26:11 <NODE_NAME> kernel: Modules linked in: oracleacfs(PO) oracleadvm(PO) oracleoks(PO) oracleafd(PO) fuse btrfs xor zstd_decompress zstd_compress xxhash raid6_pq vfat msdos fat ext2 rpcsec_gss_krb5 nfsv4 dns_resolver nfsv3 nfs_acl nfs lockd grace fscache binfmt_misc mptctl mptbase xt_CT iptable_raw xt_nat iptable_nat nf_nat_ipv4 nf_nat iptable_mangle nf_log_ipv4 nf_log_common xt_LOG xt_limit ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_owner xt_multiport xt_conntrack nf_conntrack iptable_filter dummy ext4 mbcache jbd2 fscrypto iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp ipmi_ssif kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc sg hpilo aesni_intel hpwdt crypto_simd glue_helper cryptd shpchp ipmi_si pcspkr ipmi_devintf ioatdma acpi_power_meter
Dec 6 15:26:11 <NODE_NAME> kernel: ipmi_msghandler lpc_ich pcc_cpufreq tcp_htcp auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod ata_generic pata_acpi i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm crc32c_intel ata_piix libata serio_raw hpsa nvme mdio ptp nvme_core scsi_transport_sas pps_core dca dm_mirror dm_region_hash dm_log dm_mod
Dec 6 15:26:11 <NODE_NAME> kernel: CPU: 41 PID: 35285 Comm: afdtool.bin Tainted: P O 4.14.35-1902.7.3.1.el7uek.x86_64 #2
Dec 6 15:26:11 <NODE_NAME> kernel: Hardware name: HP ProLiant DL360p Gen8, BIOS P71 05/21/2018
Dec 6 15:26:11 <NODE_NAME> kernel: task: ffff9e68bd342f80 task.stack: ffffc190a9570000
Dec 6 15:26:11 <NODE_NAME> kernel: RIP: 0010:usercopy_warn+0x8e/0xa6
Dec 6 15:26:11 <NODE_NAME> kernel: RSP: 0018:ffffc190a9573ab8 EFLAGS: 00010246
Dec 6 15:26:11 <NODE_NAME> kernel: RAX: 0000000000000080 RBX: ffffffffa51e8df5 RCX: 0000000000000000
Dec 6 15:26:11 <NODE_NAME> kernel: RDX: 0000000000000000 RSI: ffff9ea2bf2569c8 RDI: ffff9ea2bf2569c8
Dec 6 15:26:11 <NODE_NAME> kernel: RBP: ffffc190a9573ad8 R08: 0000000000000000 R09: 0000000000004476
Dec 6 15:26:11 <NODE_NAME> kernel: R10: 0000000000000004 R11: 0000000000004475 R12: 0000000000000080
Dec 6 15:26:11 <NODE_NAME> kernel: R13: ffff9e67a986c188 R14: 0000000000000001 R15: 0000000000a89070
Dec 6 15:26:11 <NODE_NAME> kernel: FS: 000014cfb9703cc0(0000) GS:ffff9ea2bf240000(0000) knlGS:0000000000000000
Dec 6 15:26:11 <NODE_NAME> kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 6 15:26:11 <NODE_NAME> kernel: CR2: 000014cfb9711000 CR3: 0000003f56e2c006 CR4: 00000000001606e0
Dec 6 15:26:11 <NODE_NAME> kernel: Call Trace:
Dec 6 15:26:11 <NODE_NAME> kernel: __check_heap_object+0xb8/0x11b
Dec 6 15:26:11 <NODE_NAME> kernel: __check_object_size+0xe3/0x1bc
Dec 6 15:26:11 <NODE_NAME> kernel: AfdgCopyout+0x2c/0x60 [oracleafd]
Dec 6 15:26:11 <NODE_NAME> kernel: afdk_fetch_all+0x110/0x1f0 [oracleafd]
Dec 6 15:26:11 <NODE_NAME> kernel: afdc_dispatch_ioctl+0x107f/0x1230 [oracleafd]
Dec 6 15:26:11 <NODE_NAME> kernel: ? __check_object_size+0x53/0x1bc
Dec 6 15:26:11 <NODE_NAME> kernel: ? _copy_from_user+0x33/0x62
Dec 6 15:26:11 <NODE_NAME> kernel: afdc_execute_ioctl+0x353/0x460 [oracleafd]
Dec 6 15:26:11 <NODE_NAME> kernel: ? mntput+0x24/0x33
Dec 6 15:26:11 <NODE_NAME> kernel: ? path_lookupat+0xed/0x203
Dec 6 15:26:11 <NODE_NAME> kernel: ? putname+0x53/0x57
Dec 6 15:26:11 <NODE_NAME> kernel: ? filename_lookup+0xef/0x183
Dec 6 15:26:11 <NODE_NAME> kernel: ? __check_object_size+0x53/0x1bc
Dec 6 15:26:11 <NODE_NAME> kernel: afd_ioctl+0x179/0x320 [oracleafd]
Dec 6 15:26:11 <NODE_NAME> kernel: blkdev_ioctl+0x878/0x912
Dec 6 15:26:11 <NODE_NAME> kernel: ? vfs_statx+0x77/0xd7
Dec 6 15:26:11 <NODE_NAME> kernel: block_ioctl+0x41/0x45
Dec 6 15:26:11 <NODE_NAME> kernel: do_vfs_ioctl+0xaa/0x602
Dec 6 15:26:11 <NODE_NAME> kernel: ? SYSC_newstat+0x3d/0x70
Dec 6 15:26:11 <NODE_NAME> kernel: SyS_ioctl+0x79/0x84
Dec 6 15:26:11 <NODE_NAME> kernel: do_syscall_64+0x79/0x1b9
Dec 6 15:26:11 <NODE_NAME> kernel: entry_SYSCALL_64_after_hwframe+0x169/0x0
Dec 6 15:26:11 <NODE_NAME> kernel: RIP: 0033:0x14cfb12802f7
Dec 6 15:26:11 <NODE_NAME> kernel: RSP: 002b:00007ffc35d28648 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Dec 6 15:26:11 <NODE_NAME> kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 000014cfb12802f7
Dec 6 15:26:11 <NODE_NAME> kernel: RDX: 0000000000a89490 RSI: 000000000020760e RDI: 0000000000000008
Dec 6 15:26:11 <NODE_NAME> kernel: RBP: 00007ffc35d28900 R08: 0000000000000800 R09: 0000000000000080
Dec 6 15:26:11 <NODE_NAME> kernel: R10: 0000000000000009 R11: 0000000000000246 R12: 0000000000000000
Dec 6 15:26:11 <NODE_NAME> kernel: R13: 0000000000a89490 R14: 0000000000a895c0 R15: 000014cfb968f220
Dec 6 15:26:11 <NODE_NAME> kernel: Code: 1d a5 48 0f 45 f2 4c 89 44 24 10 48 89 c2 48 89 4c 24 08 48 89 1c 24 4d 89 d8 4c 89 d1 48 c7 c7 58 8e 1e a5 31 c0 e8 75 e5 e7 ff <0f> 0b 48 83 c4 18 5b 5d c3 49 c7 c1 a1 7a 1f a5 4c 89 cb 4d 89
Dec 6 15:26:11 <NODE_NAME> kernel: ---[ end trace 20b55e3f6ea56c4d ]---
Dec 6 15:26:16 <NODE_NAME> bashhist: TS=1575642376.403255826|USER=oracle/4001|PID=48252|PWD=<GRID_HOME>|SSH_CLIENT=|SUDO=<USER_NAME>/5009|CMD=asmcmd afd_label <DG_NAME> <DEVICE_NAME>
Dec 6 15:26:17 <NODE_NAME> kernel: LogAndDROP INPUT IN=eth0 OUT= MAC=8c:dc:d4:b2:a6:20:2c:f8:9b:14:6e:07:08:00 SRC=10.1.130.72 DST=10.1.66.42 LEN=482 TOS=0x00 PREC=0x00 TTL=62 ID=14656 DF PROTO=UDP SPT=3772 DPT=1028 LEN=462
Dec 6 15:26:30 <NODE_NAME> bashhist: TS=1575642390.082757584|USER=oracle/4001|PID=48252|PWD=<GRID_HOME>|SSH_CLIENT=|SUDO=<USER_NAME>/5009|CMD=asmcmd afd_lslbl
Dec 6 15:26:52 <NODE_NAME> bashhist: TS=1575642412.883095020|USER=oracle/4001|PID=48252|PWD=<GRID_HOME>|SSH_CLIENT=|SUDO=<USER_NAME>/5009|CMD=asmcmd afd_lsdsk
Dec 6 15:27:04 <NODE_NAME> bashhist: TS=1575642424.027972577|USER=oracle/4001|PID=48252|PWD=<GRID_HOME>|SSH_CLIENT=|SUDO=<USER_NAME>/5009|CMD=crsctl start res ora.cssd -init
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |