Linux System Crashes When Symantec 'rtvscand' Scans a 'MVFS' Mount Point. (Doc ID 2549476.1)

Last updated on OCTOBER 15, 2021

Applies to:

Symptoms

An Oracle Linux server repeatedly crashes. Crashes can even happen on a daily or weekly basis and might even appear to crash at the same time of the day.

During these crashes there are Symantec applications running and performing daily/weekly scans.

The OS Logs or vmcore file might report the following errors:

PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000230"
PID: 12247
COMMAND: "rtvscand"
TASK: ffff9846e288a080 [THREAD_INFO: ffff9843589a8000]
CPU: 9
STATE: TASK_RUNNING (PANIC)

crash7latest> bt
PID: 12247 TASK: ffff9846e288a080 CPU: 9 COMMAND: "rtvscand"
#0 [ffff9843589aba70] machine_kexec at ffffffffa9063674
#1 [ffff9843589abad0] __crash_kexec at ffffffffa911cef2
#2 [ffff9843589abba0] crash_kexec at ffffffffa911cfe0
#3 [ffff9843589abbb8] oops_end at ffffffffa976c758
#4 [ffff9843589abbe0] no_context at ffffffffa975aafe
#5 [ffff9843589abc30] __bad_area_nosemaphore at ffffffffa975ab95
#6 [ffff9843589abc80] bad_area_nosemaphore at ffffffffa975ad06
#7 [ffff9843589abc90] __do_page_fault at ffffffffa976f6b0
#8 [ffff9843589abd00] do_page_fault at ffffffffa976f915
#9 [ffff9843589abd30] page_fault at ffffffffa976b758
[exception RIP: fsnotify+128]
RIP: ffffffffa92889d0 RSP: ffff9843589abde0 RFLAGS: 00210246
RAX: 0000000000020000 RBX: ffff9845ad62e810 RCX: 0000000000000001
RDX: ffff9845ad62e810 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff9843589abeb0 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000020000
R13: 0000000000000000 R14: 0000000000000001 R15: ffffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff9843589abeb8] security_file_permission at ffffffffa92f914c
#11 [ffff9843589abee0] iterate_dir at ffffffffa9256783
#12 [ffff9843589abf18] compat_sys_getdents64 at ffffffffa92a217c
#13 [ffff9843589abf80] sysenter_dispatch at ffffffffa9778bb0
RIP: 00000000f77cc430 RSP: 00000000e16fd780 RFLAGS: 00200246
RAX: 00000000000000dc RBX: ffffffffa9778bb0 RCX: 00000000e1463cc4
RDX: 0000000000008000 RSI: 00000000e1463cc4 RDI: 00000000f7752000
RBP: 00000000e16fd8a8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 000000000000001d
ORIG_RAX: 00000000000000dc CS: 0023 SS: 002b

[229942.098424] ------------[ cut here ]------------
[229942.099429] WARNING: CPU: 13 PID: 7089 at fs/dcache.c:1566 d_set_d_op+0xc2/0x100
[229942.100406] Modules linked in: macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag mvfs(OE) twnotify(OE) symap_custom_3_10_0_957_1_3_el7_x86_64(POE) symev_custom_3_10_0_957_1_3_el7_x86_64(OE) team_mode_activebackup team iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev i2c_i801 lpc_ich hpilo hpwdt sg ipmi_si ipmi_devintf ipmi_msghandler ioatdma pcc_cpufreq wmi dca acpi_power_meter binfmt_misc nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm crct10dif_pclmul
[229942.106065] tg3 crct10dif_common libata serio_raw crc32c_intel ptp pps_core drm_panel_orientation_quirks hpsa scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod
[229942.107886] CPU: 13 PID: 7089 Comm: find Kdump: loaded Tainted: P W OE ------------ 3.10.0-957.1.3.el7.x86_64 #1
[229942.108947] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 10/17/2018
[229942.109993] Call Trace:
[229942.111093] [<ffffffffa9761e41>] dump_stack+0x19/0x1b
[229942.112159] [<ffffffffa9097648>] __warn+0xd8/0x100
[229942.113221] [<ffffffffa909778d>] warn_slowpath_null+0x1d/0x20
[229942.114225] [<ffffffffa9259282>] d_set_d_op+0xc2/0x100
[229942.115180] [<ffffffffc0757258>] vnode_iop_lookup+0x6f1/0x747 [mvfs]
[229942.116125] [<ffffffffa925c1c8>] ? d_alloc+0x58/0x70
[229942.117033] [<ffffffffa924c1b3>] lookup_real+0x23/0x60
[229942.117815] [<ffffffffa924cbd2>] __lookup_hash+0x42/0x60
[229942.118697] [<ffffffffa975eae1>] lookup_slow+0x42/0xa7
[229942.119529] [<ffffffffa9251ea8>] path_lookupat+0x838/0x8b0
[229942.120327] [<ffffffffa91023b4>] ? getrawmonotonic64+0x34/0xc0
[229942.121129] [<ffffffffa924c137>] ? path_get+0x27/0x30
[229942.121957] [<ffffffffa9251f4b>] filename_lookup+0x2b/0xc0
[229942.122694] [<ffffffffa9253f37>] user_path_at_empty+0x67/0xc0
[229942.123559] [<ffffffffc0752344>] ? mvfs_release_thread_ptr+0x13/0x15 [mvfs]
[229942.124402] [<ffffffffc075f46f>] ? mdki_linux_destroy_call_data+0x25/0x30 [mvfs]
[229942.125226] [<ffffffffc0754a00>] ? vnode_fop_readdir+0xb2/0xe7 [mvfs]
[229942.126035] [<ffffffffa9253fa1>] user_path_at+0x11/0x20
[229942.126810] [<ffffffffa9246e33>] vfs_fstatat+0x63/0xc0
[229942.127643] [<ffffffffa92472a4>] SYSC_newfstatat+0x24/0x60
[229942.128476] [<ffffffffa92476ce>] SyS_newfstatat+0xe/0x10
[229942.129237] [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
[229942.130075] ---[ end trace db1c4b5af01330ce ]---
[230776.411260] BUG: unable to handle kernel NULL pointer dereference at 0000000000000230
[230776.411963] IP: [<ffffffffa92889d0>] fsnotify+0x80/0x510
[230776.412702] PGD 80000007776ee067 PUD 0
[230776.413412] Oops: 0000 [#1] SMP
[230776.414101] Modules linked in: macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag mvfs(OE) twnotify(OE) symap_custom_3_10_0_957_1_3_el7_x86_64(POE) symev_custom_3_10_0_957_1_3_el7_x86_64(OE) team_mode_activebackup team iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev i2c_i801 lpc_ich hpilo hpwdt sg ipmi_si ipmi_devintf ipmi_msghandler ioatdma pcc_cpufreq wmi dca acpi_power_meter binfmt_misc nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm crct10dif_pclmul
[230776.418894] tg3 crct10dif_common libata serio_raw crc32c_intel ptp pps_core drm_panel_orientation_quirks hpsa scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod
[230776.420648] CPU: 9 PID: 12247 Comm: rtvscand Kdump: loaded Tainted: P W OE ------------ 3.10.0-957.1.3.el7.x86_64 #1
[230776.422367] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 10/17/2018
[230776.423295] task: ffff9846e288a080 ti: ffff9843589a8000 task.ti: ffff9843589a8000
[230776.424124] RIP: 0010:[<ffffffffa92889d0>] [<ffffffffa92889d0>] fsnotify+0x80/0x510
[230776.424959] RSP: 0018:ffff9843589abde0 EFLAGS: 00210246
[230776.425759] RAX: 0000000000020000 RBX: ffff9845ad62e810 RCX: 0000000000000001
[230776.426672] RDX: ffff9845ad62e810 RSI: 0000000000000000 RDI: 0000000000000000
[230776.427464] RBP: ffff9843589abeb0 R08: 0000000000000000 R09: 0000000000000000
[230776.428237] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000020000
[230776.428965] R13: 0000000000000000 R14: 0000000000000001 R15: ffffffffffffffff
[230776.429704] FS: 0000000000000000(0000) GS:ffff98499f240000(0063) knlGS:00000000e16ffb40
[230776.430435] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033
[230776.431148] CR2: 0000000000000230 CR3: 000000080e6be000 CR4: 00000000003607e0
[230776.431866] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[230776.432650] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[230776.433433] Call Trace:
[230776.434170] [<ffffffffa92f914c>] security_file_permission+0x8c/0xa0
[230776.434911] [<ffffffffa9256783>] iterate_dir+0x43/0x130
[230776.435701] [<ffffffffa91386ba>] ? audit_filter_inodes+0xda/0x130
[230776.436520] [<ffffffffa92a217c>] compat_sys_getdents64+0x9c/0x120
[230776.437339] [<ffffffffa92a0c80>] ? cp_compat_stat+0x210/0x210
[230776.438078] [<ffffffffa9778bb0>] sysenter_dispatch+0xd/0x2b
[230776.438825] Code: 00 00 00 00 48 c7 45 b0 00 00 00 00 89 45 a4 74 1a 48 8b 45 90 48 83 b8 30 02 00 00 00 0f 84 f8 03 00 00 31 db eb 19 0f 1f 40 00 <48> 83 bf 30 02 00 00 00 48 8b 02 48 8d 58 e0 0f 84 bf 03 00 00
[230776.440459] RIP [<ffffffffa92889d0>] fsnotify+0x80/0x510
[230776.441282] RSP <ffff9843589abde0>
[230776.442044] CR2: 0000000000000230

Changes

 N/A

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.