OVM Guest Linux Crashes in blkif_copy_from_grant() due to ASM I/O Completion (Doc ID 2665056.1)

Last updated on MAY 13, 2020

Applies to:

Symptoms

OVM Guest Linux Crashes in blkif_copy_from_grant() due to ASM I/O Completion

vmcore analysis is shown as belows:

crash7latest> sys
KERNEL: /share/linuxrpm/vmlinux_repo/64/4.1.12-124.16.4.el6uek.x86_64/vmlinux
DUMPFILE: vmcore
CPUS: 16 [OFFLINE: 15]
DATE: xxx xxx xxx
UPTIME: 27 days, 17:44:35
LOAD AVERAGE: 3.41, 3.32, 2.36
TASKS: 1053
NODENAME: hostname xxx
RELEASE: 4.1.12-124.16.4.el6uek.x86_64
VERSION: #2 SMP Thu Jun 14 18:55:52 PDT 2018
MACHINE: x86_64 (1995 Mhz)
MEMORY: 64 GB
PANIC: "general protection fault: 0000 [#1] SMP "

The trap is actually coming from an interrupt stack

crash7latest> bt -E -c 1
CPU 1 IRQ STACK:

KERNEL-MODE EXCEPTION FRAME AT: ffff880fcd643ad8
[exception RIP: memcpy_erms+6] 
RIP: ffffffff81332c76 RSP: ffff880fcd643b88 RFLAGS: 00010086
RAX: 05202c44736fe000 RBX: ffff880fcd643c68 RCX: 0000000000001000
RDX: 0000000000001000 RSI: ffff8800e5a4a000 RDI: 05202c44736fe000
RBP: ffff880fcd643ba0 R8: 0000000000001000 R9: 0000000000001000
R10: 0000520a444736fe R11: 0000000080000000 R12: 0000000000001000
R13: 0000000000001000 R14: ffffffffa0071140 R15: 0000000000001000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018

KERNEL-MODE EXCEPTION FRAME AT: ffff880fcd643e58
[exception RIP: net_tx_action]
RIP: ffffffff81609830 RSP: ffff880fcd643f00 RFLAGS: 00000202
RAX: ffffffff81609830 RBX: ffff880f26e66c00 RCX: 0000000092ca1083
RDX: 000000018ed5c985 RSI: 0000000000016440 RDI: ffffffff81ad2110
RBP: ffff880fcd643f68 R8: 0000000000000000 R9: ffff880fc4a43478
R10: 0000000000000000 R11: 0000000080000000 R12: ffff880fcd643e78
R13: ffffffff816f2764 R14: ffff880fcd643f68 R15: 0000000000000002

The functions that leads to panic is

blkif_interrupt()
-> blkif_completion()
-> gnttab_foreach_grant_in_range()
-> blkif_copy_from_grant()
-> __memcpy

The I/O interrupt that goes to the xen_blkfront is from a ASM I/O completion.

crash7latest> bt 9822
PID: 9822 TASK: ffff8805e5a70000 CPU: 13 COMMAND: "oracle"
#0 [ffff880fc115ba88] __schedule at ffffffff816e7f0b
#1 [ffff880fc115bae0] schedule at ffffffff816e8537
#2 [ffff880fc115bb00] schedule_timeout at ffffffff816ebd7c
#3 [ffff880fc115bbb0] io_schedule_timeout at ffffffff816e7c84
#4 [ffff880fc115bbf0] __dta_asm_wait_completion_isra_28_122 at ffffffffa0317857 [oracleasm]
#5 [ffff880fc115bcc0] asm_do_io at ffffffffa03195c4 [oracleasm]
#6 [ffff880fc115bda0] asmfs_svc_io64 at ffffffffa0319691 [oracleasm]
#7 [ffff880fc115be10] asmfs_file_read at ffffffffa03197be [oracleasm]
#8 [ffff880fc115be40] __vfs_read at ffffffff812078db
#9 [ffff880fc115bec0] vfs_read at ffffffff81207fca
#10 [ffff880fc115bf00] sys_read at ffffffff81208e86
#11 [ffff880fc115bf50] system_call_fastpath at ffffffff816edc5c

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.