Oracle VM: Infiniband Interface Does Not Come Up After Reboot

(Doc ID 2351769.1)

Last updated on FEBRUARY 05, 2018

Applies to:

Oracle VM - Version 3.4.3 and later
Linux x86-64

Symptoms

IB interfaces were not detected after rebooting an Exadata Oracle VM server
lspci command was not showing Mellanox card

Log /var/log/vm_event_callout.log shows below during VM startup
2018-01-11 11:12:03 INFO 286638 uuid = 514ca136-afbe-4e64-9571-3dac3aaec43c, Command = config_booted start
2018-01-11 11:12:03 INFO 286638 Dom0 (vmnode.localhost.localdomain) is running kernel 4.1.12-61.33.1.el6uek.x86_64, OVS 3.4.2-b138.
2018-01-11 11:12:03 INFO 286638 VM config file = /OVS/Repositories//514ca136afbe4e6495713dac3aaec43c/vm.cfg
2018-01-11 11:12:03 INFO 286638 Mellanox HCA MT27500: State Up, 1 physical function (PF) and 63 virtual functions (VFs).
2018-01-11 11:12:03 INFO 286638 VF lock file /tmp/vm_event_callout_IB.lck locked.
2018-01-11 11:12:04 INFO 286638 Next available IB VF = 0000:03:00.1
2018-01-11 11:12:04 INFO 286638 PKey init: port 1, pkey = 0xffff
2018-01-11 11:12:04 INFO 286638 PKey cleanup: echo none > /sys/class/infiniband/mlx4_0/iov/0000:03:00.1/ports/1/pkey_idx/[0-127]
2018-01-11 11:12:05 INFO 286638 PKey config: echo 0 > /sys/class/infiniband/mlx4_0/iov/0000:03:00.1/ports/1/pkey_idx/0
2018-01-11 11:12:05 INFO 286638 PKey init: port 2, pkey = 0xffff
2018-01-11 11:12:05 INFO 286638 PKey cleanup: echo none > /sys/class/infiniband/mlx4_0/iov/0000:03:00.1/ports/2/pkey_idx/[0-127]
2018-01-11 11:12:06 INFO 286638 PKey config: echo 0 > /sys/class/infiniband/mlx4_0/iov/0000:03:00.1/ports/2/pkey_idx/0
2018-01-11 11:12:06 WARNING 286638 send_to_guest num_hca (1)
2018-01-11 11:12:06 INFO 286638 Write { 'num_hca' : '1' } to domian 14
2018-01-11 11:12:06 WARNING 286638 /usr/sbin/xm pci-attach 514ca136-afbe-4e64-9571-3dac3aaec43c 0000:03:00.1. (1) <<<<<<---------------- PCI attach giving a warning during VM startup
2018-01-11 11:12:06 INFO 286638 VF lock file /tmp/vm_event_callout_IB.lck unlocked.
2018-01-11 11:12:06 INFO 286638 /usr/sbin/xm unpause 514ca136-afbe-4e64-9571-3dac3aaec43c
2018-01-11 11:12:07 ERROR 286638 /usr/sbin/xm unpause 514ca136-afbe-4e64-9571-3dac3aaec43c. uuid = 514ca136-afbe-4e64-9571-3dac3aaec43c (1)
2018-01-11 11:12:07 ERROR 286638 Cannot unpause VM 514ca136-afbe-4e64-9571-3dac3aaec43c (err = 1).
2018-01-11 11:12:07 INFO 286638 uuid = 514ca136-afbe-4e64-9571-3dac3aaec43c, Command = config_booted exiting.

While attaching device manually using xm pci-attch shows below error. PCI pass through attach will not work if PoD is enabled.

[root@exadatanode]# /usr/sbin/xm pci-attach 514ca136-afbe-4e64-9571-3dac3aaec43c 0000:03:00.1
Error: failed to assign device since pod is enabled <<<<<<<<<<---------------------------------------- PCI attach failed with error
Usage: xm pci-attach [-o|--options=<opt>] <Domain> <domain:bus:slot.func> [virtual slot]

Insert a new pass-through pci device.

Changes

VM's configuration changed to set memory = '60000' and maxmem = '100000' prior to reboot

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms