T7-2 System not booting. Timeout Waiting for SP reset to Clear (Doc ID 2355213.1)

Last updated on FEBRUARY 26, 2021

Applies to:

Symptoms

Before Hypervisor transitions to OBP, System may hang with message, NOTICE: Waiting to send a message to the SP

From event log:

21 Tue Nov 28 07:26:04 2017 System Log major
FPGA initialization failed. Running in degraded mode.
20 Tue Nov 28 07:22:42 2017 System Log critical
Failed to read FPGA address 0x2d19020 with status -1
19 Tue Nov 28 06:58:43 2017 System Log minor
Host: Powered On

From console log:

2017-11-28 05:57:40.804 0:0:0:0>IOH PCIe Configuration Space Register Test
2017-11-28 05:57:41.951 0:0:0:0>IOH PCIe Memory Space Register Test
2017-11-28 05:57:44.022 0:0:0:0>Setup Tunables
2017-11-28 05:57:44.162 0:0:0:0>HB PCIE LINK ENABLE
2017-11-28 05:58:35.199 0:0:0:0>WARNING: Timeout waiting for SP reset to clear
2017-11-28 06:00:06.930 0:0:0:0>WARNING: Timeout waiting for SP reset to clear
2017-11-28 06:00:58.148 0:0:0:0>WARNING: Timeout waiting for SP reset to clear
2017-11-28 06:01:49.366 0:0:0:0>WARNING: Timeout waiting for SP reset to clear
2017-11-28 06:01:50.131 0:0:0:0>ERROR:
ereport.chassis.post.io.test-fail@/SYS/MB/IOH0/IOS4/RP0/PCIE_LINK
reporting PCPU ID=0
TestTitle=HB PCIE LINK ENABLE
Operation=Check PCIE link trained (DLL Active)
END_ERROR
2017-11-28 06:02:06.947 0:0:0:0>FDD Message to GM timed out

.
2017-11-28 06:12:04.278 0:0:0:0>Setup LE TSB Page
2017-11-28 06:12:04.361 0:0:0:0>CHECK PCI TOPOLOGY
2017-11-28 06:12:04.416 0:0:0:0> NAC VID DID Width Speed Status
2017-11-28 06:12:04.436 0:0:0:0>--------------------------------------------------------------------------------
2017-11-28 06:12:04.456 0:0:0:0> /SYS/MB/IOH0/IOS0/RP0/PCIE_LINK 108e b008 8 G3 Clean
2017-11-28 06:12:04.477 0:0:0:0> /SYS/MB/SASHBA0 1000 0097 8 G3 Clean
2017-11-28 06:12:04.497 0:0:0:0> /SYS/MB/IOH0/IOS0/RP1/PCIE_LINK 108e b008 0 G0 Clean
2017-11-28 06:12:04.519 0:0:0:0> /SYS/MB/PCIE4 ------------------------------
2017-11-28 06:12:04.539 0:0:0:0> /SYS/MB/IOH0/IOS1/RP0/PCIE_LINK 108e b008 8 G2 Clean
2017-11-28 06:12:04.560 0:0:0:0> /SYS/MB/NET0 8086 1528 8 G2 Clean
2017-11-28 06:12:04.581 0:0:0:0> /SYS/MB/NET1 8086 1528 8 G2 Clean
2017-11-28 06:12:04.602 0:0:0:0> /SYS/MB/IOH0/IOS1/RP1/PCIE_LINK 108e b008 0 G0 Clean
2017-11-28 06:12:04.622 0:0:0:0> /SYS/MB/PCIE3 ------------------------------
2017-11-28 06:12:04.643 0:0:0:0> /SYS/MB/IOH0/IOS2/RP0/PCIE_LINK 108e b008 4 G3 Clean
2017-11-28 06:12:04.664 0:0:0:0> /SYS/MB/PCIE2 1077 2031 4 G3 Clean
2017-11-28 06:12:04.685 0:0:0:0> /SYS/MB/PCIE2 1077 2031 4 G3 Clean
2017-11-28 06:12:04.706 0:0:0:0> /SYS/MB/IOH0/IOS3/RP0/PCIE_LINK 108e b008 0 G0 Clean
2017-11-28 06:12:04.727 0:0:0:0> /SYS/MB/PCIE1 ------------------------------
2017-11-28 06:12:04.747 0:0:0:0> /SYS/MB/IOH0/IOS4/RP0/PCIE_LINK -----------ERRORS!------------ *******
2017-11-28 06:12:04.768 0:0:0:0> /SYS/MB/SPM ------------------------------
2017-11-28 06:12:04.788 0:0:0:0> /SYS/MB/IOH1/IOS0/RP0/PCIE_LINK 108e b008 8 G3 Clean
2017-11-28 06:12:04.809 0:0:0:0> /SYS/MB/SASHBA1 1000 0097 8 G3 Clean
2017-11-28 06:12:04.830 0:0:0:0> /SYS/MB/IOH1/IOS0/RP1/PCIE_LINK 108e b008 4 G3 Clean
2017-11-28 06:12:04.851 0:0:0:0> /SYS/MB/PCIE6 1077 2031 4 G3 Clean
2017-11-28 06:12:04.872 0:0:0:0> /SYS/MB/PCIE6 1077 2031 4 G3 Clean
2017-11-28 06:12:04.893 0:0:0:0> /SYS/MB/IOH1/IOS1/RP0/PCIE_LINK 108e b008 0 G0 Clean
2017-11-28 06:12:04.914 0:0:0:0> /SYS/MB/PCIE8 ------------------------------
2017-11-28 06:12:04.934 0:0:0:0> /SYS/MB/IOH1/IOS2/RP0/PCIE_LINK 108e b008 0 G0 Clean
2017-11-28 06:12:04.955 0:0:0:0> /SYS/MB/PCIE7 ------------------------------
2017-11-28 06:12:04.975 0:0:0:0> /SYS/MB/IOH1/IOS3/RP0/PCIE_LINK 108e b008 8 G2 Clean
2017-11-28 06:12:04.996 0:0:0:0> /SYS/MB/NET2 8086 1528 8 G2 Clean
2017-11-28 06:12:05.017 0:0:0:0> /SYS/MB/NET3 8086 1528 8 G2 Clean
2017-11-28 06:12:05.038 0:0:0:0> /SYS/MB/IOH1/IOS3/RP1/PCIE_LINK 108e b008 0 G0 Clean
2017-11-28 06:12:05.059 0:0:0:0> /SYS/MB/PCIE5 ------------------------------
2017-11-28 06:12:05.079 0:0:0:0> /SYS/MB/IOH1/IOS4/RP0/PCIE_LINK 108e b008 1 G2 Clean
2017-11-28 06:12:05.100 0:0:0:0> /SYS/MB/USB_CTRL 104c 8241 1 G2 Clean
2017-11-28 06:12:05.135 0:0:0:0>Disable IOH Link Errors

ILOM/Traces/CRIT.log

OTHER CRITICAL 2017-11-28 05:57:55.840166 1693 fpga_plat_emx.c:776 fpga_data_get8(0x2d13001) = 0x00, err=-1
OTHER CRITICAL 2017-11-28 05:57:59.250237 2106 fpga_plat_emx.c:530 fpga_data_put8(0x2d19128) <= 0xc6, err=-1
libfpga_plat.so.1(fpga_data_put8+0x168) [0xb6630fc4]
libfpga_plat.so.1(fpga_put8+0x18) [0xb6631018]
libhalcommon.so.1(hc_fpga_reenable_system_irq+0x44) [0xb6851348]
pod(host_sys_irq_cb+0x5d4) [0x66f48]
pod(pod_gpioint_testfds+0x17c) [0x675b0]
pod() [0x6796c]
OTHER CRITICAL 2017-11-28 05:58:08.330221 1954 fpga_plat_emx.c:776 fpga_data_get8(0x2d30000) = 0xb5, err=-1
OTHER CRITICAL 2017-11-28 05:58:18.910259 2107 fpga_plat_emx.c:776 fpga_data_get8(0x2d13580) = 0x00, err=-1
OTHER CRITICAL 2017-11-28 05:58:29.180101 1693 fpga_plat_emx.c:776 fpga_data_get8(0x2d13081) = 0x00, err=-1
OTHER CRITICAL 2017-11-28 05:58:32.490141 2106 fpga_plat_emx.c:776 fpga_data_get8(0x2d19022) = 0x00, err=-1
OTHER CRITICAL 2017-11-28 05:58:41.570206 1954 fpga_plat_emx.c:530 fpga_data_put8(0x2d30000) <= 0xff, err=-1
libfpga_plat.so.1(fpga_data_put8+0x168) [0xb677cfc4]
libfpga_plat.so.1(fpga_put8+0x18) [0xb677d018]
libhalcommon.so.1(hc_set_fpga_gm_status+0x2c)

SP messages:

Dec 11 15:45:07 kernel: USER1 irq possibly not serviced.
Clearing it
Dec 11 15:45:10 kernel: parity mismatch 0x0!=0x1 on read8
0x2d13580, data 0x2, lpc_down=1, reg0=0x82d13580, tries=5
Dec 11 15:45:10 kernel: siocfg8 : 0x0
Dec 11 15:45:10 kernel: siocfg3 : 0xa
Dec 11 15:45:10 kernel: power good : 0x1
Dec 11 15:45:11 kernel: sys_irq=0xff resending irq
Dec 11 15:45:11 kernel: parity mismatch 0x0!=0x1 on read8
0x2d32000, data 0x2, lpc_down=1, reg0=0x2d32000, tries=5
Dec 11 15:45:11 kernel: siocfg8 : 0x0
Dec 11 15:45:11 kernel: siocfg3 : 0xa
Dec 11 15:45:11 kernel: power good : 0x1
Dec 11 15:45:27 kernel: sys_irq=0x0 resending irq
Dec 11 15:46:01 last message repeated 2 times
Dec 11 15:46:35  last message repeated 2 times
Dec 11 15:46:51  kernel: USER1 irq possibly not serviced.
Clearing it
Dec 11 15:46:52  kernel: parity mismatch 0x0!=0x1 on read8
0x2d13580, data 0x2, lpc_down=1, reg0=0x82d13580, tries=5
Dec 11 15:46:52  kernel: siocfg8 : 0x0
Dec 11 15:46:52  kernel: siocfg3 : 0xa
Dec 11 15:46:52  kernel: power good : 0x1
Dec 11 15:46:55  kernel: sys_irq=0xff resending irq

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.