ODA VMs and REPO in UNKNOWN State; OS, Xen or OAK Commands Hangs ; High IO Looping ; TOP High I/O Wait, INFO: Task NFSD Increments One After Another Spins (Doc ID 2403746.1)

Last updated on APRIL 12, 2024

Applies to:

Symptoms

You may encounter two or more of the following confirmed symptoms:

• VMs and REPO in UNKNOWN State
• OS, Xen or OAK Commands Hangs
• High IO Looping
• TOP High I/O Wait,
• "INFO: task nfsd:"#### "blocked for more than 120 seconds"
• lo_rw_aio_complete:454:ERROR:-116 received. Device loop10 MAJOR:7 MI

VM and Repo Symptoms

• VM unresponsiveness going into UNKNOWN state.
• VM's that were running on the two nodes disappeared.
• Attempting to shutdown the repos will fail
• Eventually the oakcli command may show no REPO filesystems present -- athough they may still show as mounted.

High IO, or Load

• Servers experience an I/O on loop devices that belong to VMs
• high load average (> 100) on basenode with high CPU I/O wait utilization
• TOP may show 50% for IO

Hang

• "df -h" where NFS mounts appear to be hung
• "crsctl check crs" is unable to the cluster ready services.
• The srvctl command from ODA_BASE canhang
• The server(s) become unresponsive = effectively locked up.
• Gracefully Power Off the DOM 0's at the ILOM level may fail.
• Power down from ILOM and startup manually may be required.

Messages file errors:

• sequential NFSD tasks incrementally fail as they are "blocked for more than 120 seconds"
• lo_rw_aio_complete:454:ERROR:-116 received. Device loop10 MAJOR:7 MINOR:10

 From the messages files you may find a few key indicators

Changes

 12.2.1.2 ( or ODA X7-2 12.2.1.1) 

Cause