Last updated on JUNE 19, 2017

Applies to:

Symptoms

• The LGWR appears to be encountering regular problems where it hangs for 3 seconds at a time.
  During this time, an iostat -x on the redo devices shows zero IO being pushed through.
• Veritas ODM (Oracle Disk Manager) is being used
• LGWR trace shows numerous entries such as:
  ...
  *** 2010-11-01 07:33:14.745
  Warning: log write elapsed time 3203ms, size 1KB
  (set event 10468 level 4 to disable this warning)
  *** 2010-11-01 07:52:57.813
  Warning: log write elapsed time 3345ms, size 3KB
  (set event 10468 level 4 to disable this warning)
  ...
• The average wait times for log file sync do not typically appear to be made up of IO waits (there is no evidence of 3 second IOs):
  NAME                    TOTAL_WAITS TOTAL_WAIT_TIME   AVG_TIME_WAITED
  ----------------------- ----------- ----------------- -----------------
  log file sync                    84         12684.472           151.006
  log file parallel write       17655         28848.331             1.634

  The affect on any session in the database at this point is to wait with a log file sync event, effectively stalling the system for 3 seconds.

• When dtrace was run to trace the issue, the problem went away.  Problem came back as soon as dtrace was stopped.
• Truss on LGWR also avoids the issue.
• At the times when LGWR suffers the slow write, the process is given a priority of 0. So it looks like the operating system is scheduling the LGWR process off CPU and it takes 3 seconds to get back on and catch up. 
  Dtrace looks to resolve the issue by either keeping the lgwr process on CPU or ensuring if it does come off CPU, it is re-instated immediately.
• System is NOT under load when this happens. It even happens more during the quiet periods.
• Renice the LDWR lgwr process makes no difference.
• From 10298 trace, the modules immediately prior to the waits are not the same and are sometimes IO related and are sometimes not which indicates that the delay is not necessarily down to a wait for I/O completion.

Changes

none

Cause