My Oracle Support Banner

Corosync PaceMaker: A Resource Monitor Fails with LRMd : lrmd[XXXX]: warning: processname_process_instance_monitor_XXX:XXX - timed out after 30000ms (Doc ID 2658927.1)

Last updated on SEPTEMBER 17, 2021

Applies to:

Linux OS - Version Oracle Linux 7.3 and later
Linux x86-64

Symptoms

 An Corosync Pacemaker node was fenced. On the OS logs Is visible the following error:

 

Mar 24 04:32:42 <HOSTNAME2> lrmd[14166]: warning: db2_oracle_instance_monitor_120000 process (PID 83418) timed out <<<<<<<<<<<<<
Mar 24 04:32:42 <HOSTNAME2> lrmd[14166]: warning: db2_oracle_instance_monitor_120000:83418 - timed out after 30000ms <<<<<<<<<<<<<
Mar 24 04:32:42 <HOSTNAME2> crmd[14170]: error: Result of monitor operation for db2_oracle_instance on <HOSTNAME2>: Timed Out <<<<<<<<<<<<<
Mar 24 04:32:42 <HOSTNAME2> crmd[14170]: notice: State transition S_IDLE -> S_POLICY_ENGINE <<<<<<<<<<<<<
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Processing failed start of <HOSTNAME2>-NEW-PROCESS on <HOSTNAME1>: unknown error 
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Processing failed start of <HOSTNAME1>-NEW-ILOSGH423JHMW on <HOSTNAME1>: unknown error 
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Processing failed monitor of db2_oracle_instance on <HOSTNAME2>: unknown error
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Processing failed start of <HOSTNAME2>-NEW-PROCESS on <HOSTNAME2>: unknown error 
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Processing failed start of <HOSTNAME1>-NEW-PROCESS on <HOSTNAME2>: unknown error
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Forcing <HOSTNAME2>-NEW-PROCESS away from <HOSTNAME1> after 1000000 failures (max=1000000)
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Forcing <HOSTNAME1>-NEW-PROCESS away from <HOSTNAME1> after 1000000 failures (max=1000000) <<<<<<<<<< <<<
Mar 24 04:32:43 <HOSTNAME2>pengine[14169]: warning: Forcing <HOSTNAME2>-NEW-PROCESS away from <HOSTNAME2>after 1000000 failures (max=1000000)
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Forcing <HOSTNAME1>-NEW-PROCESS away from <HOSTNAME2>after 1000000 failures (max=1000000) <<<<<<<<<<<<<
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: notice: * Recover db2_oracle_instance ( <HOSTNAME2> )
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: notice: * Restart db2_oracle_listener ( <HOSTNAME2> ) due to required db2_oracle_instance start <<<<<<<<<<<<<
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: notice: Calculated transition 6361, saving inputs in /var/lib/pacemaker/pengine/pe-input-621.bz2
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Processing failed start of <HOSTNAME2>-NEW-PROCESS on <HOSTNAME1>: unknown error
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Processing failed start of <HOSTNAME1>-NEW-PROCESS on <HOSTNAME1>: unknown error
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Processing failed monitor of db2_oracle_instance on <HOSTNAME2>: unknown error
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Processing failed start of <HOSTNAME2>-NEW-PROCESS on <HOSTNAME2>: unknown error
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Processing failed start of <HOSTNAME1>-NEW-ILOSGH423JHMW on <HOSTNAME2>: unknown error
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Forcing <HOSTNAME2>-NEW-PROCESS away from <HOSTNAME1> after 1000000 failures (max=1000000)
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Forcing <HOSTNAME1>-NEW-PROCESS away from <HOSTNAME1> after 1000000 failures (max=1000000)
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Forcing <HOSTNAME2>-NEW-PROCESS away from <HOSTNAME2> after 1000000 failures (max=1000000)
Mar 24 04:32:43 <HOSTNAME2> pengine[14169]: warning: Forcing <HOSTNAME1>-NEW-PROCESS away from <HOSTNAME2> after 1000000 failures (max=1000000)

<SNIP>

Mar 24 04:34:47 <HOSTNAME2> crmd[14170]: notice: Requesting fencing (reboot) of node sgbpcmprtdbp02 <<<<<<<<<<<<<  <<<<<<<<<<<<<
Mar 24 04:34:47 <HOSTNAME2> stonith-ng[14164]: notice: Client crmd.14170.6ece3fc9 wants to fence (reboot) '<HOSTNAME2>' with device '(any)' <<<<<<<<<<<<<
Mar 24 04:34:47 <HOSTNAME2> stonith-ng[14164]: notice: Requesting peer fencing (reboot) of <HOSTNAME2><<<<<<<<<<<<<

Changes

 N/A

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.