My Oracle Support Banner

ACFS Hang/Freeze During Node Reboot, After Several Million Files Were Created in ACFS (Doc ID 1416386.1)

Last updated on FEBRUARY 15, 2019

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.

Symptoms

It has been observed in a clustered 11.2.0.3 Grid Infrastructure environment that ACFS will hang/freeze for a few minutes (say 4~5 minutes) during node reboot, after several million files (say, 5 million) were created in ACFS.

OKS log:
~~~~~~~~~~
Dec 20 06:20:23 <HOSTNAME1> kernel: [Oracle OKS] Cluster Membership change - Current incarn 0x18
Dec 20 06:20:23 <HOSTNAME1> kernel: [Oracle OKS] Nodes in cluster:
Dec 20 06:20:24 <HOSTNAME1> kernel: [Oracle OKS] <HOSTNAME1> (IP 0xa9fe5000) <HOSTNAME2> (IP 0xa9fe2000)
Dec 20 06:20:24 <HOSTNAME1> kernel: [Oracle OKS] Node count 2, Local node number 1
Dec 20 06:20:24 <HOSTNAME1> kernel: ADVMK-00013: Cluster reconfiguration started.
Dec 20 06:20:52 <HOSTNAME1> ntpd[19679]: synchronized to xx.x.xx.xx, stratum 1
Dec 20 06:20:52 <HOSTNAME1> ntpd[19679]: synchronized to xx.x.xx.xx, stratum 1
Dec 20 06:21:50 <HOSTNAME1> hpasmlited[20068]: hpDeferSPDThread: End of Collecting DIMM SPD data.
Dec 20 06:21:50 <HOSTNAME1> hpasmlited[20068]: hpDeferSPDThread: End of Collecting DIMM SPD data.
Dec 20 06:24:47 <HOSTNAME1> kernel: ADVMK-00014: Cluster reconfiguration completed.
Dec 20 06:24:47 <HOSTNAME1> kernel: ADVMK-00014: Cluster reconfiguration completed.      <---- Took over 4 minutes


Note: To get a memory dump of the oks logs on a live system you can exec "cat /proc/oks/log > /tmp/oks.log.<hostname>" on each node.
Starting with 11.2.0.3.0, persistent oks logs are kept on each node. To determine their location, do: /sbin/acfsutil plogconfig -q

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.