Grid Infrastructure Startup During Patching, Install or Upgrade May Fail Due to Multicasting Requirement
(Doc ID 1212703.1)
Last updated on JULY 16, 2020
Applies to:
Oracle Database - Enterprise Edition - Version 11.2.0.2 and laterOracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Information in this document applies to any platform.
This issue impacts environments that do not have multicast enabled for the private network in the following situations:
New installations of Oracle Grid Infrastructure 11.2.0.2 where multicast is not enabled on 230.0.1.0
Upgrades to Oracle Grid Infrastructure 11.2.0.2 from a pre-11.2.0.2 release where multicast is not enabled on 230.0.1.0 or 224.0.0.251
Installation of GI PSU 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 where multicast is not enabled on 230.0.1.0 or 224.0.0.251
Installation or upgrade to 12.1.0.1.0 where multicast is not enabled on 230.0.1.0 or 224.0.0.251
Symptoms
If multicast based communication is not enabled as required either on the nodes of the cluster or on the network switches used for the private interconnect, the root.sh, which is called as part of a fresh installation of Oracle Grid Infrastructure 11.2.0.2, or the rootupgrade.sh (called as part of an upgrade to Oracle Grid Infrastructure 11.2.0.2) will only succeed on the first node of the cluster, but will fail on subsequent nodes with the symptoms shown below:
An active cluster was found during exclusive startup, restarting to join the cluster
Failed to start Oracle Clusterware stack
Failed to start Cluster Synchorinisation Service in clustered mode at /u01/app/crs/11.2.0.2/crs/install/crsconfig_lib.pm line 1016.
/u01/app/crs/11.2.0.2/perl/bin/perl -I/u01/app/crs/11.2.0.2/perl/lib -I/u01/app/crs/11.2.0.2/crs/install /u01/app/crs/11.2.0.2/crs/install/rootcrs.pl execution failed
Symptom verification
To verify that Oracle CSS daemon fails to start in clustered mode due to a multicasting issue on the network, the ocssd.log file (located under $GI_HOME/log/<nodename>/cssd/ocssd.log) must be reviewed. In case, joining the cluster fails because of such an issue, the following can be observed:
1. When CSS starts in clustered mode to join an existing cluster, we will see an entry in the CSSD log indicating that CSS will attempt to establish communication with a peer in the cluster. For this analysis, we see in the CSSD log for <node2> that communication is attempted with <node1>, which looks similar to:
2. Shortly after the above log entry we will see an attempt to establish communication to <node1> from <node2> via multicast address 230.0.1.0, port 42424 on the private interconnect (here: 192.168.x.x):
3. If the communication can be established successfully, we will see a log entry on node2 containing "gipchaLowerProcessAcks: ESTABLISH finished" for the peer node (<node1>). If the communication cannot be established, we will not see this log entry. Instead, we will see an entry indicating that the network communication cannot be established. This entry will look similar to the one shown below:
The above log entry indicates that CSSD is unable to establish network communication on the interface used for the private interconnect. In this particular case, the issue was that multicast communication on the 230.0.1.0 IP was blocked on the network used as the private interconnect.
Changes
- New installations of Oracle Grid Infrastructure 11.2.0.2
- Upgrades of a previous release to Oracle Grid Infrastructure 11.2.0.2
- Installation of the 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 GI PSUs where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses
- New installations of Oracle Grid Infrastructure 12.1.0.1 where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses
- Upgrades of a previous release to Oracle Grid Infrastructure 12.1.0.1 where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
Symptoms |
Changes |
Cause |
Solution |
Scalability RAC Community |
References |