Grid Infrastructure Startup During Patching, Install or Upgrade May Fail Due to Multicasting Requirement
(Doc ID 1212703.1)
Last updated on JULY 16, 2020
Applies to:Oracle Database - Enterprise Edition - Version 126.96.36.199 and later
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Information in this document applies to any platform.
This issue impacts environments that do not have multicast enabled for the private network in the following situations:
New installations of Oracle Grid Infrastructure 188.8.131.52 where multicast is not enabled on 184.108.40.206
Upgrades to Oracle Grid Infrastructure 220.127.116.11 from a pre-18.104.22.168 release where multicast is not enabled on 22.214.171.124 or 126.96.36.199
Installation of GI PSU 188.8.131.52.5, 184.108.40.206.6, 220.127.116.11.7 where multicast is not enabled on 18.104.22.168 or 22.214.171.124
Installation or upgrade to 126.96.36.199.0 where multicast is not enabled on 188.8.131.52 or 184.108.40.206
If multicast based communication is not enabled as required either on the nodes of the cluster or on the network switches used for the private interconnect, the root.sh, which is called as part of a fresh installation of Oracle Grid Infrastructure 220.127.116.11, or the rootupgrade.sh (called as part of an upgrade to Oracle Grid Infrastructure 18.104.22.168) will only succeed on the first node of the cluster, but will fail on subsequent nodes with the symptoms shown below:
An active cluster was found during exclusive startup, restarting to join the cluster
Failed to start Oracle Clusterware stack
Failed to start Cluster Synchorinisation Service in clustered mode at /u01/app/crs/22.214.171.124/crs/install/crsconfig_lib.pm line 1016.
/u01/app/crs/126.96.36.199/perl/bin/perl -I/u01/app/crs/188.8.131.52/perl/lib -I/u01/app/crs/184.108.40.206/crs/install /u01/app/crs/220.127.116.11/crs/install/rootcrs.pl execution failed
To verify that Oracle CSS daemon fails to start in clustered mode due to a multicasting issue on the network, the ocssd.log file (located under $GI_HOME/log/<nodename>/cssd/ocssd.log) must be reviewed. In case, joining the cluster fails because of such an issue, the following can be observed:
1. When CSS starts in clustered mode to join an existing cluster, we will see an entry in the CSSD log indicating that CSS will attempt to establish communication with a peer in the cluster. For this analysis, we see in the CSSD log for <node2> that communication is attempted with <node1>, which looks similar to:
2. Shortly after the above log entry we will see an attempt to establish communication to <node1> from <node2> via multicast address 18.104.22.168, port 42424 on the private interconnect (here: 192.168.x.x):
3. If the communication can be established successfully, we will see a log entry on node2 containing "gipchaLowerProcessAcks: ESTABLISH finished" for the peer node (<node1>). If the communication cannot be established, we will not see this log entry. Instead, we will see an entry indicating that the network communication cannot be established. This entry will look similar to the one shown below:
The above log entry indicates that CSSD is unable to establish network communication on the interface used for the private interconnect. In this particular case, the issue was that multicast communication on the 22.214.171.124 IP was blocked on the network used as the private interconnect.
- New installations of Oracle Grid Infrastructure 126.96.36.199
- Upgrades of a previous release to Oracle Grid Infrastructure 188.8.131.52
- Installation of the 184.108.40.206.5, 220.127.116.11.6, 18.104.22.168.7 GI PSUs where multicast is not enabled on the 22.214.171.124 or 126.96.36.199 multicast addresses
- New installations of Oracle Grid Infrastructure 188.8.131.52 where multicast is not enabled on the 184.108.40.206 or 220.127.116.11 multicast addresses
- Upgrades of a previous release to Oracle Grid Infrastructure 18.104.22.168 where multicast is not enabled on the 22.214.171.124 or 126.96.36.199 multicast addresses
To view full details, sign in with your My Oracle Support account.
Don't have a My Oracle Support account? Click to get started!
In this Document
|Scalability RAC Community|