Grid Infrastructure Startup During Patching, Install or Upgrade May Fail Due to Multicasting Requirement
(Doc ID 1212703.1)
Last updated on FEBRUARY 14, 2019
Applies to:Oracle Database - Enterprise Edition - Version 22.214.171.124 and later
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Information in this document applies to any platform.
This issue impacts environments that do not have multicast enabled for the private network in the following situations:
New installations of Oracle Grid Infrastructure 126.96.36.199 where multicast is not enabled on 188.8.131.52
Upgrades to Oracle Grid Infrastructure 184.108.40.206 from a pre-220.127.116.11 release where multicast is not enabled on 18.104.22.168 or 22.214.171.124
Installation of GI PSU 126.96.36.199.5, 188.8.131.52.6, 184.108.40.206.7 where multicast is not enabled on 220.127.116.11 or 18.104.22.168
Installation or upgrade to 22.214.171.124.0 where multicast is not enabled on 126.96.36.199 or 188.8.131.52
If multicast based communication is not enabled as required either on the nodes of the cluster or on the network switches used for the private interconnect, the root.sh, which is called as part of a fresh installation of Oracle Grid Infrastructure 184.108.40.206, or the rootupgrade.sh (called as part of an upgrade to Oracle Grid Infrastructure 220.127.116.11) will only succeed on the first node of the cluster, but will fail on subsequent nodes with the symptoms shown below:
An active cluster was found during exclusive startup, restarting to join the cluster
Failed to start Oracle Clusterware stack
Failed to start Cluster Synchorinisation Service in clustered mode at /u01/app/crs/18.104.22.168/crs/install/crsconfig_lib.pm line 1016.
/u01/app/crs/22.214.171.124/perl/bin/perl -I/u01/app/crs/126.96.36.199/perl/lib -I/u01/app/crs/188.8.131.52/crs/install /u01/app/crs/184.108.40.206/crs/install/rootcrs.pl execution failed
To verify that Oracle CSS daemon fails to start in clustered mode due to a multicasting issue on the network, the ocssd.log file (located under $GI_HOME/log/<nodename>/cssd/ocssd.log) must be reviewed. In case, joining the cluster fails because of such an issue, the following can be observed:
1. When CSS starts in clustered mode to join an existing cluster, we will see an entry in the CSSD log indicating that CSS will attempt to establish communication with a peer in the cluster. For this analysis, we see in the CSSD log for <node2> that communication is attempted with <node1>, which looks similar to:
2. Shortly after the above log entry we will see an attempt to establish communication to <node1> from <node2> via multicast address 220.127.116.11, port 42424 on the private interconnect (here: 192.168.x.x):
3. If the communication can be established successfully, we will see a log entry on node2 containing "gipchaLowerProcessAcks: ESTABLISH finished" for the peer node (<node1>). If the communication cannot be established, we will not see this log entry. Instead, we will see an entry indicating that the network communication cannot be established. This entry will look similar to the one shown below:
The above log entry indicates that CSSD is unable to establish network communication on the interface used for the private interconnect. In this particular case, the issue was that multicast communication on the 18.104.22.168 IP was blocked on the network used as the private interconnect.
- New installations of Oracle Grid Infrastructure 22.214.171.124
- Upgrades of a previous release to Oracle Grid Infrastructure 126.96.36.199
- Installation of the 188.8.131.52.5, 184.108.40.206.6, 220.127.116.11.7 GI PSUs where multicast is not enabled on the 18.104.22.168 or 22.214.171.124 multicast addresses
- New installations of Oracle Grid Infrastructure 126.96.36.199 where multicast is not enabled on the 188.8.131.52 or 184.108.40.206 multicast addresses
- Upgrades of a previous release to Oracle Grid Infrastructure 220.127.116.11 where multicast is not enabled on the 18.104.22.168 or 22.214.171.124 multicast addresses
To view full details, sign in with your My Oracle Support account.
Don't have a My Oracle Support account? Click to get started!
In this Document
|Scalability RAC Community|