After CRS installation, ONS can not start on 1 node

(Doc ID 744849.1)

Last updated on OCTOBER 26, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 10.2.0.1 and later
Information in this document applies to any platform.
***Checked for relevance on 23-Feb-2011***


Symptoms

2 node RAC, after CRS installation, ons is up on node 2 but can not start on node 1

onsctl start

Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
{node = node1, port = 6251}
Adding remote host node1:6251
onscfg[1]
{node = node2, port = 6251}
Adding remote host node2:6251
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
{node = node1, port = 6251}
Adding remote host node1:6251
onscfg[1]
{node = node2, port = 6251}
Adding remote host node2:6251
onsctl: ons failed to start


ons.log shows

08/10/09 11:31:39 [2] BIND (Address already in use)
08/10/09 11:31:39 [2] 0.0.24.6:6150 - listener BIND failed
08/10/09 11:31:39 [1] Local listener terminated
08/10/09 16:18:10 [2] BIND (Address already in use)
08/10/09 16:18:10 [2] 0.0.24.6:6150 - listener BIND failed
08/10/09 16:18:10 [1] Local listener terminated


strace for onsctl start shows

23448 16:35:08.955359 connect(7, {sa_family=AF_INET, sin_port=htons(6251),
sin_addr=inet_addr("122.6.8.1")}, 16<unfinished ...>
23449 16:35:08.955379 connect(8, {sa_family=AF_INET, sin_port=htons(6251),
sin_addr=inet_addr("122.6.8.2")}, 16<unfinished ...>
23450 16:35:08.955397 nanosleep({1, 0}, <unfinished ...>
23448 16:35:08.955459 <... connect resumed> ) = -1 ECONNREFUSED (Connection refused)
23448 16:35:08.955484 shutdown(7, 2 /* send and receive */) = -1 ENOTCONN (Transport endpoint is
not connected)
23448 16:35:08.955508 close(7) = 0
23449 16:35:08.955552 <... connect resumed> ) = 0
23448 16:35:08.955568 clock_gettime(CLOCK_REALTIME, <unfinished ...>
23449 16:35:08.955591 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
23448 16:35:08.955624 <... clock_gettime resumed> {1223588108, 955621000}) = 0


gdb against core file shows

(gdb) where
#0 0x000000000040cb56 in opmnHttpFormatConnect ()
#1 0x0000000000408d19 in connectionActive ()
#2 0x0000000000407774 in opmnConnectionEntry ()
#3 0x00000039c8c06137 in start_thread () from /lib64/tls/libpthread.so.0
#4 0x00000039c85c9883 in clone () from /lib64/tls/libc.so.6

Changes

New installation.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms