My Oracle Support Banner

OC4J Instances Fail to Start With Error "Address already in use" for AJP, RMI and JMS ports On Solaris Servers With IPv4/IPv6 (Doc ID 1575278.1)

Last updated on OCTOBER 10, 2018

Applies to:

Oracle Fusion Middleware - Version 10.1.3.0.0 and later
Oracle Solaris on x86-64 (64-bit)
Oracle Solaris on x86 (32-bit)
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on SPARC (32-bit)

Symptoms

 

In Solaris 10, when using multiple Oracle AS 10.1.3.5 installations, some of the OC4J instances could not be started due to multiple proceses using the same ports and clashing (specifically jms port). 

The error was reproduced in Oracle AS 10.1.3.5.0 running on Solaris Sparc 10 zone hosted by the Sun SuperCluster's Solaris 11 global zone, but it is not dependant on the usage of zones or other virtualizations. Theorically this could affect any 10.1.3.X install in Solaris SPARC and x86 servers if both IPv4 and IPv6 stacks are enabled.

The problem is detected as a port conflict by the OC4J JVM when it starts, but not by OPMN, which identifies some ports as free when they are actually in use and hence produces the effect that OPMN does not handle correctly the "in use" ports.

 

Given the root cause this problem could be theorically reproduced when there is a port range overlap in the OC4J components, as for example in the following different setups:

a) Multiple Oracle AS 10.1.3.X installations in the same physical server, when the OC4J instances share the same port ranges

b) One single Oracle AS 10.1.3.X installation with multiple OC4J instances, when the OC4J instances share the same port ranges

c) One single Oracle AS 10.1.3.X installation with OC4J instances having numprocs > 1, as all the JVM's for that OC4J instance will share the same port ranges

 

Testing Scenario

This testing session was conducted in scenario "a) Multiple Oracle AS 10.1.3.X installations in the same physical server, when the OC4J instances share the same port ranges"

In EnvA (Test1) we have an up and running OC4J instance using ports some of the "suspicious" ports 12401 12501 12601 12701

Processes in Instance: Test1
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component                    | process-type       |     pid | status   |        uid |  memused |    uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
OC4JGroup:1_group                | OC4J:oc4j_1        |   25682 | Alive    | xxxxxxxxxx |   230280 |   4:35:03 | rmi:12409,rmis:12709,jms:12609,ajp:12511
OC4JGroup:2_group                | OC4J:oc4j_2        |   25680 | Alive    | xxxxxxxxxx |   794272 |   4:35:03 | jms:12608,ajp:12510,rmis:12708,rmi:12408
OC4JGroup:3_group                | OC4J:oc4j_3        |   25681 | Alive    | xxxxxxxxxx |   304016 |   4:35:03 | jms:12607,ajp:12509,rmis:12707,rmi:12407
OC4JGroup:4_group                | OC4J:oc4j_4        |   25679 | Alive    | xxxxxxxxxx |   363816 |   4:35:03 | jms:12606,ajp:12508,rmis:12706,rmi:12406
OC4JGroup:5_group                | OC4J:oc4j_5        |   25678 | Alive    | xxxxxxxxxx |   632712 |   4:35:03 | jms:12605,ajp:12507,rmis:12705,rmi:12405
OC4JGroup:6_group                | OC4J:oc4j_6        |   25677 | Alive    | xxxxxxxxxx |   970368 |   4:35:03 | jms:12604,ajp:12506,rmis:12704,rmi:12404
OC4JGroup:7_group                | OC4J:oc4j_7        |   25676 | Alive    | xxxxxxxxxx |   855296 |   4:35:03 | jms:12603,ajp:12505,rmis:12703,rmi:12403
OC4JGroup:8_group                | OC4J:oc4j_8        |   25675 | Alive    | xxxxxxxxxx |   280576 |   4:35:03 | jms:12602,ajp:12504,rmis:12702,rmi:12402
OC4JGroup:default_group          | OC4J:home          |   25674 | Alive    | xxxxxxxxxx |   291744 |   4:35:03 | jms:12601,ajp:12503,rmis:12701,rmi:12401
ASG                              | ASG                |     N/A | Down     |        N/A |      N/A |       N/A | N/A
HTTP_Server                      | HTTP_Server        |   25673 | Alive    | xxxxxxxxxx |   659376 |   4:34:46 | http4:7783,https1:4443,http3:7203,http2:7782,http1:7781

In EnvB (Test2) we try to start another OC4J instance were the opmn defined port ranges contain as well the ports 12401 12501 12601 12701

<port id="default-web-site" range="12501-12600" protocol="ajp"/>
<port id="rmi" range="12401-12500"/>
<port id="rmis" range="12701-12800"/>
<port id="jms" range="12601-12700"/>

Testing Results

1) In the OC4J startup, OC4J log files generated in opmn/logs directory show:

--------
13/04/29 15:53:41 Start process
--------
Apr 29, 2013 3:53:43 PM com.evermind.server.XMLDataSourcesConfig parseRootNode
INFO: Legacy datasource detected...attempting to convert to new syntax.
13/04/29 15:53:44 *** (SEVERE) jms.xml: port 12,601 already in use.
13/04/29 15:53:44 *** (SEVERE) Failed to set the internal configuration of the OC4J JMS Server with: XMLJMSServerConfig[file:/oracle/products/bpel_tst/j2ee/Test/config/jms.xml]
13/04/29 15:53:44 Error starting ORMI server <servername> port 12,401: Address already in use

 

2) Running truss -fae on the opmn child process one can see sequence of tests of an OPMN process testing ports 12401 12501 12601 12701 (as an example) 

a) Port 12401 is identified as a free port  

28527/14:    52.9877    bind(15, 0xFB88BB08, 16, SOV_SOCKBSD)        = 0
28527/14:        AF_INET  name = 10.46.1.116  port = 12401

28527/14:    52.9881    bind(15, 0xFB88BB08, 16, SOV_SOCKBSD)        = 0
28527/14:        AF_INET  name = 127.0.0.1  port = 12401

28527/14:    52.9885    bind(15, 0xFB88B978, 16, SOV_SOCKBSD)        = 0
28527/14:        AF_INET  name = 0.0.0.0  port = 12401      


b) Port 12501 is identified as in use, OPMN tries for other ports until it identifies a free one.

28527/14:    52.9848    bind(15, 0xFB88BB08, 16, SOV_SOCKBSD)        = 0
28527/14:        AF_INET  name = 10.46.1.116  port = 12501

28527/14:    52.9855    bind(15, 0xFB88BB08, 16, SOV_SOCKBSD)        = 0
28527/14:        AF_INET  name = 127.0.0.1  port = 12501

28527/14:    52.9861    bind(15, 0xFB88B978, 16, SOV_SOCKBSD)        Err#125 EADDRINUSE
28527/14:        AF_INET  name = 0.0.0.0  port = 12501


c)  After OPMN detected free ports it prepares the execve command:

29409/14:    53.0079    execve("/oracle/products/bpel_tst/jdk/bin/java", 0x003D2900, 0x00355B88)  argc = 26
29409/1:     argv: /oracle/products/bpel_tst/jdk/bin/java -server
...
29409/1:      -config
29409/1:      /oracle/products/bpel_tst/j2ee/Test2/config/server.xml
29409/1:      -properties -ports
29409/1:      default-web-site:ajp:12503,rmi:12401,rmis:12701,jms:12601


d) Later on these ports are found to be "in use" by the previous JVM other jvm's in envA (Test1) and resulted in OC4J startup failure error "Address already in use" on ports that were "confirmed" by OPMN to be free (eg. Port 12401)

29409/75:    58.5942    write(26, 0xAC88D0B8, 41)            = 41
29409/75:       1 3 / 0 6 / 2 0   1 3 : 1 6 : 0 4 . 6 7 2   1 0 . 1 . 3 . 5 . 0
29409/75:         S t a r t e d\n
29409/75:    58.5946    so_socket(PF_INET6, SOCK_STREAM, IPPROTO_IP, 0, SOV_DEFAULT) = 29
29409/75:    58.5947    setsockopt(29, SOL_SOCKET, SO_REUSEADDR, 0xAC88F3AC, 4, SOV_DEFAULT) = 0
29409/75:    58.5948    getsockopt(29, SOL_SOCKET, SO_REUSEADDR, 0xAC88F420, 0xAC88F424, SOV_DEFAULT) = 0
29409/75:    58.5949    bind(29, 0xAC88F490, 32, SOV_SOCKBSD)        Err#125 EADDRINUSE
29409/75:        AF_INET6  name = ::  port = 12401
29409/75:        scope id = 0  source id = 0x0
29409/75:        flow class = 0x00  flow label = 0x00000
29409/75:    58.5951    fcntl(30, F_DUP2FD, 0x0000001D)            = 29

 

The same error happens on other ports such as 12601 and 12701, hence the JVM is unable to start correctly.

Changes

That problem was found after a new system was configured to consolidate multiple Application Server installs previously running in different environments. 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.