UCP 11.2.0.4 Seems To Hang Or Stop And Return "java.lang.NullPointerException" Errors After A Second Dataguard Switchover (Doc ID 2068441.1)

Last updated on AUGUST 01, 2023

Applies to:

Symptoms

On : 11.2.0.4.0 version, Application Continuity.

In a production system on a 2-site environment with 2 Exadata RAC nodes each (11.2.0.4.15) replicated with Active Data Guard, when the database was switched-over without stopping applications that use UCP 11.2.0.4 (with patches 18480782 and 19151967 applied) and FCF enabled, the next "java.lang.NullPointerException" error is reported:

ERROR
-----------------------
15:46:00,864 ERROR [stderr] (pool-5-thread-1) org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.lang.NullPointerException
15:46:00,865 ERROR [stderr] (pool-5-thread-1) at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:241)
15:46:00,865 ERROR [stderr] (pool-5-thread-1) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
15:46:00,865 ERROR [stderr] (pool-5-thread-1) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)

...

15:46:00,866 ERROR [stderr] (pool-5-thread-1) at org.apache.commons.monitoring.reporting.AbstractPeriodicLogger$1.run(AbstractPeriodicLogger.java:87)
15:46:00,866 ERROR [stderr] (pool-5-thread-1) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
15:46:00,866 ERROR [stderr] (pool-5-thread-1) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
15:46:00,866 ERROR [stderr] (pool-5-thread-1) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
15:46:00,867 ERROR [stderr] (pool-5-thread-1) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
15:46:00,867 ERROR [stderr] (pool-5-thread-1) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
15:46:00,867 ERROR [stderr] (pool-5-thread-1) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
15:46:00,867 ERROR [stderr] (pool-5-thread-1) at java.lang.Thread.run(Thread.java:744)
15:46:00,868 ERROR [stderr] (pool-5-thread-1) Caused by: java.lang.NullPointerException
15:46:00,868 ERROR [stderr] (pool-5-thread-1) at oracle.ucp.jdbc.oracle.OracleJDBCConnectionPool.createOnePooledConnection(OracleJDBCConnectionPool.java:1260)
15:46:00,868 ERROR [stderr] (pool-5-thread-1) at oracle.ucp.common.UniversalConnectionPoolImpl.borrowConnectionWithoutCountingRequests(UniversalConnectionPoolImpl.java:339)
15:46:00,868 ERROR [stderr] (pool-5-thread-1) at oracle.ucp.common.UniversalConnectionPoolImpl.borrowConnectionAndValidate(UniversalConnectionPoolImpl.java:168)
15:46:00,868 ERROR [stderr] (pool-5-thread-1) at oracle.ucp.common.UniversalConnectionPoolImpl.borrowConnection(UniversalConnectionPoolImpl.java:143)
15:46:00,868 ERROR [stderr] (pool-5-thread-1) at oracle.ucp.jdbc.JDBCConnectionPool.borrowConnection(JDBCConnectionPool.java:157)
15:46:00,869 ERROR [stderr] (pool-5-thread-1) at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:931)
15:46:00,869 ERROR [stderr] (pool-5-thread-1) at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:873)
15:46:00,869 ERROR [stderr] (pool-5-thread-1) at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:863)
15:46:00,869 ERROR [stderr] (pool-5-thread-1) at org.springframework.jdbc.datasource.DelegatingDataSource.getConnection(DelegatingDataSource.java:84)
...

15:46:00,870 ERROR [stderr] (pool-5-thread-1) at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
15:46:00,870 ERROR [stderr] (pool-5-thread-1) ... 12 more

STEPS
-------

15:21:53 : a good startup of UCP followed by successfull borrowed connections
15:39:38 : reception and good treatment of the first fail-over down event of the switch-over from db1 to db2
15:39:45 - 15:40:17 : some expected connection errors as switch-over requires seconds of downtime
15:40:17 : reception and good treatment of the first fail-over up event of the switch-over from db1 to db2 followed by successfull borrowed connections
15:40:38 : a good treatment of the first runtime load balancing event since the pool was launched
15:44:01 : reception and good treatment of the first fail-over down event of the switch-over back from db2 to db1
15:44:03 - 15:44:34 : some expected connection errors as switch-over back requires seconds of downtime
15:44:34 : reception and good treatment of the first fail-over up event of the switch-over back from db2 to db1
But from 15:44:34,690, all threads seem to hang after checking the smoothed% (UCP never did it before) of the instances of db2 ( down events were successfully treated just before) because after that there is any other log message for each thread. And the application did not work anymore.
15:46:00/15:51:00/15:56:00 : connection error java.lang.NullPointerException on OracleJDBCConnectionPool.createOnePooledConnection

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.