Multiple Gateways Using Cassandra for Oauth Crash After Startup
(Doc ID 2267624.1)
Last updated on FEBRUARY 06, 2024
Applies to:
Oracle API Gateway - Version 11.1.2.4.0 and laterInformation in this document applies to any platform.
Symptoms
Using Oracle API Gateway 11.1.2.4.0
The environment has a cluster with four Gateways (four is used in this example, the number of Gateways can vary), and each Gateway instance is configured to use a Cassandra database for Oauth.
On 2 nodes, when executing the command "./nodetool status kps" in this example, the fiollowing result is returned:
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving-- Address Load Tokens Owns (effective) Host ID Rack
DN <IP ADDRESS> 8.37 GB 1 100.0% <ID> rack1
UN <IP ADDRESS> 5.46 GB 1 100.0% <ID> rack1
UN <IP ADDRESS> 7.97 GB 1 100.0% <ID> rack1
DN <IP ADDRESS> 62.7 KB 1 100.0% <ID> rack1
The gateway does start showing the following error message
"Unable to start Gateway Server: Unable to load /api/management/start/group-2/instance-1 status 500",
The gateway trace files show the following:
ERROR [3660:000000000000000000000000] KPS: KPS Schema: Unexpected exception for get query:
KeyQuery: keys: [storeId], values: [oauth_authorizations]:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:42)
at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:100)
at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:88)
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:104)
. . .
Caused by: TimedOutException()
at org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7962)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:603)
at org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:587)
at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:95)
... 21 more
ERROR [3660:000000000000000000000000] Schema check failed for store: OAuth_Authorizations:
java.lang.RuntimeException: KPS: KPS Schema: Unexpected exception for get query: KeyQuery: keys: [storeId], values: [oauth_authorizations]
at com.vordel.kps.storeImpl.Utils.runtimeException(Utils.java:33)
at com.vordel.kps.storeImpl.AbstractTransaction.get(AbstractTransaction.java:113)
at com.vordel.kps.storeImpl.AbstractTransaction.get(AbstractTransaction.java:121)
at com.vordel.kps.admin.Schema.initSchema(Schema.java:195)
at com.vordel.kps.admin.Schema.<init>(Schema.java:68)
. . .
Caused by: me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:42)
at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:100)
at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:88)
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:104)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:253)
... 10 more
Caused by: TimedOutException()
at org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7962)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:603)
at org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:587)
at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:95)
... 21 more
ERROR [3a31:000000000000000000000000] error processing SLA alert:
java.lang.NullPointerException
at com.vordel.circuit.sla.SLARequirement$1.run(SLARequirement.java:203)
at com.vordel.circuit.sla.SLAStatisticsAnalyser.run(SLAStatisticsAnalyser.java:105)
A Core file is generated and the stack trace from this file is shown below:
#1 0x000000313ac33e05 in abort () from /lib64/libc.so.6
#2 0x00007f9d8da09a35 in os::abort(bool) () from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#3 0x00007f9d8db8a7c7 in VMError::report_and_die() () from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#4 0x00007f9d8da0e91f in JVM_handle_linux_signal () from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007f9d8d94aa4c in PhaseIdealLoop::build_loop_late_post(Node*) () from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#7 0x00007f9d8d94afcc in PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&) ()
from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#8 0x00007f9d8d951a40 in PhaseIdealLoop::build_and_optimize(bool, bool) () from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#9 0x00007f9d8d62d36a in Compile::Optimize() () from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#10 0x00007f9d8d62ecf9 in Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool) ()
from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#11 0x00007f9d8d599566 in C2Compiler::compile_method(ciEnv*, ciMethod*, int) ()
from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#12 0x00007f9d8d63671e in CompileBroker::invoke_compiler_on_method(CompileTask*) ()
from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#13 0x00007f9d8d6374a8 in CompileBroker::compiler_thread_loop() () from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#14 0x00007f9d8db4311f in JavaThread::thread_main_inner() () from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#15 0x00007f9d8db43225 in JavaThread::run() () from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#16 0x00007f9d8da0a938 in java_start(Thread*) () from <OAG HOME>/apigateway/platform/jre/lib/amd64/server/libjvm.so
#17 0x000000313b0079d1 in start_thread () from /lib64/libpthread.so.0
#18 0x000000313ace88fd in clone () from /lib64/libc.so.6
As well, a Java core file is created with the following output:
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f77138e9a4c, pid=18268, tid=140149208893184
#
# JRE version: Java(TM) SE Runtime Environment (7.0_85-b15) (build 1.7.0_85-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.85-b06 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x763a4c] PhaseIdealLoop::build_loop_late_post(Node*)+0x13c
#
# Core dump written. Default location: <OAG HOME>/apigateway/groups/group-2/instance-1/core
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
. . .
. . .
Stack: [0x00007f7707bfc000,0x00007f7707cfd000], sp=0x00007f7707cf7f80, free space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x763a4c] PhaseIdealLoop::build_loop_late_post(Node*)+0x13c
V [libjvm.so+0x763fcc] PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&)+0x13c
V [libjvm.so+0x76aa40] PhaseIdealLoop::build_and_optimize(bool, bool)+0x840
V [libjvm.so+0x44636a] Compile::Optimize()+0x4ca
V [libjvm.so+0x447cf9] Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool)+0x12f9
V [libjvm.so+0x3b2566] C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0x176
V [libjvm.so+0x44f71e] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x33e
V [libjvm.so+0x4504a8] CompileBroker::compiler_thread_loop()+0x3e8
V [libjvm.so+0x95c11f] JavaThread::thread_main_inner()+0xdf
V [libjvm.so+0x95c225] JavaThread::run()+0xf5
V [libjvm.so+0x823938] java_start(Thread*)+0x108
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |