Status HA Changes to ENDANGERED And Never Goes to NODE-SAFE
(Doc ID 2295146.1)
Last updated on MARCH 29, 2021
Applies to:
Oracle Communications BRM - Elastic Charging Engine - Version 11.3.0.2.0 and laterInformation in this document applies to any platform.
Symptoms
After a rolling upgrade, the distribution of the partition data never ended, so the StatusHA never returned to NODE-SAFE status.
There was an exception that happened in ecs1 when the rollingUpgrade was running:
ERROR
-----------------------
2017-07-05 11:30:39.477 CST ERROR - - - - ProcessorFrameworkMessageBundle-17000: Unexpected exception: (Wrapped: Request interrupted while waiting for response from [Member(Id=28, Timestamp=2017-07-04 13:25:32.357, Address=<HOSTNAME>:<PORT>, MachineId=21280, Location=<LOC>:11581,member:ecs1, Role=OracleCommunicationBrmChargingServerChargingLauncher)]) null
(Wrapped: Request interrupted while waiting for response from [Member(Id=28, Timestamp=2017-07-04 13:25:32.357, Address=<HOSTNAME>:<PORT>, MachineId=21280, Location=<LOC>:11581,member:ecs1, Role=OracleCommunicationBrmChargingServerChargingLauncher)]) java.lang.InterruptedException
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.processPollInterrupt(Grid.CDB:36)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:39)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.invoke(PartitionedCache.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.invoke(PartitionedCache.CDB:36)
at com.tangosol.coherence.component.util.SafeNamedCache.invoke$Router(SafeNamedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.invoke(SafeNamedCache.CDB:5)
at oracle.communication.brm.charging.util.coherence.internal.CoherenceTemplateImpl.invoke(CoherenceTemplateImpl.java:380)
at oracle.communication.brm.charging.processor.framework.internal.coherence.CoherenceUsageProcessorInvokerActivity.process(CoherenceUsageProcessorInvokerActivity.java:117)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.orchestration.framework.internal.ListActivityImpl.process(ListActivityImpl.java:50)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.orchestration.framework.internal.ConditionActivityImpl.process(ConditionActivityImpl.java:65)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.orchestration.framework.internal.RetryActivityImpl.process(RetryActivityImpl.java:61)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.orchestration.framework.internal.RetryActivityImpl.process(RetryActivityImpl.java:61)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.orchestration.framework.internal.HandlerActivityImpl.process(HandlerActivityImpl.java:69)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.orchestration.framework.internal.SwitchActivityImpl.process(SwitchActivityImpl.java:89)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.orchestration.framework.internal.ListActivityImpl.process(ListActivityImpl.java:50)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.brs.internal.coherence.CoherenceBatchRequestInvocable.dispatch(CoherenceBatchRequestInvocable.java:187)
at oracle.communication.brm.charging.brs.internal.coherence.CoherenceBatchRequestInvocable.run(CoherenceBatchRequestInvocable.java:121)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.InvocationService.onInvocationRequest(InvocationService.CDB:16)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.InvocationService$InvocationRequest.run(InvocationService.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:68)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:45)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException
at com.oracle.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:64)
at com.oracle.common.base.SingleWaiterCooperativeNotifier.await(SingleWaiterCooperativeNotifier.java:49)
at com.tangosol.coherence.component.net.Poll.waitCompletion(Poll.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:34)
... 31 more
2017-07-05 11:30:39.478 CST ERROR - - - - ProcessorFrameworkMessageBundle-17000: Unexpected exception: Uninitialized service: SafeDistributedCacheService: STOPPED
java.lang.IllegalStateException: Uninitialized service: SafeDistributedCacheService: STOPPED
at com.tangosol.coherence.component.util.SafeService.getResourceRegistry(SafeService.CDB:6)
at com.tangosol.application.ContainerHelper.getSwitchServiceContext(ContainerHelper.java:103)
at com.tangosol.net.ExtensibleConfigurableCacheFactory.checkContextPermission(ExtensibleConfigurableCacheFactory.java:812)
at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureCacheInternal(ExtensibleConfigurableCacheFactory.java:249)
at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureTypedCache(ExtensibleConfigurableCacheFactory.java:206)
at com.tangosol.net.ConfigurableCacheFactory.ensureCache(ConfigurableCacheFactory.java:57)
at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:247)
at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:217)
at oracle.communication.brm.charging.util.coherence.internal.CoherenceTemplateImpl.getCache(CoherenceTemplateImpl.java:468)
at oracle.communication.brm.charging.util.coherence.internal.CoherenceTemplateImpl.get(CoherenceTemplateImpl.java:178)
at oracle.communication.brm.charging.config.creditprofile.internal.coherence.CoherenceCreditProfileRepository.find(CoherenceCreditProfileRepository.java:71)
at oracle.communication.brm.charging.config.creditprofile.internal.CreditProfileReference.resolve(CreditProfileReference.java:53)
at oracle.communication.brm.charging.config.creditprofile.internal.CreditProfileReference.resolve(CreditProfileReference.java:25)
at oracle.communication.brm.charging.util.transaction.AbstractLazyReference.get(AbstractLazyReference.java:84)
at oracle.communication.brm.charging.config.creditprofile.internal.CreditProfileReference.toString(CreditProfileReference.java:58)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at oracle.communication.brm.charging.balance.internal.BalanceItemSpecImpl.toString(BalanceItemSpecImpl.java:186)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at java.util.AbstractMap.toString(AbstractMap.java:559)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at oracle.communication.brm.charging.balance.internal.BalanceImpl.toString(BalanceImpl.java:4280)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at java.util.AbstractMap.toString(AbstractMap.java:559)
at oracle.communication.brm.charging.util.misc.LazyHashMap.toString(LazyHashMap.java:194)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at oracle.communication.brm.charging.customer.internal.CustomerImpl.toString(CustomerImpl.java:2502)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at oracle.communication.brm.charging.orchestration.framework.internal.MessageImpl.toString(MessageImpl.java:97)
at oracle.communication.brm.charging.processor.framework.internal.AbstractExceptionDetailThrottler.logException(AbstractExceptionDetailThrottler.java:144)
at oracle.communication.brm.charging.processor.framework.internal.AbstractExceptionDetailThrottler.throttleExceptionLogging(AbstractExceptionDetailThrottler.java:128)
at oracle.communication.brm.charging.processor.usage.internal.UsageProcessorErrorHandler.process(UsageProcessorErrorHandler.java:67)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.orchestration.framework.internal.ListActivityImpl.process(ListActivityImpl.java:50)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.orchestration.framework.internal.HandlerActivityImpl.process(HandlerActivityImpl.java:75)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.orchestration.framework.internal.ConditionActivityImpl.process(ConditionActivityImpl.java:65)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.orchestration.framework.internal.ListActivityImpl.process(ListActivityImpl.java:50)
at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
at oracle.communication.brm.charging.processor.framework.AbstractSerializableActivity.process(AbstractSerializableActivity.java:63)
at oracle.communication.brm.charging.processor.framework.internal.coherence.CoherenceUsageProcessor.process(CoherenceUsageProcessor.java:186)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.invoke(PartitionedCache.CDB:10)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onInvokeRequest(PartitionedCache.CDB:60)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvokeRequest.run(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService$DaemonPool$WrapperTask.run(PartitionedService.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:68)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:45)
at java.lang.Thread.run(Thread.java:745)
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
Symptoms |
Changes |
Cause |
Solution |
References |