Status HA Changes to ENDANGERED And Never Goes to NODE-SAFE (Doc ID 2295146.1)

Last updated on FEBRUARY 03, 2019

Applies to:

Symptoms

On : 11.3.0.2.0 version, High Availability

After a rolling upgrade, the distribution of the partition data never ended, so the StatusHA never returned to NODE-SAFE status.

There was an exception that happened in ecs1 when the rollingUpgrade was running:

ERROR
-----------------------
2017-07-05 11:30:39.477 CST ERROR - - - - ProcessorFrameworkMessageBundle-17000: Unexpected exception: (Wrapped: Request interrupted while waiting for response from [Member(Id=28, Timestamp=2017-07-04 13:25:32.357, Address=##.###.###.##:20090, MachineId=21280, Location=site:xxx,machine:xxx01,process:11581,member:ecs1, Role=OracleCommunicationBrmChargingServerChargingLauncher)]) null
(Wrapped: Request interrupted while waiting for response from [Member(Id=28, Timestamp=2017-07-04 13:25:32.357, Address=##.###.###.##:20090, MachineId=21280, Location=site:xxx,machine:xxx01,process:11581,member:ecs1, Role=OracleCommunicationBrmChargingServerChargingLauncher)]) java.lang.InterruptedException
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.processPollInterrupt(Grid.CDB:36)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:39)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:1)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.invoke(PartitionedCache.CDB:28)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.invoke(PartitionedCache.CDB:36)
  at com.tangosol.coherence.component.util.SafeNamedCache.invoke$Router(SafeNamedCache.CDB:1)
  at com.tangosol.coherence.component.util.SafeNamedCache.invoke(SafeNamedCache.CDB:5)
  at oracle.communication.brm.charging.util.coherence.internal.CoherenceTemplateImpl.invoke(CoherenceTemplateImpl.java:380)
  at oracle.communication.brm.charging.processor.framework.internal.coherence.CoherenceUsageProcessorInvokerActivity.process(CoherenceUsageProcessorInvokerActivity.java:117)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.orchestration.framework.internal.ListActivityImpl.process(ListActivityImpl.java:50)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.orchestration.framework.internal.ConditionActivityImpl.process(ConditionActivityImpl.java:65)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.orchestration.framework.internal.RetryActivityImpl.process(RetryActivityImpl.java:61)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.orchestration.framework.internal.RetryActivityImpl.process(RetryActivityImpl.java:61)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.orchestration.framework.internal.HandlerActivityImpl.process(HandlerActivityImpl.java:69)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.orchestration.framework.internal.SwitchActivityImpl.process(SwitchActivityImpl.java:89)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.orchestration.framework.internal.ListActivityImpl.process(ListActivityImpl.java:50)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.brs.internal.coherence.CoherenceBatchRequestInvocable.dispatch(CoherenceBatchRequestInvocable.java:187)
  at oracle.communication.brm.charging.brs.internal.coherence.CoherenceBatchRequestInvocable.run(CoherenceBatchRequestInvocable.java:121)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.InvocationService.onInvocationRequest(InvocationService.CDB:16)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.InvocationService$InvocationRequest.run(InvocationService.CDB:1)
  at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
  at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
  at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:68)
  at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:45)
  at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException
  at com.oracle.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:64)
  at com.oracle.common.base.SingleWaiterCooperativeNotifier.await(SingleWaiterCooperativeNotifier.java:49)
  at com.tangosol.coherence.component.net.Poll.waitCompletion(Poll.CDB:6)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:34)
  ... 31 more
2017-07-05 11:30:39.478 CST ERROR - - - - ProcessorFrameworkMessageBundle-17000: Unexpected exception: Uninitialized service: SafeDistributedCacheService: STOPPED
java.lang.IllegalStateException: Uninitialized service: SafeDistributedCacheService: STOPPED
  at com.tangosol.coherence.component.util.SafeService.getResourceRegistry(SafeService.CDB:6)
  at com.tangosol.application.ContainerHelper.getSwitchServiceContext(ContainerHelper.java:103)
  at com.tangosol.net.ExtensibleConfigurableCacheFactory.checkContextPermission(ExtensibleConfigurableCacheFactory.java:812)
  at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureCacheInternal(ExtensibleConfigurableCacheFactory.java:249)
  at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureTypedCache(ExtensibleConfigurableCacheFactory.java:206)
  at com.tangosol.net.ConfigurableCacheFactory.ensureCache(ConfigurableCacheFactory.java:57)
  at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:247)
  at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:217)
  at oracle.communication.brm.charging.util.coherence.internal.CoherenceTemplateImpl.getCache(CoherenceTemplateImpl.java:468)
  at oracle.communication.brm.charging.util.coherence.internal.CoherenceTemplateImpl.get(CoherenceTemplateImpl.java:178)
  at oracle.communication.brm.charging.config.creditprofile.internal.coherence.CoherenceCreditProfileRepository.find(CoherenceCreditProfileRepository.java:71)
  at oracle.communication.brm.charging.config.creditprofile.internal.CreditProfileReference.resolve(CreditProfileReference.java:53)
  at oracle.communication.brm.charging.config.creditprofile.internal.CreditProfileReference.resolve(CreditProfileReference.java:25)
  at oracle.communication.brm.charging.util.transaction.AbstractLazyReference.get(AbstractLazyReference.java:84)
  at oracle.communication.brm.charging.config.creditprofile.internal.CreditProfileReference.toString(CreditProfileReference.java:58)
  at java.lang.String.valueOf(String.java:2994)
  at java.lang.StringBuilder.append(StringBuilder.java:131)
  at oracle.communication.brm.charging.balance.internal.BalanceItemSpecImpl.toString(BalanceItemSpecImpl.java:186)
  at java.lang.String.valueOf(String.java:2994)
  at java.lang.StringBuilder.append(StringBuilder.java:131)
  at java.util.AbstractMap.toString(AbstractMap.java:559)
  at java.lang.String.valueOf(String.java:2994)
  at java.lang.StringBuilder.append(StringBuilder.java:131)
  at oracle.communication.brm.charging.balance.internal.BalanceImpl.toString(BalanceImpl.java:4280)
  at java.lang.String.valueOf(String.java:2994)
  at java.lang.StringBuilder.append(StringBuilder.java:131)
  at java.util.AbstractMap.toString(AbstractMap.java:559)
  at oracle.communication.brm.charging.util.misc.LazyHashMap.toString(LazyHashMap.java:194)
  at java.lang.String.valueOf(String.java:2994)
  at java.lang.StringBuilder.append(StringBuilder.java:131)
  at oracle.communication.brm.charging.customer.internal.CustomerImpl.toString(CustomerImpl.java:2502)
  at java.lang.String.valueOf(String.java:2994)
  at java.lang.StringBuilder.append(StringBuilder.java:131)
  at oracle.communication.brm.charging.orchestration.framework.internal.MessageImpl.toString(MessageImpl.java:97)
  at oracle.communication.brm.charging.processor.framework.internal.AbstractExceptionDetailThrottler.logException(AbstractExceptionDetailThrottler.java:144)
  at oracle.communication.brm.charging.processor.framework.internal.AbstractExceptionDetailThrottler.throttleExceptionLogging(AbstractExceptionDetailThrottler.java:128)
  at oracle.communication.brm.charging.processor.usage.internal.UsageProcessorErrorHandler.process(UsageProcessorErrorHandler.java:67)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.orchestration.framework.internal.ListActivityImpl.process(ListActivityImpl.java:50)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.orchestration.framework.internal.HandlerActivityImpl.process(HandlerActivityImpl.java:75)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.orchestration.framework.internal.ConditionActivityImpl.process(ConditionActivityImpl.java:65)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.orchestration.framework.internal.ListActivityImpl.process(ListActivityImpl.java:50)
  at oracle.communication.brm.charging.orchestration.framework.internal.ManagedActivityImpl.process(ManagedActivityImpl.java:76)
  at oracle.communication.brm.charging.processor.framework.AbstractSerializableActivity.process(AbstractSerializableActivity.java:63)
  at oracle.communication.brm.charging.processor.framework.internal.coherence.CoherenceUsageProcessor.process(CoherenceUsageProcessor.java:186)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.invoke(PartitionedCache.CDB:10)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onInvokeRequest(PartitionedCache.CDB:60)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$InvokeRequest.run(PartitionedCache.CDB:1)
  at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
  at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
  at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService$DaemonPool$WrapperTask.run(PartitionedService.CDB:1)
  at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:68)
  at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:45)
  at java.lang.Thread.run(Thread.java:745)

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.