Application Processes Restarting or Not Working Correctly and the Database Will Not Shut Down (Doc ID 1180404.1)

Last updated on JULY 27, 2017

Applies to:

Oracle Communications Network Charging and Control - Version 4.3.0 and later
Information in this document applies to any platform.

Symptoms

On all versions of Oracle Communications Network Charging and Control (OCNCC or NCC), all the NCC applications can become unresponsive and/or crash constantly.  Further specific symptoms are:

A. slee_acs reports database connection errors in /var/adm/messages and/or its log file, /IN/service_packages/ACS/tmp/slee_acs.log:

slee_acs: [ID 675952 user.error] slee_acs(27433) ERROR: Failed to connect to oracle as user / (password hidden)
slee_acs: [ID 675952 user.error] slee_acs(27433) ERROR: acsChassisSetup: Database connect failed

Other processes may also report database connection errors in the syslog or their own log files. These are a couple of examples:

BeClient: [ID 675952 user.error] BeClient(27437) ERROR: Error loading beLocationPlugin: Unable to connect to db
dapIF: [ID 848595 user.crit] dapIF(27440) CRITICAL: Cannot connect to Oracle as '/'. - this process is now aborting to produce a core file for analysis

 

B. Some application processes may not start or could constantly restart. This can be seen using slee-ctrl or pslist. Here is an example using slee-ctrl:

$ slee-ctrl status
SLEE Control: v,1.0.10: script v,1.19: functions v,1.46: pslist v,1.112

[acs_oper] slee-ctrl> status
------------------------ Sun Nov 7 23:07:36 GMT 2010 --------------------------
C APP USER PID PPID STIME COMMAND
1 SLEE acs_oper 27434 8638 23:07:19 /IN/service_packages/SLEE/bin/timerIF
1 SLEE acs_oper 27435 8638 23:07:19 N/service_packages/ACS/bin/acsStatsLocalSLEE
1 SLEE acs_oper 27436 8638 23:07:19 /IN/service_packages/SLEE/bin/replicationIF
1 SLEE acs_oper 27438 8638 23:07:19 ervice_packages/DCA/bin/diameterControlAgent
1 SLEE acs_oper 27439 8638 23:07:19 IN/service_packages/DCD/bin/diameterBeClient
1 SLEE acs_oper 27441 8638 23:07:19 IN/service_packages/LCA/bin/ldapControlAgent
1 SLEE acs_oper 27442 8638 23:07:19 N/service_packages/SLEE/bin/xmlTcapInterface
1 SLEE acs_oper 27443 8638 23:07:19 /IN/service_packages/IS41/bin/suaCdmaGateway
1 SLEE acs_oper 27444 8638 23:07:19 /IN/service_packages/SLEE/bin//watchdog
0 SLEE acs_oper process not found: BeClient
0 SLEE acs_oper process not found: dapIF
0 SLEE acs_oper process not found: slee_acs
total processes found = 9 [ 12 expected, 3 not found ]
================================= run-level 3 ==================================

Subsequent checks may show that one or more process PID numbers are changing.


C. The database is not running. The following command does not show any running database processes owned by the oracle user or fewer than is normal.

$ ps -ef | grep ora_

 

If the database was running correctly then there would be 13 or more processes like this:
oracle 21394 8638 0 21:34:44 ? 0:02 ora_cjq0_SCP
oracle 14627 8638 0 01:55:39 ? 0:37 ora_pmon_SCP
oracle 14825 8638 0 01:55:41 ? 0:00 ora_reco_SCP
oracle 15193 8638 0 01:55:50 ? 0:01 ora_qmnc_SCP
oracle 14761 8638 0 01:55:40 ? 0:32 ora_ckpt_SCP
oracle 14685 8638 0 01:55:40 ? 0:03 ora_psp0_SCP
oracle 14712 8638 0 01:55:40 ? 0:03 ora_mman_SCP
oracle 14744 8638 0 01:55:40 ? 0:05 ora_lgwr_SCP
oracle 15677 8638 0 01:56:00 ? 0:00 ora_q001_SCP
oracle 15669 8638 0 01:56:00 ? 0:00 ora_q000_SCP
oracle 14843 8638 0 01:55:41 ? 0:07 ora_mmon_SCP
oracle 14861 8638 0 01:55:41 ? 1:02 ora_mmnl_SCP
oracle 14729 8638 0 01:55:40 ? 0:07 ora_dbw0_SCP
oracle 14785 8638 0 01:55:40 ? 0:05 ora_smon_SCP

And the ipcs command shows that there are still database shared resources:

$ ipcs
IPC status from <running system> as of Tuesday August 17 19:28:56 GMT 2010
T ID KEY MODE OWNER GROUP
Message Queues:
Shared Memory:
m 2560 0x9383b67c --rw-r----- oracle dba
m 6145 0x1006110 --rw-rw-rw- acs_oper esg
m 2 0x41f76 --rw-rw-rw- smf_oper esg
m 1540 0x455bae --rw-rw-rw- smf_oper esg
Semaphores:
s 720896 0x830d48bc --ra-r----- oracle dba
s 8 0x642f22 --ra-ra-ra- smf_oper esg
s 9 0x1472 --ra-ra-ra- smf_oper esg
s 10 0x6707f8 --ra-ra-ra- smf_oper esg

 

This assumes a single instance which should be the case for all UAS/SCP/SLC machines.

Changes

Maintenance or patch installation that involves a database restart.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms