smsAlarmRelay Cores Are Generated In A High Frequency (Doc ID 1307792.1)

Last updated on JULY 16, 2013

Applies to:

Oracle Communications Network Charging and Control - Version 3.1.0 and later
Information in this document applies to any platform.

Symptoms

The smsAlarmRelay application often dumps core files causing filesystem /var/crash to fill up.

root@usms01# cd /var/crash
root@usms01# ls -ltr |tail
-rw------- 1 root root 16573919 Mar 8 08:35 core-usms01-2263-smsAlarmRelay
-rw------- 1 root root 16573919 Mar 8 08:38 core-usms01-2604-smsAlarmRelay
-rw------- 1 root root 16573919 Mar 8 08:44 core-usms01-2930-smsAlarmRelay
-rw------- 1 root root 16573919 Mar 8 08:49 core-usms01-3360-smsAlarmRelay
-rw------- 1 root root 16573919 Mar 8 08:53 core-usms01-3733-smsAlarmRelay
root@usms01#


Depending on the way the system is being monitored, frequent crashes of the process can also be noticed based on:

Following messages are also written to the /var/adm/messages file very often:

Mar 8 03:10:19 usms01 smsStatsThreshold: [ID 953149 user.warning] smsStatsThreshold(20901) WARNING: SQL statement: select APPLICATION_ID ,STATISTIC_ID ,NODE_NAME ,nvl(DETAIL,chr(0)) , timestamp ,VALUE ,ID from SMF_STATISTICS where ID>:b0 order by ID
Mar 8 03:10:19 usms01 smsStatsThreshold: [ID 953149 user.warning] smsStatsThreshold(20901) WARNING: An Oracle warning ocurred. This indicates a program bug.
Mar 8 03:10:19 usms01 smsStatsThreshold: [ID 953149 user.warning] smsStatsThreshold(20901) WARNING: SQL error code 0 at smsStatsRule.pc:644
Mar 8 03:10:19 usms01 smsStatsThreshold: [ID 953149 user.warning] smsStatsThreshold(20901) WARNING: SQL warning: Data truncated.
...........
...........
Mar 8 03:27:22 usms01 smsAlarmRelay: [ID 675952 user.error] smsAlarmRelay(9230) ERROR: smsAlarmRelay: for statement: SELECT m.ID, m.MACHINE, m.TIME, m.CPU, m.NAME, m.SUBSYSTEM, m.SEVERITY, m.DESCRIPTION, m.COUNT, m.STATUS, m.OPCOMMENT, m.CHANGE_SEQUENCE, m.ALARM_TYPE_ID, m.MANAGED_OBJECT_INSTANCE, d.EVENT_TYPE, d.PROBABLE_CAUSE, d.SEVERITY, d.SPECIFIC_PROBLEM, d.RECOMMENDED_ACTION, d.ADDITIONAL_TEXT, d.PRESENT_TO_AR, d.AUTOCLEAR_PERIOD FROM SMF_ALARM_MESSAGE m, SMF_ALARM_DEFN d WHERE d.PRESENT_TO_AR <> 0 AND m.ALARM_TYPE_ID = d.ALARM_TYPE_ID AND m.ALARM_TYPE_ID <> 0 AND m.CHANGE_SEQUENCE in (158584421,158584453,158584454,158584455,158584456,158584483,158584484,158584485,158584486,158584590,158584591,158584592,158584593,158584594,158584595,158584596,158584597,158584598,158584599,158584600,158584601,158584602,158584603,158584604,158584605,158584612,158584613,158584614,158584615,158584616,158584617,158584618,158584619,158584620,158584621,158584622,158584623,158584624,158584625,158584626,158584627,158584628
Mar 8 03:27:22 usms01 smsAlarmRelay: [ID 675952 user.error] smsAlarmRelay(9230) ERROR: smsAlarmRelay: for statement: SELECT m.ID, m.MACHINE, m.TIME, m.CPU, m.NAME, m.SUBSYSTEM, m.SEVERITY, m.DESCRIPTION, m.COUNT, m.STATUS, m.OPCOMMENT, m.CHANGE_SEQUENCE, m.ALARM_TYPE_ID, m.MANAGED_OBJECT_INSTANCE, d.EVENT_TYPE, d.PROBABLE_CAUSE, d.SEVERITY, d.SPECIFIC_PROBLEM, d.RECOMMENDED_ACTION, d.ADDITIONAL_TEXT, d.PRESENT_TO_AR, d.AUTOCLEAR_PERIOD FROM SMF_ALARM_MESSAGE m, SMF_ALARM_DEFN d WHERE d.PRESENT_TO_AR <> 0 AND m.ALARM_TYPE_ID = d.ALARM_TYPE_ID AND m.ALARM_TYPE_ID <> 0 AND m.CHANGE_SEQUENCE in (158584421,158584453,158584454,158584455,158584456,158584483,158584484,158584485,158584486,158584590,158584591,158584592,158584593,158584594,158584595,158584596,158584597,158584598,158584599,158584600,158584601,158584602,158584603,158584604,158584605,158584612,158584613,158584614,158584615,158584616,158584617,158584618,158584619,158584620,158584621,158584622,158584623,158584624,158584625,158584626,158584627,158584628
Mar 8 03:27:24 usms01 genunix: [ID 603404 kern.notice] NOTICE: core_log: smsAlarmRelay[9230] core dumped: /var/crash/core-usms01-9230-smsAlarmRelay
Mar 8 03:27:24 usms01 genunix: [ID 603404 kern.notice] NOTICE: core_log: smsAlarmRelay[9230] core dumped: /var/crash/core-usms01-9230-smsAlarmRelay
Mar 8 03:27:25 usms01 smsAlarmRelay: [ID 167701 user.notice] smsAlarmRelay(14299) NOTICE main=0 smsAlarmRelay.c@541: Startup successful
Mar 8 03:28:35 usms01 smsAlarmRelay: [ID 675952 user.error] smsAlarmRelay(14299) ERROR: smsAlarmRelay: SQL Error: (-1405)
Mar 8 03:28:35 usms01 smsAlarmRelay: [ID 675952 user.error] smsAlarmRelay(14299) ERROR: smsAlarmRelay: for statement: SELECT m.ID, m.MACHINE, m.TIME, m.CPU, m.NAME, m.SUBSYSTEM, m.SEVERITY, m.DESCRIPTION, m.COUNT, m.STATUS, m.OPCOMMENT, m.CHANGE_SEQUENCE, m.ALARM_TYPE_ID, m.MANAGED_OBJECT_INSTANCE, d.EVENT_TYPE, d.PROBABLE_CAUSE, d.SEVERITY, d.SPECIFIC_PROBLEM, d.RECOMMENDED_ACTION, d.ADDITIONAL_TEXT, d.PRESENT_TO_AR, d.AUTOCLEAR_PERIOD FROM SMF_ALARM_MESSAGE m, SMF_ALARM_DEFN d WHERE d.PRESENT_TO_AR <> 0 AND m.ALARM_TYPE_ID = d.ALARM_TYPE_ID AND m.ALARM_TYPE_ID <> 0 AND m.CHANGE_SEQUENCE in (158584900,158584901,158584902,158584903,158584904,158584905,158584906,158584907,158584908,158584909,158584910) ORDER BY time
Mar 8 03:28:35 usms01 smsAlarmRelay: [ID 675952 user.error] smsAlarmRelay(14299) ERROR: smsAlarmRelay: for statement: SELECT m.ID, m.MACHINE, m.TIME, m.CPU, m.NAME, m.SUBSYSTEM, m.SEVERITY, m.DESCRIPTION, m.COUNT, m.STATUS, m.OPCOMMENT, m.CHANGE_SEQUENCE, m.ALARM_TYPE_ID, m.MANAGED_OBJECT_INSTANCE, d.EVENT_TYPE, d.PROBABLE_CAUSE, d.SEVERITY, d.SPECIFIC_PROBLEM, d.RECOMMENDED_ACTION, d.ADDITIONAL_TEXT, d.PRESENT_TO_AR, d.AUTOCLEAR_PERIOD FROM SMF_ALARM_MESSAGE m, SMF_ALARM_DEFN d WHERE d.PRESENT_TO_AR <> 0 AND m.ALARM_TYPE_ID = d.ALARM_TYPE_ID AND m.ALARM_TYPE_ID <> 0 AND m.CHANGE_SEQUENCE in (158584900,158584901,158584902,158584903,158584904,158584905,158584906,158584907,158584908,158584909,158584910) ORDER BY time
Mar 8 03:28:36 usms01 smsAlarmRelay: [ID 675952 user.error] smsAlarmRelay(14299) ERROR: smsAlarmRelay: SQL Error: (-1405)
Mar 8 03:28:36 usms01 smsAlarmRelay: [ID 675952 user.error] smsAlarmRelay(14299) ERROR: smsAlarmRelay: SQL Error: (-1405)
Mar 8 03:28:36 usms01 smsAlarmRelay: [ID 675952 user.error] smsAlarmRelay(14299) ERROR: smsAlarmRelay: for statement: SELECT m.ID, m.MACHINE, m.TIME, m.CPU, m.NAME, m.SUBSYSTEM, m.SEVERITY, m.DESCRIPTION, m.COUNT, m.STATUS, m.OPCOMMENT, m.CHANGE_SEQUENCE, m.ALARM_TYPE_ID, m.MANAGED_OBJECT_INSTANCE, d.EVENT_TYPE, d.PROBABLE_CAUSE, d.SEVERITY, d.SPECIFIC_PROBLEM, d.RECOMMENDED_ACTION, d.ADDITIONAL_TEXT, d.PRESENT_TO_AR, d.AUTOCLEAR_PERIOD FROM SMF_ALARM_MESSAGE m, SMF_ALARM_DEFN d WHERE d.PRESENT_TO_AR <> 0 AND m.ALARM_TYPE_ID = d.ALARM_TYPE_ID AND m.ALARM_TYPE_ID <> 0 AND m.CHANGE_SEQUENCE in (158584909,158584910) ORDER BY time
Mar 8 03:28:36 usms01 smsAlarmRelay: [ID 675952 user.error] smsAlarmRelay(14299) ERROR: smsAlarmRelay: for statement: SELECT m.ID, m.MACHINE, m.TIME, m.CPU, m.NAME, m.SUBSYSTEM, m.SEVERITY, m.DESCRIPTION, m.COUNT, m.STATUS, m.OPCOMMENT, m.CHANGE_SEQUENCE, m.ALARM_TYPE_ID, m.MANAGED_OBJECT_INSTANCE, d.EVENT_TYPE, d.PROBABLE_CAUSE, d.SEVERITY, d.SPECIFIC_PROBLEM, d.RECOMMENDED_ACTION, d.ADDITIONAL_TEXT, d.PRESENT_TO_AR, d.AUTOCLEAR_PERIOD FROM SMF_ALARM_MESSAGE m, SMF_ALARM_DEFN d WHERE d.PRESENT_TO_AR <> 0 AND m.ALARM_TYPE_ID = d.ALARM_TYPE_ID AND m.ALARM_TYPE_ID <> 0 AND m.CHANGE_SEQUENCE in (158584909,158584910) ORDER BY time

Changes

None

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms