Exadata getting ORA-600 [Userthread::Run_1] and ORA-7445 [_intel_new_memcpy] errors (Doc ID 1084729.1)

Last updated on OCTOBER 28, 2010

Applies to:

Oracle Exadata Storage Server Software - Version: to - Release: 11.2 to 11.2
Information in this document applies to any platform.


Exadata software version

One or more Exadata cells (storage nodes) get ORA-600 [UserThread::run_1], [18NN] and ORA-7445 [_intel_new_memcpy] errors.

This may cause lost redundancy, multiple errors in ASM and database alert logs and database hanging.

To recover from this condition all database and ASM instances have to be stopped and affected cells rebooted.

On the Exadata cell there would be errors like this in the cell alert log /opt/oracle/cell11.<cell name>/trace/alert.log:

Errors in file /opt/oracle/cell11.<cell name>/trace/rstrc_nnnn_m.trc (incident=iii):
RS-7445 [Serv CELLSRV absent] [It will be restarted] [] [] []
Incident details in: /opt/oracle/cell11.<cell name>/incident/incdir_iii/rstrc_nnnn_m_ikkk.trc

In the cell RS trace/incident trace file diag/asm/cell/<cell name>/incident/incdir_iii/rstrc_nnnn_m_ikkk.trc:

ORACLE_HOME = /opt/oracle/cell11.
System name: Linux
Node name: <fully qualified cell name>
Release: 2.6.18-
Version: #1 SMP <date>
Machine: x86_64
CELL SW Version: OSS_11.

RS-7445 [Serv CELLSRV absent] [It will be restarted] [] [] []
========= Dump for incident iii (RS 7445) ========
Starting a Diag Context default dump (level=3)

----- Call Stack Trace -----
ossrsutl_dump_incid (ent) <- ossrsutl_monitor_sr (vc) <- ossrsutl_monitor_sr (vc_prc) <- sossrs_prc_start <- ossrsutl_monitor_mo (npr_thd) <- start_thread

Cell alert history (cellcli "list alerthistory") would show messages like these:

NN <date and time> critical "ORA-00600: internal error code, arguments: [UserThread::run_1], [1841], []"
NN+1 <date and time> critical "ORA-07445: exception encountered: core dump [__intel_new_memcpy()] [11] [0x000000000] [] ORA-00600: internal error code, arguments: [UserThread::run_1], [1841], []"
NN+2 <date and time> critical "RS-7445 [Service CELLSRV is hung] [It will be restarted] []"


Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms