Exadata getting ORA-600 [Userthread::Run_1] and ORA-7445 [_intel_new_memcpy] errors (Doc ID 1084729.1)

Last updated on OCTOBER 28, 2010

Applies to:

Oracle Exadata Storage Server Software - Version: 11.2.1.2.1 to 11.2.1.2.1 - Release: 11.2 to 11.2
Information in this document applies to any platform.

Symptoms


Exadata software version 11.2.1.2.1.

One or more Exadata cells (storage nodes) get ORA-600 [UserThread::run_1], [18NN] and ORA-7445 [_intel_new_memcpy] errors.

This may cause lost redundancy, multiple errors in ASM and database alert logs and database hanging.

To recover from this condition all database and ASM instances have to be stopped and affected cells rebooted.

On the Exadata cell there would be errors like this in the cell alert log /opt/oracle/cell11.2.1.2.1_LINUX.X64_100131/log/diag/asm/cell/<cell name>/trace/alert.log:

Errors in file /opt/oracle/cell11.2.1.2.1_LINUX.X64_100131/log/diag/asm/cell/<cell name>/trace/rstrc_nnnn_m.trc (incident=iii):
RS-7445 [Serv CELLSRV absent] [It will be restarted] [] [] []
Incident details in: /opt/oracle/cell11.2.1.2.1_LINUX.X64_100131/log/diag/asm/cell/<cell name>/incident/incdir_iii/rstrc_nnnn_m_ikkk.trc


In the cell RS trace/incident trace file diag/asm/cell/<cell name>/incident/incdir_iii/rstrc_nnnn_m_ikkk.trc:

ORACLE_HOME = /opt/oracle/cell11.2.1.2.1_LINUX.X64_100131
System name: Linux
Node name: <fully qualified cell name>
Release: 2.6.18-128.1.16.0.1.el5
Version: #1 SMP <date>
Machine: x86_64
CELL SW Version: OSS_11.2.1.2.1_LINUX.X64_100131

RS-7445 [Serv CELLSRV absent] [It will be restarted] [] [] []
========= Dump for incident iii (RS 7445) ========
Starting a Diag Context default dump (level=3)

----- Call Stack Trace -----
ossrsutl_dump_incid (ent) <- ossrsutl_monitor_sr (vc) <- ossrsutl_monitor_sr (vc_prc) <- sossrs_prc_start <- ossrsutl_monitor_mo (npr_thd) <- start_thread


Cell alert history (cellcli "list alerthistory") would show messages like these:


NN <date and time> critical "ORA-00600: internal error code, arguments: [UserThread::run_1], [1841], []"
NN+1 <date and time> critical "ORA-07445: exception encountered: core dump [__intel_new_memcpy()] [11] [0x000000000] [] ORA-00600: internal error code, arguments: [UserThread::run_1], [1841], []"
NN+2 <date and time> critical "RS-7445 [Service CELLSRV is hung] [It will be restarted] []"
...


Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms