Exadata Server Running Solaris 11 Express Rebooted After Kernel Panic
(Doc ID 1402475.1)
Last updated on OCTOBER 03, 2018
Applies to:
Solaris Operating SystemInformation in this document applies to any platform.
Symptoms
Exadata system running Solaris 11 Express on the database compute nodes can fail with a kernel panic and reboot. It's worth noting that Solaris 11 Express is a Solaris 11 pre-release operating environment.
The following entries in the /var/log/messages file:
...
Jan 23 17:18:18 exadb01.oracle.com sshd[13]: [ID 800047 auth.crit] fatal: Timeout before authentication for 11.48.23.213
Jan 24 08:10:58 exadb01.oracle.com unix: [ID 836849 kern.notice]
Jan 24 08:10:58 exadb01.oracle.com ^Mpanic[cpu21]/thread=ffffff00b8017c40:
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 103648 kern.notice] mutex_exit: not owner, lp=ffffff1bf127eae0 owner=0 thread=ffffff00b8017c40
Jan 24 08:10:58 exadb01.oracle.com unix: [ID 100000 kern.notice]
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017ac0 unix:mutex_panic+73 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 103648 kern.notice] mutex_exit: not owner, lp=ffffff1bf127eae0 owner=0 thread=ffffff00b8017c40
Jan 24 08:10:58 exadb01.oracle.com unix: [ID 100000 kern.notice]
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017ac0 unix:mutex_panic+73 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017ae0 unix:mutex_vector_exit+41 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017b00 unix:hat_exit+15 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017b30 unix:htable_free+69 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017b50 unix:htable_reap+76 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017b80 genunix:kmem_cache_reap+38 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017c20 genunix:taskq_thread+248 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017c30 unix:thread_start+8 ()
Jan 24 08:10:58 exadb01.oracle.com unix: [ID 100000 kern.notice]
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 672855 kern.notice] syncing file systems...
Jan 24 08:11:28 exadb01.oracle.com unix: [ID 836849 kern.notice]
Jan 24 08:11:28 exadb01.oracle.com ^Mpanic[cpu21]/thread=ffffff00b8017c40:
Jan 24 08:11:28 exadb01.oracle.com genunix: [ID 715357 kern.notice] panic sync timeout
Jan 24 08:11:28 exadb01.oracle.com unix: [ID 100000 kern.notice]
Jan 24 08:11:28 exadb01.oracle.com genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
Jan 24 08:25:57 exadb01.oracle.com genunix: [ID 100000 kern.notice]
Jan 24 08:25:57 exadb01.oracle.com genunix: [ID 665016 kern.notice] ^M100% done: 15102485 pages dumped,
Jan 24 08:25:57 exadb01.oracle.com genunix: [ID 851671 kern.notice] dump succeeded
Jan 24 08:28:12 exadb01.oracle.com genunix: [ID 540533 kern.notice] ^MSunOS Release 5.11 Version snv_151a 64-bit
...
Jan 24 08:31:53 exadb01.oracle.com savecore: [ID 570001 auth.error] reboot after panic: mutex_exit: not owner, lp=ffffff1bf127eae0 owner=0 thread=ffffff00b8017c40
...
Jan 24 08:34:31 exadb01.oracle.com EVENT-TIME: Sat Jan 24 08:34:31 EST 2011
Jan 24 08:34:31 exadb01.oracle.com PLATFORM: SUN-FIRE-X4170-M2-SERVER, CSN: 1049FMM0X0, HOSTNAME: exadb01.oracle.com
Jan 24 08:34:31 exadb01.oracle.com SOURCE: software-diagnosis, REV: 0.1
Jan 24 08:34:31 exadb01.oracle.com EVENT-ID: 09df89be-791f-c2fa-e2ae-bc9b86402141
Jan 24 08:34:31 exadb01.oracle.com DESC: The system has rebooted after a kernel panic. Refer to http://sun.com/msg/SUNOS-8000-KL for more information.
Jan 24 08:34:31 exadb01.oracle.com AUTO-RESPONSE: The failed system image was dumped to the dump device. If savecore is enabled (see dumpadm(1M)) a copy of the dump will be written to the savecore directory /var/crash/solaris.
Jan 24 08:34:31 exadb01.oracle.com IMPACT: There may be some performance impact while the panic is copied to the savecore directory. Disk space usage by panics can be substantial.
Jan 24 08:34:31 exadb01.oracle.com REC-ACTION: If savecore is not enabled then please take steps to preserve the crash image.
Jan 24 08:34:31 exadb01.oracle.com Use 'fmdump -Vp -u 09df89be-791f-c2fa-e2ae-bc9b86402141' to view more panic detail. Please refer to the knowledge article for additional information.
...
Jan 23 17:18:18 exadb01.oracle.com sshd[13]: [ID 800047 auth.crit] fatal: Timeout before authentication for 11.48.23.213
Jan 24 08:10:58 exadb01.oracle.com unix: [ID 836849 kern.notice]
Jan 24 08:10:58 exadb01.oracle.com ^Mpanic[cpu21]/thread=ffffff00b8017c40:
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 103648 kern.notice] mutex_exit: not owner, lp=ffffff1bf127eae0 owner=0 thread=ffffff00b8017c40
Jan 24 08:10:58 exadb01.oracle.com unix: [ID 100000 kern.notice]
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017ac0 unix:mutex_panic+73 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 103648 kern.notice] mutex_exit: not owner, lp=ffffff1bf127eae0 owner=0 thread=ffffff00b8017c40
Jan 24 08:10:58 exadb01.oracle.com unix: [ID 100000 kern.notice]
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017ac0 unix:mutex_panic+73 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017ae0 unix:mutex_vector_exit+41 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017b00 unix:hat_exit+15 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017b30 unix:htable_free+69 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017b50 unix:htable_reap+76 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017b80 genunix:kmem_cache_reap+38 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017c20 genunix:taskq_thread+248 ()
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 655072 kern.notice] ffffff00b8017c30 unix:thread_start+8 ()
Jan 24 08:10:58 exadb01.oracle.com unix: [ID 100000 kern.notice]
Jan 24 08:10:58 exadb01.oracle.com genunix: [ID 672855 kern.notice] syncing file systems...
Jan 24 08:11:28 exadb01.oracle.com unix: [ID 836849 kern.notice]
Jan 24 08:11:28 exadb01.oracle.com ^Mpanic[cpu21]/thread=ffffff00b8017c40:
Jan 24 08:11:28 exadb01.oracle.com genunix: [ID 715357 kern.notice] panic sync timeout
Jan 24 08:11:28 exadb01.oracle.com unix: [ID 100000 kern.notice]
Jan 24 08:11:28 exadb01.oracle.com genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
Jan 24 08:25:57 exadb01.oracle.com genunix: [ID 100000 kern.notice]
Jan 24 08:25:57 exadb01.oracle.com genunix: [ID 665016 kern.notice] ^M100% done: 15102485 pages dumped,
Jan 24 08:25:57 exadb01.oracle.com genunix: [ID 851671 kern.notice] dump succeeded
Jan 24 08:28:12 exadb01.oracle.com genunix: [ID 540533 kern.notice] ^MSunOS Release 5.11 Version snv_151a 64-bit
...
Jan 24 08:31:53 exadb01.oracle.com savecore: [ID 570001 auth.error] reboot after panic: mutex_exit: not owner, lp=ffffff1bf127eae0 owner=0 thread=ffffff00b8017c40
...
Jan 24 08:34:31 exadb01.oracle.com EVENT-TIME: Sat Jan 24 08:34:31 EST 2011
Jan 24 08:34:31 exadb01.oracle.com PLATFORM: SUN-FIRE-X4170-M2-SERVER, CSN: 1049FMM0X0, HOSTNAME: exadb01.oracle.com
Jan 24 08:34:31 exadb01.oracle.com SOURCE: software-diagnosis, REV: 0.1
Jan 24 08:34:31 exadb01.oracle.com EVENT-ID: 09df89be-791f-c2fa-e2ae-bc9b86402141
Jan 24 08:34:31 exadb01.oracle.com DESC: The system has rebooted after a kernel panic. Refer to http://sun.com/msg/SUNOS-8000-KL for more information.
Jan 24 08:34:31 exadb01.oracle.com AUTO-RESPONSE: The failed system image was dumped to the dump device. If savecore is enabled (see dumpadm(1M)) a copy of the dump will be written to the savecore directory /var/crash/solaris.
Jan 24 08:34:31 exadb01.oracle.com IMPACT: There may be some performance impact while the panic is copied to the savecore directory. Disk space usage by panics can be substantial.
Jan 24 08:34:31 exadb01.oracle.com REC-ACTION: If savecore is not enabled then please take steps to preserve the crash image.
Jan 24 08:34:31 exadb01.oracle.com Use 'fmdump -Vp -u 09df89be-791f-c2fa-e2ae-bc9b86402141' to view more panic detail. Please refer to the knowledge article for additional information.
...
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |