My Oracle Support Banner

ZFS can Panic When Storage Is Re-added (Doc ID 2070645.1)

Last updated on JULY 29, 2016

Applies to:

Solaris Operating System - Version 11.2 and later
Information in this document applies to any platform.
ZFS can panic when storage is re-added one disk at a time

Symptoms

 Typical steps that can lead to this issue are:

1. Power off the old upper JBOD enclosure and remove it from the rack.
- Half of the mirrored pairs went "UNAVAIL" and the zpool went "DEGRADED", as expected.


2. Install new upper JBOD enclosure into the rack in the old upper enclosure's place, and power it on.


3. One by one, remove the physical disks from the old upper JBOD enclosure and insert them to the new JBOD enclosure (this requires a tray swap, which takes a few minutes per disk).
- After a minute or two, most of the disks came "ONLINE" on their own, but for any that didn't come online by themselves we ran "fmadm repaired <disk_id>" and they came online.


4. Once all the disks are swapped, check zpool status and confirm/wait until all resilvering are complete.
- The resilvering were all done by the time we got back to the console and ran the zpool status command.


5. Power off the old lower JBOD enclosure and remove it from the rack.
- Half of the mirrored pairs went "UNAVAIL" and the zpool went "DEGRADED", as expected.


6. Install new lower JBOD enclosure into the rack in the old lower enclosure's place, and power it on.


7. One by one, remove the physical disks from the old lower JBOD enclosure and insert them into the new JBOD enclosure (this requires a tray swap, which takes a few minutes per disk).


==== Panic occurred after the 2nd or 3rd disk was inserted in the lower JBOD enclosure ===

From this point onward, the zpool can be imported only after the second tray is physically powered off (and one can detach the "UNAVAIL" vdevs to get get non-mirrored zpool as "healthy" configuration). But, every attempt to attach the disk from that JBOD will result in panic similar to:

  

Oct 13 08:06:03 <hostname> genunix: [ID 603766 kern.notice] assertion failed: zap_add(os, DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_BPMAP_DEFER, sizeof (uint64_t), 1, &bm->bm_defer_obj, tx) == 0 (0x11 == 0x0), file: ../../common/fs/
zfs/bpmap.c, line: 691

 

Changes

 Powering off one mirror of the JBOD and then powering it on and attaching one disk at a time

The zpool will go look something like this:

  

pool: xxx
state: DEGRADED
status: One or more devices are unavailable in response to persistent errors.
      Sufficient replicas exist for the pool to continue functioning in a
      degraded state.
action: Determine if the device needs to be replaced, and clear the errors
      using 'zpool clear' or 'fmadm repaired', or replace the device
      with 'zpool replace'.
scan: resilvered 105M in 307445734561424665h30m with 0 errors on Wed Dec 31 19:00:00 1969
config:

      NAME                       STATE     READ WRITE CKSUM
      tank01                     DEGRADED     0     0     0
        mirror-0                 DEGRADED     0     0     0
          c0t5000C5004F4BA963d0  ONLINE       0     0     0
          c0t5000C5005CF852F9d0  UNAVAIL      0     0     0
        mirror-1                 DEGRADED     0     0     0
          c0t5000C5004F4C7642d0  ONLINE       0     0     0
          c0t5000C5007CE378FCd0  UNAVAIL      0     0     0
        mirror-2                 DEGRADED     0     0     0
          c0t5000C5005DAB6C1Ed0  ONLINE       0     0     0
          c0t5000C5005DAB4A25d0  UNAVAIL      0     0     0
        mirror-3                 DEGRADED     0     0     0
          c0t5000CCA248DA4C65d0  ONLINE       0     0     0
          c0t5000C5005DA3D7DCd0  UNAVAIL      0     0     0
        mirror-4                 DEGRADED     0     0     0
          c0t5000C5004E64109Cd0  ONLINE       0     0     0
          c0t5000C5005DAB8FFAd0  UNAVAIL      0     0     0
        mirror-5                 DEGRADED     0     0     0
          c0t5000C5004F4B47B6d0  ONLINE       0     0     0
          c0t5000C5005DAA4BB8d0  UNAVAIL      0     0     0
        mirror-6                 DEGRADED     0     0     0
          c0t50014EE6AE1EAE57d0  ONLINE       0     0     0
          c0t5000C5004F394870d0  UNAVAIL      0     0     0
        mirror-7                 DEGRADED     0     0     0
          c0t5000CCA248D102B7d0  ONLINE       0     0     0
          c0t5000C5004E642182d0  UNAVAIL      0     0     0
        mirror-8                 DEGRADED     0     0     0
          c0t5000C5005DAB8CCDd0  ONLINE       0     0     0
          c0t5000C500538E5904d0  UNAVAIL      0     0     0
        mirror-9                 DEGRADED     0     0     0
          c0t5000C50052BD3A8Fd0  ONLINE       0     0     0
          c0t5000C5004F4B6E9Ad0  UNAVAIL      0     0     0
        mirror-10                DEGRADED     0     0     0
          c0t5000C5005B4835E4d0  ONLINE       0     0     0
          c0t5000C500664C7184d0  UNAVAIL      0     0     0
        mirror-11                DEGRADED     0     0     0
          c0t5000C5007CDF0F56d0  ONLINE       0     0     0
          c0t5000C5004F421C4Ed0  UNAVAIL      0     0     0

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
 Bug
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.