My Oracle Support Banner

Important Service Issues When Upgrading to OS8.8.x (Doc ID 2545437.1)

Last updated on OCTOBER 01, 2020

Applies to:

Sun ZFS Storage 7420 - Version All Versions and later
Internal ZFS ZS5-4 Rack - Version All Versions and later
Oracle ZFS Storage ZS5-4 - Version All Versions and later
Oracle ZFS Storage ZS4-4 - Version All Versions and later
Oracle ZFS Storage Appliance Racked System ZS7-2 for PCA/CatC - Version All Versions and later
Information in this document applies to any platform.

Details

IMPORTANT SERVICE ISSUES WHEN UPGRADING A ZFS STORAGE APPLIANCE(ZFSSA)TO OS8.8.x

IMPORTANT SERVICE ISSUES WITH OS8.8.X WHICH REQUIRE THREE WORKFLOWS TO BE RUN

FOLLOW BOTH DOCS FOR INSTRUCTIONS ON APPLYING WORKFLOWS

Actions

Issues effecting OS8.8.x

Below is a list of known bugs & documents related to OS8.8.x. containing fixes, workarounds or directions to identify known issues.

To be sure the fix is in the OS8.8.x version you are planning on downloading and upgrading to, use our software download page listed below. 

Oracle ZFS Storage Appliance: Software Updates (Doc ID 2021771.1)

  1. Select the "Release Notes" Link to the right of the OS8.8.x version you are planning on downloading, which will open a pdf.
  2. Navigate to "Closed Issues" which are fixed in this release. It will list the Synopsis & BUG #

 


Invalid stash objects must be removed before any update to OS 8.8.x

This issue is addressed once your ZFS Storage Appliance(ZFSSA) is running OS8.8.7; however, the proactive workflows still need to be run prior to updating for an older OS release.

You will no longer need to run the clean_orpan_objects.akwf see BUG with fix below.

 

This issue is addressed in OS8.8.7 an newer

  Unpublished Bug 30068279 - Backport 29705956 to AK-8.8-MICRO - Orphaned worksheets (owner deleted from appliance) causes cluster rejoin to fail invalid stash object

 

Prior to an upgrade of a standalone or clustered ZFSSA node(s) to OS8.8.x, you should run both proactive workflows listed below to identify and remove the invalid stash objects before you upgrade to prevent cluster join issues.

Run these two proactive workflows on both nodes  to identify and remove invalid stash objects before an upgrade to OS8.8.x:

Workflow name: Filename:
Scrub bundles with invalid format (29055824) checkOldBundles.akwf
Identify and clean orphaned workflows and worksheets

clean_orphan_objects.akwf

Reference:
   How to Upload and Execute a Workflow on a ZFS Storage Appliance (Doc ID 2359446.1)

This is a proactive approach to detect and clean up the invalid stash objects before the effect the ZFSSA.

Failure to implement these two workflows prior to an OS update from OS8.7x to OS8.8x or OS8.8x to a higher OS8.8.x version will result in this problem being seen:

244 error(s) have occurred during cluster rejoin.
The most recent error was: 'exchangeFail invoked'.
Cluster rejoin will be retried until successful.

Joining cluster ...

If this is encountered, reference:

  ZFS Storage Appliance: Software Update to Version 8.8.x Fails at Cluster Join With 'exchangeFail invoked'. (Doc ID 2487089.1)

and engage Oracle support for assistance.

 

 

Below is the Service Alert for this issue:

Using ZFS Encryption in Solaris and Storage Appliance (ZFSSA) May Cause the System to Run Out of Memory (Doc ID 2629925.1)

 


Upgrading to OS8.8.18 (from OS8.7 or earlier 8.8.X version) where the upgraded node sits at joining cluster.

Below are two different cluster join issues with OS8.8.6 & OS8.8.18  #1 has a fix #2 is Proactive and or Reactive depending.

 

Upgrade To OS8.8.18 May Fail Due To "replay svc tx for scrk property time_updatecheck not found" (Doc ID 2640228.1)

This is addressed in OS8.8.20 and newer.

  Unpublished Bug 30821271 - 20610833 causes upgrade problems due to time_updatecheck

 

A reactive workaround at the fail safe shell needs to be performed by Oracle Support.

If you experience the issue above open a service request (SR) with the ZFSSA support team.

 

Post-upgrade to AK 8.8.X, ZFS Head Fails to Rejoin Cluster with 'exchangeFail Invoked' Error (Doc ID 2650954.1)

This issue requires a proactive workflow to be run before the ZFSSA is upgraded to OS8.8.6 & higher in order to prevent the cluster join issue.

If you have upgraded to OS8.8.6 or higher and experience the above cluster join issue open a SR with our ZFSSA Support Team and a ZFS Storage Appliance Technical Support Engineer will assist.

  Unpublished Bug 30990403 - Post-upgrade to AK 8.8.18, ZFS head fails to rejoin cluster due to invalid trustedcerts object

 

The proactive workaround is documented in:

Post-upgrade to AK 8.8.X, ZFS Head Fails to Rejoin Cluster with 'exchangeFail Invoked' Error (Doc ID 2650954.1)

 

The checkZeroTrustedcert_v1.1.akwf workflow can be run prior to upgrading to OS8.8.6 & higer versions. 

 


After upgrading to OS8.8.18 through OS8.8.24, a link aggregation(aggr) with VNIC/VLAN can no longer communicate between appliance heads.

This is addressed in OS8.8.25 and newer.

  Unpublished Bug 31470350 - Aggr with VNIC/VLAN not working after upgrade to OS8.8.20

  Unpublished Bug 31631412 - Remove VNICs MAC address persistence between reboots

 

The workaround is documented in:

ZFS Storage Appliance (ZFSSA) Software Unable to Communicate Between Cluster Heads (Doc ID 2686434.1)


 


Upgrading to OS8.8.22 while using the Chrome Browser can  cause akProgrammerError under Maintenance -> System.

Engineering is working on a solution in Bug below.

  Unpublished Bug 31572434 - AKPROGRAMMERERROR USING CHROME AFTER UPGRADE TO OS8.8.22. ERRORS AT MAINTENANCE -> SYSTEM

 

The workaround is documented in:

Error "akProgrammerError" Using Chrome After Upgrade to OS8.8.22. Errors at Maintenance -> System (Doc ID 2687231.1)

 


IMPORTANT Service issue with Upgrading OS8.8.x on 7120 


Do not install OS8.8.x on a Sun ZFS Storage 7120. If there is a suspected security issue, and you believe OS8.8.x is needed, open a SR with our ZFSSA TSE so we can review the issue and advise.

Oracle working on this issue in the Bugs listed below:

  Unpublished Bug 29288699 - AK8.8 upgrade fails on appliance 7120 due to mpt device probe issue

  Unpublished Bug 30165054 - OS8.8.x should not install on unsupported systems like 7120

 

We recommend upgrading to our recommended and latest appliance version OS8.7.29.

Our software downloads pages mentions the 7120 is not supported with OS8.8.x. See info below. 

Oracle ZFS Storage Appliance: Software Updates (Doc ID 2021771.1)

 

ProductMinimum Supported
Version
Recommended
Version
Latest
Version
7120 OS8.6.0 (2013.1.6.0) OS8.7.29 (2013.1.7.29) OS8.7.29 (2013.1.7.29)

 

 

 

 


7420, 7320, and 7120 systems support both 2011.1 and 2013.1 releases, except 7120 systems do not support OS8.8.x (2013.1.8.x).
    
7120 systems (OS8.8.x not supported)

 __________________________________________________________________________________________________________________________________________

IOM (I/O Controller Module) Fault During Cluster Upgrade to OS8.8.22 or Higher

IOM faults after upgrading AK to 8.8.22 and above.  If upgrading a cluster, after upgrading the first controller, upon reboot the upgraded controller will fault the IOM and the peer node can not be upgraded until the fault is addressed.  Check maintenance problems show for the event, problem and revision. If your issue matches follow steps in Doc below.

 

<EVENT_ID_MSG_ID>         STORAGE-8000-Q2 Major

The new problem class is "fault.enclosure.controller.firmware-detected.failure"

Revision         : 0000

 

 ZFS Storage Appliance IOM Fault During Cluster Upgrade to AK 8.8.22 or Higher After First Head Reboots - STORAGE-8000-Q2 Major (Doc ID 2711650.1)


Unpublished Bug 31906955 - DE2 tray firmware revision getting reset to invalid version

Unpublished Enh 31901536 - SES enclosure DE will fault IOM if it has invalid fw revision after upgrading 1st head

 

Enabling suspend notifications during maintenance / service activities can prevent false failures from generating unnecessary service requests. Oracle ZFS Storage Appliance: Suspend Notifications ( AK-8005-8W ) (Doc ID 2712745.1)

 

____________________________________________________________________________________________________________________________________________

Contacts

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Details
 IMPORTANT SERVICE ISSUES WHEN UPGRADING A ZFS STORAGE APPLIANCE(ZFSSA)TO OS8.8.x
Actions
 Invalid stash objects must be removed before any update to OS 8.8.x
 Backend multipathing does not recognize some newer drive models.
 Some NDMP Backups Inconsistent And Problematic
 ZFSSA Node with InfiniBand Hangs, NFS Shares Get Stuck on Clients
 "Deadlock: cycle in blocking chain" panic
 lockd: server <HOSTNAME> not responding
 ZFSSA not responding over HTTP
 ZFSSA reports it is an UNSUPPORTED platform
 Veritas OST and Microsoft VSS plugins require workflows not included in OS8.8.0 through 8.8.3
 Cluster Ethernet link intermittent issues
 Problems mounting NFS Shares
 Database crashes on NFS v3 & 4 shares after upgrading to OS 8.7.25 through 8.8.5
 Upgrading to OS8.8.18 (from OS8.7 or earlier 8.8.X version) where the upgraded node sits at joining cluster.
 Upgrade To OS8.8.18 May Fail Due To "replay svc tx for scrk property time_updatecheck not found" (Doc ID 2640228.1)
 Post-upgrade to AK 8.8.X, ZFS Head Fails to Rejoin Cluster with 'exchangeFail Invoked' Error (Doc ID 2650954.1)
 After upgrading to OS8.8.18 through OS8.8.24, a link aggregation(aggr) with VNIC/VLAN can no longer communicate between appliance heads.
 Upgrading to OS8.8.22 while using the Chrome Browser can  cause akProgrammerError under Maintenance -> System.
 IMPORTANT Service issue with Upgrading OS8.8.x on 7120 
 IOM (I/O Controller Module) Fault During Cluster Upgrade to OS8.8.22 or Higher
 Enabling suspend notifications during maintenance / service activities can prevent false failures from generating unnecessary service requests. Oracle ZFS Storage Appliance: Suspend Notifications ( AK-8005-8W ) (Doc ID 2712745.1)
Contacts
References

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.