My Oracle Support Banner

[PCA] - Storage Network upgrade failure at Spine Switch backup stage. (Doc ID 2690221.1)

Last updated on JULY 24, 2020

Applies to:

Private Cloud Appliance - Version 2.4.2 and later
Information in this document applies to any platform.

Symptoms

Storage Network upgrade fails like this eventhough the pre_check completed successfully:

[root@ovcamn05r1 ~]# pca_upgrader -U -t storage-network
PCA Rack Type: PCA X8_BASE.
Please refer to log file /var/log/pca_upgrader_2020_06_29-12.24.33.log for more details.

Beginning PCA Storage Network Pre-Upgrade Checks...

Rack Type Check 1/14
PCA Version Check 2/14
Upgrade Locks Check 3/14
Backup Tasks Check 4/14
Storage Port Channel Status Check 5/14
Check ZFSSA MGMT Network 6/14
Spine Switch Firmware Check 7/14
ZFSSA Resilvering Jobs Check 8/14
Check Cluster Status 9/14
ZFSSA Hardware Error Check 10/14
AK Firmware Version Check 11/14
iSCSI Target Check 12/14
Check ZFSSA Default Shares 13/14
ZFSSA Network Configuration Check 14/14

PCA Storage Network Pre-Upgrade Checks completed after 0 minutes

Beginning PCA Storage Network Upgrade
Disable PCA Backups 1/6
Take Spine Switch Backup 2/6
PCA Storage Network Upgrade failed during SETUP; terminating and cleaning up.
Remove PCA Upgrade Locks 1
Re-enable PCA Backups 2
PCA Storage Network Upgrade failed
PCA Storage Network Upgrade completed after 0 minutes
---------------------------------------------------------------------------
PCA Storage Network Pre-Upgrade Checks Passed
---------------------------------------------------------------------------
Rack Type Check Passed
PCA Version Check Passed
Upgrade Locks Check Passed
Backup Tasks Check Passed
Storage Port Channel Status Check Passed
Check ZFSSA MGMT Network Passed
Spine Switch Firmware Check Passed
ZFSSA Resilvering Jobs Check Passed
Check Cluster Status Passed
ZFSSA Hardware Error Check Passed
AK Firmware Version Check Passed
iSCSI Target Check Passed
Check ZFSSA Default Shares Passed
ZFSSA Network Configuration Check Passed
---------------------------------------------------------------------------
PCA Storage Network Upgrade Failed
---------------------------------------------------------------------------
Disable PCA Backups Passed
Take Spine Switch Backup Failed
Take ZFSSA Configuration Backup Not Run
Place Storage Network Upgrade Locks Not Run
Perform Storage Network Upgrade Not Run
Remove Firmware Upgrade Lock Not Run
---------------------------------------------------------------------------
PCA Storage Network Post-Upgrade Checks Not Run
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Overall Status Failed
---------------------------------------------------------------------------
PCA Storage Network Pre-Upgrade Checks Passed
PCA Storage Network Upgrade Failed
PCA Storage Network Post-Upgrade Checks Not Run
Please refer to log file /var/log/pca_upgrader_2020_06_29-13.10.41.log for
more details.

LOG SNIPPET:

[2020-06-29 13:11:47 216190] ERROR (cisco:153) POST request to cisco switch
192.168.4.204 failed, status code 500 (Login incorrect.
Login failed.
Please login with USER and PASS
Please login with USER and PASS
***** Transfer of file aborted, Login failed*****
***** Transfer of file aborted *****)
[2020-06-29 13:11:47 216190] ERROR (cisco:154) {
"jsonrpc": "2.0",
"error": {
"code": -32602,
"message": "Invalid params",
"data": {
"msg": "Login incorrect.\nLogin failed.\nPlease
login with USER and PASS\nPlease login with USER and PASS\n***** Transfer of
file aborted, Login failed*****\n***** Transfer of file aborted *****\n"
}
},
"id": 1
}
[2020-06-29 13:11:47 216190] ERROR (cisco:158) Exception while POST request
for Cisco Session POST request to cisco switch 192.168.4.204 failed, status
code 500 (Login incorrect.
Login failed.
Please login with USER and PASS
Please login with USER and PASS
***** Transfer of file aborted, Login failed*****
***** Transfer of file aborted *****)
[2020-06-29 13:11:47 216190] ERROR (cisco:563) Failed to backup the switch
configuration: 192.168.4.204
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/ovca/core/network/data/cisco/cisco.py",
line 560, in save_backup
self._show(cmds=cmds)
File
"/usr/lib/python2.6/site-packages/ovca/core/network/data/cisco/cisco.py",
line 184, in _show
return self.post(node="ins", payload=payload, content_type="json-rpc")
File
"/usr/lib/python2.6/site-packages/ovca/core/network/data/cisco/cisco.py",
line 155, in post
raise Exception(msg)
Exception: POST request to cisco switch 192.168.4.204 failed, status code 500
(Login incorrect.
Login failed.
Please login with USER and PASS
Please login with USER and PASS
***** Transfer of file aborted, Login failed*****
***** Transfer of file aborted *****)

[2020-06-29 13:11:48 216190] ERROR (cisco:153) POST request to cisco switch
192.168.4.205 failed, status code 500 (Login incorrect.
Login failed.
Please login with USER and PASS
Please login with USER and PASS
***** Transfer of file aborted, Login failed*****
***** Transfer of file aborted *****)

[2020-06-29 13:11:48 216190] ERROR (cisco:154) {
"jsonrpc": "2.0",
"error": {
"code": -32602,
"message": "Invalid params",
"data": {
"msg": "Login incorrect.\nLogin failed.\nPlease
login with USER and PASS\nPlease login with USER and PASS\n***** Transfer of
file aborted, Login failed*****\n***** Transfer of file aborted *****\n"
}
},
"id": 1
}
[2020-06-29 13:11:48 216190] ERROR (cisco:158) Exception while POST request
for Cisco Session POST request to cisco switch 192.168.4.205 failed, status
code 500 (Login incorrect.
Login failed.
Please login with USER and PASS
Please login with USER and PASS
***** Transfer of file aborted, Login failed*****
***** Transfer of file aborted *****)
[2020-06-29 13:11:48 216190] ERROR (cisco:563) Failed to backup the switch
configuration: 192.168.4.205
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/ovca/core/network/data/cisco/cisco.py",
line 560, in save_backup
self._show(cmds=cmds)
File
"/usr/lib/python2.6/site-packages/ovca/core/network/data/cisco/cisco.py",
line 184, in _show
return self.post(node="ins", payload=payload, content_type="json-rpc")
File
"/usr/lib/python2.6/site-packages/ovca/core/network/data/cisco/cisco.py",
line 155, in post
raise Exception(msg)

Changes

N/A

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.