Exalogic Virtual: How To Debug And Fix Authentication Failure, Assets Blacklisting And Incorrect Credential Issues In EMOC?
(Doc ID 2431602.1)
Last updated on NOVEMBER 14, 2022
Applies to:
Oracle Exalogic Elastic Cloud Software - Version 2.0.6.0.0 and laterLinux x86-64
Oracle Virtual Server x86-64
Goal
This MOS Note provides detailed steps as follows:
- Symptoms and error patterns to confirm and verify which assets have Authentication failures/Incorrect credentials issues
- Steps to manual fix authentication issues by manually updating the existing credentials.
- Verify there are no authentication issues after updating the credentials.
In Exalogic Virtual racks, often we see authentication failures and incorrect credential related issues in EMOC. Authentication failures happen due to having incorrect credentials in EMOC for following Exalogic Infrastructure assets.
- Compute Nodes
- Compute Nodes ILOMs
- Storage Nodes
- Storage Nodes ILOMs
- Infiniband Switches
Having authentication failures on above assets in EMOC leads to EMOC blacklisting above assets in EMOC Domain model. This leads to other issues when doing vDC tasks in EMOC few of which are listed below.
- vServer creation Issues from EMOC.
- Stop & Start of existing vServers from EMOC.
- Repeated incidents from EMOC which shows assets as being down.
- Creation of New networks failing from EMOC.
- Incorrect values shown at vDC account level in EMOC in terms of memory, CPUs, network IPs etc.
- vServers status shown incorrectly from EMOC.
- Proxy Controller (PC) service taking long time to start.
The cause for the authentication related issues in most of the cases has been due to known issues with the Exapasswd tool.
Solution
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Goal |
| Solution |
| STEP 1: Gather IP's of Compute Nodes & Compute Nodes ILOMs, Storage Heads & Storage Heads ILOMs, IB Switches, OVM Manager |
| STEP 2: Check which Assets have Authentication Failures/Incorrect Credential issues |
| 2.1. Check which Compute Nodes and Compute Node ILOMs Assets have Authentication Failures |
| 2.1.1 Check from Proxy Controller cacao logs which Compute Node OS & ILOM IPs are having authentication failures and are blacklisted. |
| 2.1.2 Check from EMOC BUI Console If the Compute Nodes which have authentication errors are blacklisted from EMOC BUI. |
| 2.1.3 Compile list of Affected Compute Node OS & Compute Node Asset IP's which have authentication issues |
| 2.2. Check which Storage Nodes and Storage Nodes ILOMs assets have authentication failures |
| 2.2.1 Check from Proxy Controller cacao logs if there are authentication failures on Storage Node OS & ILOM IPs |
| 2.2.2 Check from EMOC BUI Console if Storage Nodes are blacklisted |
| 2.2.3 Compile list of Affected Storage Head OS & Storage Head ILOM IP's which have authentication issues |
| 2.3. Check which IB Switches assets have authentication failures |
| 2.3.1 Check from Proxy Controller cacao logs if there are authentication failures on IB Switches IPs |
| 2.3.2 Compile list of Affected IB Switches IP's which have authentication issues |
| 2.4. Check if OVM Manager Asset has Authentication failures/Incorrect credential issues |
| 2.4.1. Check from EMOC BUI Console if OVM Manager is reachable |
| 2.4.2. Check from Proxy Controller cacao logs & OVMM Admin Server Log if there are authentication failures on OVM Manager |
| 2.5 Compile list of all the assets which have Credential Issues |
| STEP 3: Fixing Incorrect Crendetials for assets which have Authentication issues |
| 3.1 Fixing Affected Compute Node OS & Compute Node ILOM authentication/incorrect credential issues |
| 3.1.1 Fixing Compute Node ILOM SSH & ILOM IPMI Credential issues |
| 3.1.2 Fixing Compute Node OS SSH Credential issues |
| 3.2 Fixing Storage Heads OS & Storage Heads ILOM authentication/incorrect credential issues |
| 3.2.1 Fixing Storage Head ILOM SSH & Storage Head ILOM IPMI Credential issues |
| 3.2.2 Fixing Storage Head OS SSH Credential issues |
| 3.3 Fixing IB Switches SSH & IPMI Credentials |
| 3.3.1 Fixing IB Switch SSH Credentials |
| 3.3.2 Fixing IB Switch IPMI Credentials |
| 3.4 Fixing OVM Manager Credential issues in EMOC |
| STEP 4: Stop & Start EC & PC Services & Verify if the Authentication/Incorrect Crendential Issues are Fixed |
| 4.0 Stop & Start EC & PC Services in EC1 and PC1, PC2 Control vServers and clean up all old cacao logs on PC VM's |
| 4.1 Validate if Authentication Issues are fixed for Compute Node OS & Compute Node ILOMs |
| 4.1.1 Validate if Blacklisting is removed for Compute Node assets in EMOC. |
| 4.1.2 Validate that there are no authentication errors or blacklisting errors on Compute Node assets in PC cacoa logs. |
| 4.2 Validate if Authentication Issues are fixed for Storage Head OS & Storage Head ILOMs |
| 4.2.1 Validate if Blacklisting is removed for Storage Head assets in EMOC. |
| 4.2.2 Validate that there are no authentication errors or blacklisting errors on Storage Head assets in PC cacoa logs. |
| 4.3 Validate if Authentication Issues are fixed for IB Switches which had issues before |
| 4.4 Validate if Authentication Issues are fixed for IB Switches which had issues before |
| 4.4.1 Validate if OVM Manager is Reachable from EMOC BUI |
| 4.4.1 Validate OVM Manager is NOT marked as unreachable from Proxy Controller cacao logs |
| Collect Full Exalogs Contact Support if still Authentication failure/Blacklisting of the assets is observed after following procedure in this Note |
| References |