My Oracle Support Banner

Problem When Recovering Aggregation or IPMP Configuration in OVM for SPARC 3.3 on Fujitsu M10 (Doc ID 2121915.1)

Last updated on JANUARY 03, 2018

Applies to:

Solaris SPARC Operating System - Version 11 11/11 to 11.3 [Release 11.0]
Information in this document applies to any platform.

Symptoms

Two issues can occur if procedure below are followed:

1) An interface is not recovered by the aggregation.

2) The /etc/dladm/datalink-DefaultFixed.conf file get corrupted if  missing interface is added.

 

1. dladm create-aggr -l net1 -l net3 aggr0
(net3 is a device on PCIE1)

root@mten1:~# uname -a
SunOS mten1 5.11 11.3 sun4v sparc sun4v

root@mten1:~# ldm list
NAME STATE FLAGS CONS VCPU MEMORY UTIL NORM UPTIME
primary active -n-cv- UART 4 30976M 0.1% 0.1% 4h 53m

root@mten1:~# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1 primary IOV
/SYS/MBU/NET0 PCIE PCIE0 primary OCC
/SYS/PCI0 PCIE PCIE0 primary EMP
/SYS/MBU/SASHBA PCIE PCIE0 primary OCC
/SYS/MBU/NET2 PCIE PCIE1 primary OCC
/SYS/PCI1 PCIE PCIE1 primary EMP
/SYS/PCI2 PCIE PCIE1 primary OCC
/SYS/MBU/NET0/IOVNET.PF0 PF PCIE0 primary
/SYS/MBU/NET0/IOVNET.PF1 PF PCIE0 primary
/SYS/MBU/NET2/IOVNET.PF0 PF PCIE1 primary
/SYS/MBU/NET2/IOVNET.PF1 PF PCIE1 primary

root@mten1:~# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 Ethernet up 1000 full igb0
net1 Ethernet unknown 0 unknown igb1
net2 Ethernet unknown 0 unknown igb2
net3 Ethernet unknown 0 unknown igb3
net4 Ethernet unknown 0 unknown nxge0
net5 Ethernet unknown 0 unknown nxge1

root@mten1:~# dladm show-phys -L
LINK DEVICE LOC
net0 igb0 /SYS/MBU
net1 igb1 /SYS/MBU
net2 igb2 /SYS/MBU
net3 igb3 /SYS/MBU
net4 nxge0 PCI#2
net5 nxge1 PCI#2

root@mten1:~# dladm show-link
LINK CLASS MTU STATE OVER
net0 phys 1500 up --
net1 phys 1500 unknown --
net2 phys 1500 unknown --
net3 phys 1500 unknown --
net4 phys 1500 unknown --
net5 phys 1500 unknown --

root@mten1:~# dladm create-aggr -m dlmp -l net1 -l net3 aggr0

root@mten1:~# dladm show-aggr
LINK MODE POLICY ADDRPOLICY LACPACTIVITY LACPTIMER
aggr0 dlmp -- -- -- --

root@mten1:~# dladm show-aggr -L
LINK PORT AGGREGATABLE SYNC COLL DIST DEFAULTED EXPIRED
aggr0 net1 -- -- -- -- -- --
-- net3 -- -- -- -- -- --

root@mten1:~# ipadm create-ip aggr0

root@mten1:~# ipadm create-addr -T static -a local=192.168.1.222/24 aggr0/v4

root@mten1:~# ipadm show-addr aggr0/v4
ADDROBJ TYPE STATE ADDR
aggr0/v4 static ok 192.168.1.222/24

2. ldm start-reconf primary

root@mten1:~# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

3. ldm remove-io PCIE1 primary
(remove the bus by Static method)

root@mten1:~# ldm remove-io PCIE1 primary
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
root@mten1:~# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1
/SYS/MBU/NET0 PCIE PCIE0 primary OCC
/SYS/PCI0 PCIE PCIE0 primary EMP
/SYS/MBU/SASHBA PCIE PCIE0 primary OCC
/SYS/MBU/NET2 PCIE PCIE1 UNK
/SYS/PCI1 PCIE PCIE1 UNK
/SYS/PCI2 PCIE PCIE1 UNK
/SYS/MBU/NET0/IOVNET.PF0 PF PCIE0 primary
/SYS/MBU/NET0/IOVNET.PF1 PF PCIE0 primary

4. reboot

5. dladm show-aggr aggr0
net3 has been deleted from aggr0.

root@mten1:~# dladm show-aggr -L aggr0
LINK PORT AGGREGATABLE SYNC COLL DIST DEFAULTED EXPIRED
aggr0 net1 -- -- -- -- -- --

root@mten1:~# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1
/SYS/MBU/NET0 PCIE PCIE0 primary OCC
/SYS/PCI0 PCIE PCIE0 primary EMP
/SYS/MBU/SASHBA PCIE PCIE0 primary OCC
/SYS/MBU/NET2 PCIE PCIE1 UNK
/SYS/PCI1 PCIE PCIE1 UNK
/SYS/PCI2 PCIE PCIE1 UNK
/SYS/MBU/NET0/IOVNET.PF0 PF PCIE0 primary
/SYS/MBU/NET0/IOVNET.PF1 PF PCIE0 primary

6. ldm add-io PCIE1 primary
(add the bus by Dynamic method)

root@mten1:~# ldm add-io PCIE1 primary

root@mten1:~# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1 primary IOV
/SYS/MBU/NET0 PCIE PCIE0 primary OCC
/SYS/PCI0 PCIE PCIE0 primary EMP
/SYS/MBU/SASHBA PCIE PCIE0 primary OCC
/SYS/MBU/NET2 PCIE PCIE1 primary OCC
/SYS/PCI1 PCIE PCIE1 primary EMP
/SYS/PCI2 PCIE PCIE1 primary OCC
/SYS/MBU/NET0/IOVNET.PF0 PF PCIE0 primary
/SYS/MBU/NET0/IOVNET.PF1 PF PCIE0 primary
/SYS/MBU/NET2/IOVNET.PF0 PF PCIE1 primary
/SYS/MBU/NET2/IOVNET.PF1 PF PCIE1 primary

7. dladm show-aggr aggr0
net3 is added to the primary domain, but is not showing in aggr0.
This is the first problem.

root@mten1:~# dladm show-aggr -L aggr0
LINK PORT AGGREGATABLE SYNC COLL DIST DEFAULTED EXPIRED
aggr0 net1 -- -- -- -- -- --

root@mten1:~# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 Ethernet up 1000 full igb0
net1 Ethernet up 1000 full igb1
net2 Ethernet unknown 0 unknown igb2
net3 Ethernet unknown 1000 full igb3
net4 Ethernet unknown 0 unknown nxge0
net5 Ethernet unknown 0 unknown nxge1

root@mten1:~# dladm show-phys -L
LINK DEVICE LOC
net0 igb0 /SYS/MBU
net1 igb1 /SYS/MBU
net2 igb2 /SYS/MBU
net3 igb3 /SYS/MBU
net4 nxge0 PCI#2
net5 nxge1 PCI#2

root@mten1:~# dladm show-link
LINK CLASS MTU STATE OVER
net0 phys 1500 up --
net1 phys 1500 up --
net2 phys 1500 unknown --
net3 phys 1500 unknown --
net4 phys 1500 unknown --
net5 phys 1500 unknown --
aggr0 aggr 1500 up net1

8. dladm add-aggr -l net3 aggr0
net3 is shown in aggr0.

root@mten1:~# dladm add-aggr -l net3 aggr0
root@mten1:~# dladm show-aggr -L aggr0
LINK PORT AGGREGATABLE SYNC COLL DIST DEFAULTED EXPIRED
aggr0 net1 -- -- -- -- -- --
-- net3 -- -- -- -- -- --
root@mten1:~# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 Ethernet up 1000 full igb0
net1 Ethernet up 1000 full igb1
net2 Ethernet unknown 0 unknown igb2
net3 Ethernet up 1000 full igb3
net4 Ethernet unknown 0 unknown nxge0
net5 Ethernet unknown 0 unknown nxge1

9. reboot

10. dladm show-aggr aggr0
The aggr0 device has disappeared. This is the second problem.

root@mten1:~# dladm show-aggr

root@mten1:~# dladm show-aggr -L

root@mten1:~# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 Ethernet up 1000 full igb0
net1 Ethernet unknown 0 unknown igb1
net2 Ethernet unknown 0 unknown igb2
net3 Ethernet unknown 0 unknown igb3
net4 Ethernet unknown 0 unknown nxge0
net5 Ethernet unknown 0 unknown nxge1

root@mten1:~# dladm show-link
LINK CLASS MTU STATE OVER
net0 phys 1500 up --
net1 phys 1500 unknown --
net2 phys 1500 unknown --
net3 phys 1500 unknown --
net4 phys 1500 unknown --
net5 phys 1500 unknown --

root@mten1:~# cat /etc/dladm/datalink-DefaultFixed.conf
linkname=aggr0; class="uint64",4;media=uint64,4;lacp_timer=uint64,1;lacp_mode=uint64,0;force=boolean,false;fix_macaddr=bool
ean,false;policy=uint64,4;aggr_mode=uint64,1;portnames=string,net1:net3:net3:;nports=uint64,3;key=uint64,0;

The "portnames" and "nports" are abnormal values.

 

 

NOTE:

 

The same issues are seen if IPMP interface is used  instead of the aggregation.

The commands below can be used to create the IPMP interface:

root@mten1:~# ipadm create-ipmp ipmp0

root@mten1:~# ipadm create-ip net1

root@mten1:~# ipadm create-ip net2

root@mten1:~# ipadm add-ipmp -i net1 -i net2 ipmp0

 

Changes

 The customer was testing recovery of the aggregation after a fault/failure

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.