Vous êtes sur la page 1sur 106

Sun Blade 6000 Ethernet Switched

NEM 24p 10GbE


Software Configuration Guide

Part Number E26660-01


October 2010

Copyright 2010, 2011, Oracle and/or its affiliates. All rights reserved.
This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by
intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce,
translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse
engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.
The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report
them to us in writing.
If this is software or related software documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S.
Government, the following notice is applicable:
U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government
customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and
agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the
restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government
contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500
Oracle Parkway, Redwood City, CA 94065.
This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for
use in any inherently dangerous applications, including applications which may create a risk of personal injury. If you use this software or
hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to
ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in
dangerous applications.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel
Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or
registered trademarks of SPARC International, Inc.
This software or hardware and documentation may provide access to or information on content, products, and services from third parties. Oracle
Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products,
and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of
third-party content, products, or services.
Copyright 2010, 2011, Oracle et/ou ses affilis. Tous droits rservs.
Ce logiciel et la documentation qui l'accompagne sont protgs par les lois sur la proprit intellectuelle. Ils sont concds sous licence et soumis
des restrictions d'utilisation et de divulgation. Sauf disposition de votre contrat de licence ou de la loi, vous ne pouvez pas copier, reproduire,
traduire, diffuser, modifier, breveter, transmettre, distribuer, exposer, excuter, publier ou afficher le logiciel, mme partiellement, sous quelque
forme et par quelque procd que ce soit. Par ailleurs, il est interdit de procder toute ingnierie inverse du logiciel, de le dsassembler ou de le
dcompiler, except des fins d'interoprabilit avec des logiciels tiers ou tel que prescrit par la loi.
Les informations fournies dans ce document sont susceptibles de modification sans pravis. Par ailleurs, Oracle Corporation ne garantit pas
qu'elles soient exemptes d'erreurs et vous invite, le cas chant, lui en faire part par crit.
Si ce logiciel, ou la documentation qui l'accompagne, est concd sous licence au Gouvernement des Etats-Unis, ou toute entit qui dlivre la
licence de ce logiciel ou l'utilise pour le compte du Gouvernement des Etats-Unis, la notice suivante s'applique :
U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government
customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and
agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the
restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government
contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500
Oracle Parkway, Redwood City, CA 94065.
Ce logiciel ou matriel a t dvelopp pour un usage gnral dans le cadre d'applications de gestion des informations. Ce logiciel ou matriel
n'est pas conu ni n'est destin tre utilis dans des applications risque, notamment dans des applications pouvant causer des dommages
corporels. Si vous utilisez ce logiciel ou matriel dans le cadre d'applications dangereuses, il est de votre responsabilit de prendre toutes les
mesures de secours, de sauvegarde, de redondance et autres mesures ncessaires son utilisation dans des conditions optimales de scurit. Oracle
Corporation et ses affilis dclinent toute responsabilit quant aux dommages causs par l'utilisation de ce logiciel ou matriel pour ce type
d'applications.
Oracle et Java sont des marques dposes d'Oracle Corporation et/ou de ses affilis. Tout autre nom mentionn peut correspondre des marques
appartenant d'autres propritaires qu'Oracle.
AMD, Opteron, le logo AMD et le logo AMD Opteron sont des marques ou des marques dposes d'Advanced Micro Devices. Intel et Intel Xeon
sont des marques ou des marques dposes d'Intel Corporation. Toutes les marques SPARC sont utilises sous licence et sont des marques ou des
marques dposes de SPARC International, Inc.
Ce logiciel ou matriel et la documentation qui l'accompagne peuvent fournir des informations ou des liens donnant accs des contenus, des
produits et des services manant de tiers. Oracle Corporation et ses affilis dclinent toute responsabilit ou garantie expresse quant aux contenus,
produits ou services manant de tiers. En aucun cas, Oracle Corporation et ses affilis ne sauraient tre tenus pour responsables des pertes subies,
des cots occasionns ou des dommages causs par l'accs des contenus, produits ou services tiers, ou leur utilisation.

Contents
Using This Documentation

Related Documentation
Acronyms and Abbreviations
CLI Command Modes
Feedback
Support and Accessibility
Configuring SEFOS

5
5
6
7
7
8

Connecting to SEFOS
Configuring SEFOS System Features
Default Configurations
Configuring the Default IP Address
Configuring the IP Address for an Interface
Configuring the Restoration File Name
Saving the Current Configurations for Restoration
Erasing a Saved Configuration File
Copying a Configuration File To A Remote Location
Copying a Configuration From A Remote Location to Flash
Copying System Logs To A Remote Location
Copying a File From One Remote Site or Flash to Another Remote Site or Flash
Enabling or Disabling Incremental Save Flag
Enabling or Disabling the Auto Save Flag
Configuring the Default VLAN Identifier
Enabling or Disabling Trap Generation on an Interface
Configuring Debug Logging
Configuring ACL Filters
Configuring Quality of Services
Configuring Port Mirroring
Configuring Rate Limiting
Sample Configurations
Topology Switching Feature
Getting Started
Configuring VLAN Forwarding
Rapid Spanning Tree Protocol
Configuring Link Aggregation
Saving Configurations
Topology - Routing Feature
Configuring Static Unicast Route Entries
Configuring Dynamic Routing - RIP
Configuring Dynamic Routing - OSPF
Topology SFP+ LAG and Backplane Portface Connection
Configuring Basic SFP+ LAG and Backplane
Topology Layer 2 Switch LAG and QSFP Splitter SAN Connection
Configuring a Layer 2 Switch LAG and QSFP Splitter SAN Connection
Topology Switched NEM High Availability Connection

9
10
10
12
13
14
15
17
18
18
18
18
19
19
19
21
22
22
25
28
30
32
33
33
36
38
41
44
46
46
53
55
57
57
61
61
66
3

Setting Up a Four NEM/Two Chassis High Availibility Configuration


66
Topology Four Switched NEMs, High Availability Connection
85
Setting Up a Four NEM/Four Chassis High Availibility Configuration
85
Setting Up QSFP Ports Into a Link Aggregation......................................................................... 86
Checking the Connectivity Status on the NEMs......................................................................... 87

4 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Chapter

1
Using This Documentation
The SEFOS offers a rich set of system features, such as login services, copying and writing facilities, duplex and
negotiation support, and many other features. This document explains how to configure the system features in a
switch running SEFOS. You should have a basic knowledge of the protocol as a prerequisite.
Related Documentation
Acronyms and Abbreviations
CLI Command Modes
Feedback
Support and Accessibility

Related Documentation
Description

Links

All Oracle products

http://oracle.com/documentation

Sun Blade 6000 Ethernet Switched


NEM 24p 10GbE

http://www.oracle.com/pls/topic/lookup?ctx=SB6K-24p-10GbE

Sun Network 10GbE Switch 72p

http://www.oracle.com/pls/topic/lookup?ctx=SN-10GbE-72p

Sun Blade 6000 Modular System

http://www.oracle.com/pls/topic/lookup?ctx=sb6000

Oracle Integrated Lights Out


Manager (Oracle ILOM) 3.0

http://www.oracle.com/pls/topic/lookup?ctx=ilom30

Acronyms and Abbreviations


The following acronyms and abbreviations are used in this document:

ACL

Access Control List

BOOTP

Bootstrap Protocol

CLI

Command Line Interface

DHCP

Dynamic Host Configuration Protocol

DSCP

Differentiated Services Code Point

HTTP

Hyper Text Transfer Protocol

ICMP

Internet Control Message Protocol

IP

Internet Protocol

MAC

Media Access Control

MTU

Maximum Transfer Unit

NVRAM

Non-Volatile Random Access Memory

PNAC

Port based Network Access Control

QoS

Quality-of-Service

RADIUS

Remote Authentication Dial-In User Service

RARP

Reverse Address Resolution Protocol

SEFOS

Sun Ethernet Fabric Operating System

SNMP

Simple Network Management Protocol

TCP

Transmission Control Protocol

CLI Command Modes


The following table lists the configuration modes used in this document with their access and exit methods.
Command Mode

Access Method

Prompt

Exit Method

User EXEC

Access SEFOS from the


Oracle ILOM prompt with
read-only rights (privilege
level 1).

SEFOS>

Use the logout or exit


command to return to the
Oracle ILOM prompt.

Privileged EXEC

Access SEFOS from


Oracle ILOM prompt with
full administrative rights
(privilege level 15).

SEFOS#

Use the logout or exit


command to return to the
Oracle ILOM prompt.

Global configuration

SEFOS(config)#
From User EXEC mode,
use the enable command.

Use the end command to


return to Privileged EXEC
mode.

Interface configuration

From Global Configuration SEFOS(config-if)#


mode, use the
interface
interface-type

Use the exit command to


return to Global
Configuration mode, or use
the end command to return

6 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Command Mode

Access Method

Prompt

interface-id
command.

Exit Method
to Privileged EXEC mode.

Feedback
Provide feedback on this documentation at:
http://www.oracle.com/goto/docfeedback

Support and Accessibility


Description

Links

Access electronic support through


My Oracle Support

http://support.oracle.com
For hearing impaired:
http://www.oracle.com/accessibility/support.html

Learn about Oracle's commitment


to accessibility

http://www.oracle.com/us/corporate/accessibility/index.html

Chapter

2
Configuring SEFOS
This chapter describes how to configure SEFOS software.
Connecting to SEFOS
Configuring SEFOS System Features
Default Configurations
Configuring the Default IP Address
Configuring the IP Address for an Interface
Configuring an Interface to Acquire Dynamic IP
Configuring the Base MAC Address
Configuring the Login Authentication Method
Configuring Authorized Managers for Configuring the Switch
Configuring the Restoration File Name
Saving the Current Configurations for Restoration
Erasing a Saved Configuration File
Copying a Configuration File To A Remote Location
Copying a Configuration From A Remote Location to Flash
Copying System Logs To A Remote Location
Copying a File From One Remote Site or Flash to Another Remote Site or Flash
Enabling or Disabling Incremental Save Flag
Enabling or Disabling the Auto Save Flag
Configuring the Default VLAN Identifier

8 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Configuring Switch Clock


Enabling/Disabling Console CLI through Serial Port
Enabling/Disabling HTTP
Configuring HTTP Port Number
Configuring Management VLAN List
Configuring MTU Frame Size for an Interface
Enabling or Disabling Trap Generation on an Interface
Configuring Debug Logging
Configuring ACL Filters
Configuring Quality of Services
Configuring Port Mirroring
Configuring Rate Limiting
Configuring Timers for Redundancy Manager
Configuring a Manual Switch Over from Active to Standby
Configuring Sizing Parameters

Connecting to SEFOS
To configure the SEFOS software, you must first establish a connection to your system. The connection must be
through the NEM's serial port or through ssh to the NEM's IP address.
Refer to the Oracle ILOM documentation for instructions on how to establish these connections.
You can connect to the NEM from the CMM. As an added security measure, such a connection is
read-only in SEFOS.
1. To access SEFOS from the ILOM prompt:
-> cd /NEM/fs_cli/
The SEFOS# prompt appears. For example:
-> cd /NEM/fs_cli/
cd: Connecting to Fabric Switch CLI as user root
SEFOS#
You are now connected to SEFOS and you can begin to configure the SEFOS features.
2. When you are finished using SEFOS, type exit to return to the ILOM prompt:
SEFOS# exit
Connection closed by foreign host.

cd: The session with /NEM/fs_cli has ended.


->
3. When you are finished with ILOM, type exit:
exit

Configuring SEFOS System Features

Figure 1. Configuration of Basic System Features

Figure 2. Configuration of Advanced System Features

Default Configurations
Feature
IP Address and Subnet Mask

Default Setting
10.0.0.1 and 255.0.0.0

IP Allocation Mode

Manual

Dynamic IP Allocation Protocol

DHCP

10 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Feature
Base MAC Address

Default Setting
The switch's base MAC address is derived from the
system's standard unique MAC address, shown on
the yellow sticker. This MAC address should be
sufficient for normal functionality of the switch, so
you do not have to change it.

Login Authentication Mode

Local

Authorized Managers

All users are allowed to configure the switch

Restoration File Name

switch.conf

CLI Console

Enabled

HTTP Status

Disabled

HTTP Port Number

80

Management VLAN

VLAN 1

MTU

9162

Trap Generation on an Interface

Enabled

Checksum on Tunnel Interface

Disabled

MTU Discovery on Tunnel Interface

Disabled

Debug logging

Logs in console

Max number of ISID

131072

Max number of ISID per context

65536

Max number of ports per ISID

128

Max number of ports per ISID_per_context

128

Max number of SVLANs per ISID

4094

Max number of CVLANs per ISID

4094

11

Configuring the Default IP Address


When you configure the default IP address, it will be written to the NVRAM. This default IP address will be used as
the IP address of the default interface when the switch is restarted.
1. Configure the default IP address.
Enter the Global Configuration mode:
SEFOS# configure terminal
Configure the default IP address and subnet mask as 10.0.0.100 and 255.255.0.0, respectively:
SEFOS(config)# default ip address 10.0.0.100 subnet-mask 255.255.0.0
Exit from the Global Configuration mode:
SEFOS(config)#end
2. View the default IP address and subnet mask:
SEFOS# show nvram
Default IP Address

: 10.0.0.1

Default Subnet Mask

: 255.0.0.0

Default IP Address Config Mode

: Manual

Default IP Address Allocation Protocol

: DHCP

Switch Base MAC Address

: 00:14:4f:6c:63:0f

Default Interface Name

: Ex0/1

Default RM Interface Name

: NONE

Config Restore Option

: No restore

Config Save Option

: No save

Auto Save

: Disable

Incremental Save

: Enable

Roll Back

: Enable

Config Save IP Address

: 0.0.0.0

Config Save Filename

: switch.conf

Config Restore Filename

: switch.conf

PIM Mode

: Sparse Mode

IGS Forwarding Mode

: MAC based

Cli Serial Console

: No

SNMP EngineID

: 80.00.08.1c.04.46.53

SNMP Engine Boots

: 16

Default VLAN Identifier

: 1

Stack PortCount

: 0

12 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

ColdStandby

: Disable

NPAPI Mode

: Synchronous

Dynamic Port Count

: 24

VLAN 1 (the default VLAN) will have this IP address and subnet mask after the switch is
restarted only if the allocation method is set to manual.

Configuring the IP Address for an Interface


This procedure describes how to configure the IP address that will be used for sending and receiving the packets.
1. Configure an IP address for a VLAN interface.
Enter the Global Configuration mode:
SEFOS# configure terminal
Enter the Interface Configuration mode:
SEFOS(config)# interface vlan 1
Shut down the VLAN interface:
SEFOS(config-if)# shut down
Configure the IP address and subnet mask:
SEFOS(config-if)# ip address 10.0.0.100

255.0.0.0

Bring up the VLAN interface:


SEFOS(config-if)# no shut down
Exit from the Interface Configuration mode:
SEFOS(config)# end
You must shut down the interface before you can configure the IP address for that interface.

2. View the configured interface IP address:


SEFOS# show ip interface
vlan1 is up, line protocol is up
Internet Address is 10.0.0.100/8
Broadcast Address

10.255.255.255

13

Configuring the Restoration File Name


This procedure describes how to write the restoration file to the NVRAM. This file will be used as the configuration
restoration file.
1. Configure the restoration file name.
Enter the Global Configuration mode:
SEFOS# configure terminal
Configure the configuration restoration file name for the switch:
SEFOS(config)# default restore-file myconfig.conf
Exit from the Global Configuration mode:
SEFOS(config)# end
2. View the default restoration file name:
SEFOS# show nvram
Default IP Address

: 10.0.0.1

Default Subnet Mask

: 255.0.0.0

Default IP Address Config Mode

: Manual

Default IP Address Allocation Protocol

: DHCP

Switch Base MAC Address

: 00:14:4f:6c:63:0f

Default Interface Name

: Ex0/1

Default RM Interface Name

: NONE

Config Restore Option

: No restore

Config Save Option

: No save

Auto Save

: Disable

Incremental Save

: Enable

Roll Back

: Enable

Config Save IP Address

: 0.0.0.0

Config Save Filename

: switch.conf

Config Restore Filename

: myconfig.conf

PIM Mode

: Sparse Mode

IGS Forwarding Mode

: MAC based

Cli Serial Console

: No

SNMP EngineID

: 80.00.08.1c.04.46.53

SNMP Engine Boots

: 16

Default VLAN Identifier

: 1

Stack PortCount

: 0

ColdStandby

: Disable

14 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

NPAPI Mode

: Synchronous

Dynamic Port Count

: 24

Saving the Current Configurations for Restoration


This procedure describes how to write the running configuration to a flash file, a startup configuration file or to a
remote site.
1. Save the configuration that is currently running to a file:
SEFOS# write startup-config
2. View the current restoration settings:
SEFOS# show nvram
Default IP Address

: 10.0.0.1

Default Subnet Mask

: 255.0.0.0

Default IP Address Config Mode

: Manual

Default IP Address Allocation Protocol

: DHCP

Switch Base MAC Address

: 00:14:4f:6c:63:0f

Default Interface Name

: Ex0/1

Default RM Interface Name

: NONE

Config Restore Option

: Restore

Config Save Option

: Startup save

Auto Save

: Enable

Incremental Save

: Enable

Roll Back

: Enable

Config Save IP Address

: 0.0.0.0

Config Save Filename

: switch.conf

Config Restore Filename

: myconfig.conf

PIM Mode

: Sparse Mode

IGS Forwarding Mode

: MAC based

Cli Serial Console

: No

SNMP EngineID

: 80.00.08.1c.04.46.53

SNMP Engine Boots

: 16

Default VLAN Identifier

: 1

Stack PortCount

: 0

ColdStandby

: Disable

NPAPI Mode

: Synchronous

Dynamic Port Count

: 24

15

3. View the restoration status:


SEFOS# show system information
Hardware Version

: 2.5.4_00161614

Firmware Version

: 1.0.2.0

Switch Name

: NEM 24P

System Contact

: Sun Microsystems, Inc.

System Location

: Sun Microsystems, Inc.

Logging Option

: Console Logging

Login Authentication Mode

: Local

Config Save Status

: Successful

Remote Save Status

: Not Initiated

Config Restore Status

: Not Initiated

4. Reboot the switch.


Refer to the documentation that came with your switch for those instructions.
5. View the restoration status after rebooting the switch:
SEFOS# show

system information

Hardware Version

: 2.5.4_00161614

Firmware Version

: 1.0.2.0

Switch Name

: NEM 24P

System Contact

: Sun Microsystems, Inc.

System Location

: Sun Microsystems, Inc.

Logging Option

: Console Logging

Login Authentication Mode

: Local

Config Save Status

: Not Initiated

Remote Save Status

: Not Initiated

Config Restore Status

: Successful

The current configurations will be saved to a file that you specified when you ran the
restore-file command in the previous section.
The default name for the restoration file is switch.conf.
The switch will start with the saved configuration when you reboot the switch.

16 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Erasing a Saved Configuration File


This procedure describes how to clear the contents of the startup configuration or set the parameters in NVRAM to
their default values.
1. Erase the saved configuration file:
SEFOS# erase startup-config
2. View the erase status settings:
SEFOS# show nvram
Default IP Address

: 10.0.0.1

Default Subnet Mask

: 255.0.0.0

Default IP Address Config Mode

: Manual

Default IP Address Allocation Protocol

: DHCP

Switch Base MAC Address

: 00:14:4f:6c:63:0f

Default Interface Name

: Ex0/1

Default RM Interface Name

: NONE

Config Restore Option

: No restore

Config Save Option

: No save

Auto Save

: Enable

Incremental Save

: Enable

Roll Back

: Enable

Config Save IP Address

: 0.0.0.0

Config Save Filename

: switch.conf

Config Restore Filename

: myconfig.conf

PIM Mode

: Sparse Mode

IGS Forwarding Mode

: MAC based

Cli Serial Console

: No

SNMP EngineID

: 80.00.08.1c.04.46.53

SNMP Engine Boots

: 16

Default VLAN Identifier

: 1

Stack PortCount

: 0

ColdStandby

: Disable

NPAPI Mode

: Synchronous

Dynamic Port Count

: 24

The switch will start with the default configurations when you reboot the switch.

17

Copying a Configuration File To A Remote Location


This procedure describes how to back up the initial configuration file to flash or to a remote location.
Copy the configuration file - switch.conf - to the location 12.0.0.100.
SEFOS#

copy startup-config tftp://12.0.0.100/switch.conf

Copying a Configuration From A Remote Location to Flash


This procedure describes how to copy the backup configuration file from a remote location to the current location of
exe for restoration.
Copy the startup configuration file switch.conf stored in the host 12.0.0.100 to the current location of exe:
SEFOS#

copy tftp://12.0.0.100/switch.conf startup-config

This action will copy the file switch.conf from the host 12.0.0.100 to the current exe location. This file will be used
as the startup configuration file when you reboot the switch.

Copying System Logs To A Remote Location


This procedure describes how to upload the log file to a remote location.
Upload the log file to the remote location 12.0.0.100:
SEFOS# copy logs tftp://12.0.0.100/logfile

Copying a File From One Remote Site or Flash to Another Remote Site
or Flash
This procedure describes how to copy a file from one remote site or flash to another remote site or flash.
Copy the file script.txt from the remote location 12.0.0.100 to flash:
SEFOS# copy tftp://12.0.0.100/script.txt

flash:script.txt

18 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Enabling or Disabling Incremental Save Flag


Enabling the incremental save flag updates the in-memory database for every configuration at runtime.
Disabling the incremental save flag does not update the in-memory database for any configuration at runtime.
However, you can save the memory database by running the write startup-config command. See the
section Saving the Current Configurations for Restoration for those instructions.
Enable the incremental save flag on the switch.
Enter the Global Configuration mode:
SEFOS# configure terminal
Enable the incremental save flag on the switch:
SEFOS(config)# incremental-save enable
Disable the incremental save flag on the switch:
SEFOS(config)# incremental-save disable
Exit from the Global Configuration mode:
SEFOS(config)# end

Enabling or Disabling the Auto Save Flag


This procedure describes how to enable or disable updates of the runtime configuration in the configuration file.
Enable or disable the auto save flag on the switch.
Enter the Global Configuration mode:
SEFOS# configure terminal
Enable the auto save flag on the switch:
SEFOS(config)# auto-save trigger enable
Disable the auto save flag on the switch:
SEFOS(config)# auto-save trigger disable
Exit from the Global Configuration mode:
SEFOS(config)# end

Configuring the Default VLAN Identifier


This procedure describes how to write the VLAN ID to the NVRAM. This ID will be used as the default VLAN ID
when the switch is restarted.
1. Configure the default VLAN identifier.
Enter the Global Configuration mode:
SEFOS# configure terminal
19

Configure the default VLAN ID as 10:


SEFOS(config)# default vlan id 10
Exit from the Global Configuration mode:
SEFOS(config)# end
2. View the default VLAN ID:
SEFOS# show nvram
Default IP Address

: 10.0.0.1

Default Subnet Mask

: 255.0.0.0

Default IP Address Config Mode

: Manual

Default IP Address Allocation Protocol

: DHCP

Switch Base MAC Address

: 00:14:4f:6c:63:0f

Default Interface Name

: Ex0/1

Default RM Interface Name

: NONE

Config Restore Option

: No restore

Config Save Option

: No save

Auto Save

: Enable

Incremental Save

: Enable

Roll Back

: Enable

Config Save IP Address

: 0.0.0.0

Config Save Filename

: switch.conf

Config Restore Filename

: myconfig.conf

PIM Mode

: Sparse Mode

IGS Forwarding Mode

: MAC based

Cli Serial Console

: No

SNMP EngineID

: 80.00.08.1c.04.46.53

SNMP Engine Boots

: 16

Default VLAN Identifier

: 10

Stack PortCount

: 0

ColdStandby

: Disable

NPAPI Mode

: Synchronous

Dynamic Port Count

: 24

You should not change the default VLAN ID when some configurations are already
saved.
After you have configured the default VLAN ID, you must restart the switch before
saving any configuration.
20 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Enabling or Disabling Trap Generation on an Interface


This procedure describes how to enable or disable trap generation either on the physical interface or on the portchannel interface.
1. Disable the SNMP trap on the interface extreme-ethernet 0/1.
Enter the Global Configuration mode:
SEFOS# configure terminal
Enter the Interface Configuration mode for extreme-ethernet 0/1:
SEFOS(config)# interface extreme-ethernet 0/1
Disable the trap on the interface:
SEFOS(config-if)# no snmp trap link-status
Exit from configuration mode:
SEFOS(config-if)# end
2. View the trap state for the interface extreme-ethernet 0/1:
SEFOS# show interface extreme-ethernet 0/1
Ex0/1 up, line protocol is up (connected)
Bridge Port Type: Customer Bridge Port
Hardware Address is 00:08:02:03:04:01
MTU

9216 bytes, Full duplex, 10 Gbps,

No-Negotiation

HOL Block Prevention enabled.


Input flow-control is on, output flow-control is on
Link Up/Down Trap is disabled
Reception Counters
Octets

: 26635006

Unicast Packets

: 3064

Discarded Packets

: 0

Error Packets

: 0

Unknown Protocol

: 0

Transmission Counters
Octets

: 3725169

Unicast Packets

: 0

Discarded Packets

: 0

Error Packets

: 0

21

If the trap is enabled, then the switch sends trap messages to the SNMP Manager on
specific events such as link up, link down, and so on.

Configuring Debug Logging


This procedure describes how to configure where the debug logs are to be displayed (on the console or to a file).
1. Modify the logging option of debug traces.
Enter the Global Configuration mode:
SEFOS# configure terminal
Log the debug traces to a file:
SEFOS(config)# debug-logging file
Exit from the Global Configuration mode:
SEFOS(config)# end
2. View the logging option:
SEFOS# show system information
Hardware Version

: 5.2.3

Firmware Version

: 4.0.0.0

Switch Name

: SEFOS

System Contact

: info@sun.com

System Location

: Sun Microsystems Inc.

Logging Option

: File Logging

Login Authentication Mode

: Local

Config Save Status

: Not Initiated

Remote Save Status

: Not Initiated

Config Restore Status

: Not Initiated

Configuring ACL Filters


ACL filters are used to filter packets at the hardware based on certain filtering criteria configured or programmed in
the switch. The switch examines each packet to determine if it should be blocked or if it should be forwarded based
on the configured access lists.

22 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

The following example shows how to block the IP traffic from a host with an IP address of 12.0.0.100. See Figure 2
for the topology for this procedure. Port 1 of the switch is connected to the host. Run the following commands on
switch 1. The IP address of Host 1 is assumed to be 12.0.0.100 in this procedure.
1. Configure the IP address of the switch to 12.0.0.1:
SEFOS# configure terminal
SEFOS(config)# interface vlan 1
SEFOS(config-if)# shutdown
SEFOS(config-if)# ip address 12.0.0.1

255.0.0.0

SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS(config)# interface extreme-ethernet 0/1
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS(config)# interface extreme-ethernet 0/2
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
2. Create an IP filter with ID 11:
SEFOS(config)# ip access-list extended 11
The filter type can be extended or standard. Standard filters are used to filter the traffic based on the source IP
address and the destination IP address. Extended filters can also be used to specify the protocol ID, TCP/UDP port
numbers, DSCP values, and flow label. In this example, the IP packets with 12.0.0.100 as the source address will be
filtered.
3. Deny the IP traffic from host 12.0.0.100 to any network or host:
SEFOS(config-ext-nacl)# deny ip host 12.0.0.100 any
SEFOS(config-ext-nacl)# exit
4. Apply the IP filter 11 to port1:
SEFOS(config)# interface extreme-ethernet 0/1
SEFOS(config-if)# ip access-group 11 in
SEFOS(config-if)# exit
5. View the configuration details:
SEFOS# show access-lists
IP ACCESS LISTS
----------------Extended IP Access List 11
-----------------------------

23

Filter Priority

: 1

Filter Protocol Type

: ANY

IP address Type

: IPV4

Source IP address

: 12.0.0.100

Source IP address mask

: 255.255.255.255

Source IP Prefix Length

: 32

Destination IP address

: 0.0.0.0

Destination IP address mask

: 0.0.0.0

Destination IP Prefix Length

: 0

Flow Identifier

: 0

In Port List

: Ex0/1

Out Port List

: NIL

Filter TOS

: NIL

Filter DSCP

: NIL

Filter Action

: Deny

Status

: Active

6. Send the forwarding traffic from a smart bit machine to switch Port 1 with a source IPv4 address of
12.0.0.100.
The packet will not be forwarded to Port 2.
7. Remove the IP filter from port 1:
SEFOS# configure terminal
SEFOS(config)# interface extreme-ethernet 0/1
SEFOS(config-if)# no ip access-group 11 in
SEFOS(config-if)# end
SEFOS# show access-lists
IP ACCESS LISTS
----------------Extended IP Access List 11
----------------------------Filter Priority

: 1

Filter Protocol Type

: ANY

IP address Type

: IPV4

Source IP address

: 12.0.0.1

Source IP address mask

: 255.255.255.255

Source IP Prefix Length

: 32

24 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Destination IP address

: 0.0.0.0

Destination IP address mask

: 0.0.0.0

Destination IP Prefix Length

: 0

Flow Identifier

: 0

In Port List

: NIL

Out Port List

: NIL

Filter TOS

: NIL

Filter DSCP

: NIL

Filter Action

: Deny

Status

: InActive

8. Send the forwarding traffic from a smart bit machine to switch Port 1 with a source IPv4 address of
12.0.0.100.
The packets will be forwarded to Port 2.

Configuring Quality of Services


Packets received at the switch can be classified to a particular class of service based on the filters configured.
Certain policies can be applied to them before forwarding the packet. The following example illustrates the
classification of the TCP packets received in the switch and changing the DSCP value in the IP header of the TCP
packets to 46.
See Figure 2 for the topology for this procedure. Connect Port 1 and Port 2 to the IXIA/SMART BITS.
1. Enable Port 1 and Port 2:
SEFOS# configure terminal
SEFOS(config)# interface extreme-ethernet 0/2
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS(config)# interface extreme-ethernet 0/1
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
2. Enable QoS:
SEFOS# configure terminal
SEFOS(config)# qos enable
SEFOS(config)# end
3. Create an access control list filter for TCP packets:
SEFOS# configure terminal
25

SEFOS(config)# ip access-list extended 11


SEFOS(config-ext-nacl)# permit tcp any any
SEFOS(config-ext-nacl)# exit
SEFOS(config)# interface extreme-ethernet 0/1
SEFOS(config-if)# ip access-group 11 in
SEFOS(config-if)#

end

4. Specify the class mapping for the incoming packet and policy mapping for the classified packet.
Enter the Global Configuration mode:
SEFOS# configure terminal
Create a class map and enter into the Class Map Configuration mode:
SEFOS(config)# class-map 10
Configure the match criteria for the class map with the criteria specified by the access list 11 (already
configured earlier as all tcp packets):
SEFOS(config-cls-map)# match access-group ip-access-list 11
SEFOS(config-cls-map)# set class 100
Exit from the Class Map Configuration mode:
SEFOS(config-cls-map)# exit
Create a policy map to apply on the packet before forwarding it:
SEFOS(config)# policy-map 10
Create a policy map for class 100 packets and set the DSCP value in the IP header for the outgoing
packets matched by this class as 46:
SEFOS(config-ply-map)# set policy class 100 default-priority-type ipdscp 46
SEFOS(config-ply-map)# exit
5. View the configuration details:
SEFOS# show access-lists
IP ACCESS LISTS
----------------Extended IP Access List 11
----------------------------Filter Priority

: 1

Filter Protocol Type

: TCP

IP address Type

: IPV4

Source IP address

: 0.0.0.0

Source IP address mask

: 0.0.0.0

Source IP Prefix Length

: 0

Destination IP address

: 0.0.0.0

26 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Destination IP address mask

: 0.0.0.0

Destination IP Prefix Length

: 0

Flow Identifier

: 0

In Port List

: Ex0/1

Out Port List

: NIL

Filter TOS

: NIL

Filter DSCP

: NIL

Filter Source Ports From

: 1

Filter Source Ports Till

: 65535

Filter Destination Ports From

: 1

Filter Destination Ports Till

: 65535

Filter Action

: Permit

Status

: Active

MAC ACCESS LISTS


----------------No MAC Access Lists have been configured
SEFOS# show class-map 10
QoS Class Map Entries
--------------------ClassMapId

: 10

L2FilterId

: None

L3FilterId

: 11

PriorityMapId

: None

CLASS

: 100

PolicyMapId

: 10

PreColor

: None

Status

: Active

SEFOS# show policy-map 10


QoS Policy Map Entries
---------------------PolicyMapId

: 10

IfIndex

: 0

Class

: 100

DefaultPHB

: IP DSCP 46

27

MeterId

: 0

ConNClass

: 0

ExcNClass

: 0

VioNClass

: 0

ConfAct

: None.

ExcAct

: None.

VioAct

: None.

6. Verify the functionality of the policy configuration by generating 100 TCP packets from IXIA to Port 1.
You will notice that these packets are received at Port 2 and have a DSCP value 46.

Configuring Port Mirroring


Port mirroring is used to monitor the packets of a particular port on another port. The following example shows how
to mirror all incoming packets on Port 1 to Port 2.
See Figure 1 for the topology for this procedure. Port 1 of the switch is connected to the host and Port 2 is connected
to a packet-capturing device to verify that packet is mirrored at Port 2. Run the following commands on switch 1.
The IP address of Host 1 is assumed to be 12.0.0.100 in this procedure.
1. Configure the IP address of the switch as 12.0.0.1:
SEFOS# configure terminal
SEFOS(config)# interface vlan 1
SEFOS(config-if)# shutdown
SEFOS(config-if)# ip address 12.0.0.1

255.0.0.0

SEFOS(config-if)# no shutdown
SEFOS(config-if)# end
2. View the ping between the host and the switch:
SEFOS# ping 12.0.0.100
Reply Received From :12.0.0.100, TimeTaken : 30 msecs
Reply Received From :12.0.0.100, TimeTaken : 30 msecs
Reply Received From :12.0.0.100, TimeTaken : 30 msecs
--- 12.0.0.100 Ping Statistics --3 Packets Transmitted, 3 Packets Received, 0% Packets Loss

28 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

The ping reply from Host 1 to the switch is not captured at Port 2, which is connected to the packet capturing device
prior to the port-mirroring configuration.
3. Run the following commands on the switch to enable mirroring for incoming packets at Port 1 to Port 2.
Enter the Global Configuration mode:
SEFOS# configure terminal
Configure the destination interface for mirroring as Port 2:
SEFOS(config)# monitor session destination interface extreme-ethernet 0/2
Configure the source interface for mirroring as Port 1 and apply mirroring for incoming packets:
SEFOS(config)# monitor session source interface extreme-ethernet 0/1 rx
Exit from the Global Configuration mode:
SEFOS(config-if)# end
View the configuration details:
SEFOS# show port-monitoring
Port Monitoring

is enabled

Monitor Port : Ex0/2


Port

Ingress-Monitoring

Egress-Monitoring

----

------------------

-----------------

Ex0/1

Enabled

Disabled

Ex0/2

Disabled

Disabled

Ex0/3

Disabled

Disabled

Ex0/4

Disabled

Disabled

Ex0/5

Disabled

Disabled

Ex0/6

Disabled

Disabled

Ex0/7

Disabled

Disabled

Ex0/8

Disabled

Disabled

Ex0/9

Disabled

Disabled

Ex0/10

Disabled

Disabled

Ex0/11

Disabled

Disabled

Ex0/12

Disabled

Disabled

Ex0/13

Disabled

Disabled

Ex0/14

Disabled

Disabled

Ex0/15

Disabled

Disabled

Ex0/16

Disabled

Disabled

Ex0/17

Disabled

Disabled

Ex0/18

Disabled

Disabled

29

Ex0/19

Disabled

Disabled

Ex0/20

Disabled

Disabled

Ex0/21

Disabled

Disabled

Ex0/22

Disabled

Disabled

Ex0/23

Disabled

Disabled

Ex0/24

Disabled

Disabled

4. Ping 12.0.0.100 and verify that the ICMP reply was received at Port 2:
SEFOS# ping 12.0.0.100
Reply Received From :12.0.0.100, TimeTaken : 30 msecs
Reply Received From :12.0.0.100, TimeTaken : 30 msecs
Reply Received From :12.0.0.100, TimeTaken : 30 msecs
--- 12.0.0.100 Ping Statistics --3 Packets Transmitted, 3 Packets Received, 0% Packets Loss

5. Verify that the ping reply from Host 1 to the switch is captured at Port 2 using the packet-capturing
device.

Configuring Rate Limiting


SEFOS can be configured to limit the rate of traffic received on a particular interface. If the traffic is above the
configured threshold level, then the packet will get dropped. The following example illustrates the configuration for
limiting a multicast traffic at Port 1 to a rate of 50 packets per second.
See Figure 2 for the topology for this procedure. Port 1 and Port 2 are connected to SMART BITS/IXIA to monitor
the rate of packet forwarding. Configure the rate limiting for multicast packets on Port 1 as 50 packets per second
and then generate a multicast traffic from IXIA at the rate of 1 Mbps. Packets received at Port 2 must be at the rate
of 50 packets per second.
1. Run the following commands in the switch to enable Port 1 and Port 2:
SEFOS# configure terminal
SEFOS(config)# interface extreme-ethernet 0/1
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS(config)# interface extreme-ethernet 0/2
SEFOS(config-if)# no shutdown
SEFOS(config-if)# end
2. Set the rate limit to 50 packets per second.
Enter the Global Configuration mode:

30 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

SEFOS# configure terminal


Enter the Interface Configuration mode:
SEFOS(config)# interface extreme-ethernet 0/1
Set the rate limit to 50 packets per second:
SEFOS(config-if)# storm-control multicast level 50
Exit from Global Configuration mode:
SEFOS(config-if)# end
3. View the configuration details:
SEFOS# show interface extreme-ethernet 0/1 storm-control
Ex0/1
DLF Storm Control

: Disabled

Broadcast Storm Control

: Disabled

Multicast Storm Control

: Enabled

Multicast Storm Control

: 50

4. View the functionality by generating multicast packet from IXIA at the rate of 1 Mbps to Port 1.
You will notice that the packets received at the second port of IXIA are at the rate of 50 packets per second only.

31

Chapter

3
Sample Configurations
This chapter provides sample configurations for the Layer 2 switching features used in the Sun Blade 6000 Ethernet
Switched NEM 24p 10GbE from Oracle using the Sun Ethernet Fabric Operating System (SEFOS).
Topology Switching Feature
Getting Started
Configuring VLAN Forwarding
Rapid Spanning Tree Protocol
Configuring Link Aggregation
Saving Configurations
Topology - Routing Feature
Configuring Static Unicast Route Entries
Configuring Dynamic Routing - RIP
Configuring Dynamic Routing - OSPF
Topology SFP+ LAG and Backplane Portface Connection
Configuring Basic SFP+ LAG and Backplane
Topology Layer 2 Switch LAG and QSFP Splitter SAN Connection
Configuring a Layer 2 Switch LAG and QSFP Splitter SAN Connection
Topology Switched NEM High Availability Connection
Setting Up a Four NEM/Two Chassis High Availibility Configuration
Topology Four Switched NEMs, High Availability Connection
Setting Up a Four NEM/Four Chassis High Availibility Configuration

32 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Setting Up QSFP Ports Into a Link Aggregation


Checking the Connectivity Status on the NEMs

Topology Switching Feature


The following figure shows a sample topology for the switching feature.

Figure 3. Sample Topology For the Switching Feature

Getting Started
1. Connect to SEFOS.
If you are not already connected to SEFOS, see the instructions in Chapter 1, then return here. You should see the
SEFOS prompt when you are connected to SEFOS:
SEFOS#
2. View the status of the switch interfaces:
SEFOS# show interface status
Port

Status

Duplex

Speed

Negotiation

----

------

------

-----

-----------

Ex0/1

connected

Full

10 Gbps

No-Negotiation

Ex0/2

not connected

Full

10 Gbps

No-Negotiation

Ex0/3

not connected

Full

10 Gbps

No-Negotiation

Ex0/4

not connected

Full

10 Gbps

No-Negotiation

33

Ex0/5

not connected

Full

10 Gbps

No-Negotiation

Ex0/6

not connected

Full

10 Gbps

No-Negotiation

Ex0/7

not connected

Full

10 Gbps

No-Negotiation

Ex0/8

not connected

Full

10 Gbps

No-Negotiation

Ex0/9

not connected

Full

10 Gbps

No-Negotiation

Ex0/10

not connected

Full

10 Gbps

No-Negotiation

Ex0/11

not connected

Full

10 Gbps

No-Negotiation

Ex0/12

not connected

Full

10 Gbps

No-Negotiation

Ex0/13

not connected

Full

10 Gbps

No-Negotiation

Ex0/14

not connected

Full

10 Gbps

No-Negotiation

Ex0/15

connected

Full

10 Gbps

No-Negotiation

Ex0/16

connected

Full

10 Gbps

No-Negotiation

Ex0/17

not connected

Full

10 Gbps

No-Negotiation

Ex0/18

not connected

Full

10 Gbps

No-Negotiation

Ex0/19

not connected

Full

10 Gbps

No-Negotiation

Ex0/20

not connected

Full

10 Gbps

No-Negotiation

Ex0/21

not connected

Full

10 Gbps

No-Negotiation

Ex0/22

not connected

Full

10 Gbps

No-Negotiation

Ex0/23

not connected

Full

10 Gbps

No-Negotiation

Ex0/24

connected

Full

10 Gbps

No-Negotiation

3. View the default IP address:


SEFOS# show ip interface
vlan1 is up, line protocol is up
Internet Address is 10.0.0.1/8
Broadcast Address

10.255.255.255

When the system is booted up, it will come up with a default IP address of 10.0.0.1 for the default VLAN 1.
4. Enter the Global Configuration mode:
SEFOS# configure terminal
SEFOS(config)#
5. Change the default IP address of VLAN 1:
SEFOS(config)# interface vlan 1
SEFOS(config-if)# shutdown
SEFOS(config-if)# ip address 20.0.0.1 255.0.0.0
SEFOS(config-if)# no shutdown

34 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

SEFOS(config-if)# end
6. View the default IP address of VLAN 1:
SEFOS# show ip interface
vlan1 is up, line protocol is up
Internet Address is 20.0.0.1/8
Broadcast Address

20.255.255.255

SEFOS#
7. Ping Switch 1 from Blade Server A.
The ping should be successful.
8. Check the MAC address entry:
SEFOS# show mac-address-table
Vlan

Mac Address

Type

Ports

----

-----------

----

-----

00:1b:21:4a:ab:81

Learnt

Ex0/15

Total Mac Addresses displayed: 1

The MAC address shown should correspond to the MAC address of Blade Server A.
9. Repeat Step 5 on Switch 2 to change the default IP address of VLAN 1 IP to 20.0.0.2.
10. Ping from Blade Server C to Blade Server A or Blade Server B.
The ping should be successful.
11. Configure the topology.
To configure the topology, all the interfaces should be up.
SEFOS# configure terminal
SEFOS(config)# interface extreme-ethernet 0/3
SEFOS(config-if)# no shutdown
SEFOS(config-if)# end
In Switch 1, make the interface extreme-ethernet 0/4, 0/5, 0/6, 0/15, and 0/16 up as shown above. In Switch 2, make
the interface extreme-ethernet 0/3, 0/4, 0/5, 0/6, 0/15, and 0/16 up as shown above.
Following are the expected results from the configurations made to load the Sun Ethernet Fabric Operating
System on to the Sun Blade 6000 Ethernet Switched NEM 24p 10GbE target.
1. Ping from Blade Server A and Blade Server B should be successful to switch 1 (20.0.0.1).
2. Ping from Blade Server C should be successful to switch 2 (20.0.0.2).
35

3. Use a capture packet utility to capture packet examples on Blade Server A, Blade Server B and Blade Server
C for the port connected to the switch. All of these hosts should receive STP, GMRP, GVRP packets continuously.
By default, all the ports on a target will be members of default VLAN 1.

Configuring VLAN Forwarding


This procedure describes how to configure and test VLAN forwarding feature of SEFOS running on a Sun Blade
6000 Ethernet Switched NEM 24p 10GbE target. This procedure tests whether the incoming packets are properly
switched on the member ports of the VLAN.
See Figure 3 for the topology for this procedure.
1. Follow the instructions in the Getting Started section, if you have not already done so.
2. Disable GVRP.
VLAN 1 is the default VLAN. All the ports will be members of the default VLAN.
3. Verify the VLAN membership:
SEFOS# show vlan
Vlan database
------------Vlan ID

: 1

Member Ports

: Ex0/1, Ex0/2, Ex0/3, Ex0/4, Ex0/5, Ex0/6


Ex0/7, Ex0/8, ...

Untagged Ports

: Ex0/1, Ex0/2, Ex0/3, Ex0/4, Ex0/5, Ex0/6


Ex0/7, Ex0/8, ...

Forbidden Ports

: None

Name

Status

: Permanent

4. Ping from Blade Server A to Blade Server B:


# ping 20.0.0.20
PING 20.0.0.20 (20.0.0.20) 56(84) bytes of data.
64 bytes from 20.0.0.20: icmp_seq=1 ttl=255 time=0.384 ms
64 bytes from 20.0.0.20: icmp_seq=2 ttl=255 time=0.325 ms
64 bytes from 20.0.0.20: icmp_seq=3 ttl=255 time=0.283 ms
64 bytes from 20.0.0.20: icmp_seq=4 ttl=255 time=0.255 ms
64 bytes from 20.0.0.20: icmp_seq=5 ttl=255 time=0.224 ms
--- 20.0.0.20 ping statistics --5 packets transmitted, 5 received, 0% packet loss, time 4002ms
rtt min/avg/max/mdev = 0.224/0.294/0.384/0.056 ms

36 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

5. Ping from Blade Server B to Blade Server A:


# ping 20.0.0.10
PING 20.0.0.10 (20.0.0.10) 56(84) bytes of data.
64 bytes from 20.0.0.10: icmp_seq=1 ttl=255 time=0.248 ms
64 bytes from 20.0.0.10: icmp_seq=2 ttl=255 time=0.451 ms
64 bytes from 20.0.0.10: icmp_seq=3 ttl=255 time=0.356 ms
64 bytes from 20.0.0.10: icmp_seq=4 ttl=255 time=0.277 ms
--- 20.0.0.10 ping statistics --4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.248/0.333/0.451/0.078 ms

6. Change the member ports of VLAN 1 on Switch 2:


SEFOS# configure terminal
SEFOS(config)# interface extreme-ethernet 0/3
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS(config)# interface extreme-ethernet 0/4
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS(config)# interface extreme-ethernet 0/5
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS(config)# interface extreme-ethernet 0/6
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
7. Ping from Blade Server C to Blade Server B:
# ping 20.0.0.20
PING 20.0.0.20 (20.0.0.20) 56(84) bytes of data.
64 bytes from 20.0.0.20: icmp_seq=1 ttl=255 time=0.458 ms
64 bytes from 20.0.0.20: icmp_seq=2 ttl=255 time=0.237 ms
64 bytes from 20.0.0.20: icmp_seq=3 ttl=255 time=0.304 ms
64 bytes from 20.0.0.20: icmp_seq=4 ttl=255 time=0.188 ms

37

--- 20.0.0.20 ping statistics --4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.188/0.296/0.458/0.103 ms

Rapid Spanning Tree Protocol


This procedure shows the traffic flow in the Rapid Spanning Tree protocol configuration on SEFOS running on the
Sun Blade 6000 Ethernet Switched NEM 24p 10GbE targets. This procedure verifies whether a loop is present in the
topology, if it is detected and if the traffic is blocked for the alternate port.
See Figure 3 for the topology for this procedure.
1. Follow the instructions in the Getting Started section, if you have not already done so.
2. Configure VLAN forwarding in Switch 1 and Switch 2 using the instructions in the section Configuring
VLAN Forwarding.
3. Run the following command on Switch 1 to verify the port status in RSTP mode:
SEFOS# show spanning-tree
Root Id

Priority

32768

Address

00:14:4f:6c:63:0f

Cost

Port

0 [0]

This bridge is the root


Max age 20 Sec, forward delay 15 Sec
MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id

Priority
Address

32768
00:14:4f:6c:63:0f

Max age is 20 sec, forward delay is 15 sec


Dynamic Path Cost is Disabled
Name

Role

State

Cost

Prio

Type

----

----

-----

----

----

------

Ex0/3

Designated

Forwarding

2000

128

P2P

Ex0/4

Designated

Forwarding

2000

128

P2P

Ex0/5

Designated

Forwarding

2000

128

P2P

Ex0/6

Designated

Forwarding

2000

128

P2P

Ex0/15

Designated

Forwarding

2000

128

P2P

38 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Ex0/16

Designated

Forwarding

2000

128

P2P

Ex0/24

Designated

Forwarding

2000

128

P2P

4. Run the following command on Switch 2 to verify the port status in RSTP mode:
SEFOS# show spanning-tree
Root Id

Priority

32768

Address

00:14:4f:6c:63:0f

Cost

2000

Port

3 [Ex0/3]

Max age 20 Sec, forward delay 15 Sec


MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id

Priority
Address

32768
00:15:ed:09:09:10

Max age is 20 sec, forward delay is 15 sec


Dynamic Path Cost is Disabled
Name

Role

State

Cost

Prio

Type

----

----

-----

----

----

------

Ex0/3

Root

Forwarding

2000

128

P2P

Ex0/4

Alternate

Discarding

2000

128

P2P

Ex0/5

Alternate

Discarding

2000

128

P2P

Ex0/6

Alternate

Discarding

2000

128

P2P

Ex0/17

Designated

Forwarding

2000

128

P2P

Ex0/18

Designated

Forwarding

2000

128

P2P

5. Ping from Blade Server A to Blade Server C:


# ping 20.0.0.30
PING 20.0.0.30 (20.0.0.30) 56(84) bytes of data.
64 bytes from 20.0.0.30: icmp_seq=1 ttl=255 time=0.421 ms
64 bytes from 20.0.0.30: icmp_seq=2 ttl=255 time=0.303 ms
64 bytes from 20.0.0.30: icmp_seq=3 ttl=255 time=0.294 ms
64 bytes from 20.0.0.30: icmp_seq=4 ttl=255 time=0.208 ms

Traffic is forwarded through the extreme-ethernet 0/3 port on Switch 1.


6. Shut down the extreme-ethernet 0/3 port from Switch 2 and verify the port status:
39

SEFOS# configure terminal


SEFOS(config)# int ext 0/3; shut; exit
SEFOS(config)# exit
SEFOS# show spanning-tree
Root Id

Priority

32768

Address

00:14:4f:6c:63:0f

Cost

2000

Port

4 [Ex0/4]

Max age 20 Sec, forward delay 15 Sec


MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id

Priority
Address

32768
00:15:ed:09:09:10

Max age is 20 sec, forward delay is 15 sec


Dynamic Path Cost is Disabled
Name

Role

State

Cost

Prio

Type

----

----

-----

----

----

------

Ex0/4

Root

Forwarding

2000

128

P2P

Ex0/5

Alternate

Discarding

2000

128

P2P

Ex0/6

Alternate

Discarding

2000

128

P2P

Ex0/17

Designated

Forwarding

2000

128

P2P

Ex0/18

Designated

Forwarding

2000

128

P2P

7. Ping from Blade Server A to Blade Server C:


# ping 20.0.0.30
Reply Received From :20.0.0.30, TimeTaken : 10 msecs
Reply Received From :20.0.0.30, TimeTaken : 10 msecs
Reply Received From :20.0.0.30, TimeTaken : 10 msecs

Traffic should be forwarded through the extreme-ethernet 0/4 port on Switch 1. If the Rapid Spanning Tree protocol
is working as expected, there will be a change in the port state. This procedure shows that if the root port in the
RSTP mode goes down for any reason, an alternate port in the RSTP will forward the traffic.

40 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Configuring Link Aggregation


This procedure describes how to configure and test the Link Aggregation feature of SEFOS running on a Sun Blade
6000 Ethernet Switched NEM 24p 10GbE target.
See Figure 3 for the topology for this procedure.
1. Follow the instructions in the Getting Started section, if you have not already done so.
2. Create a port-channel group 100 on Switch 1 and link the ports in the group:
SEFOS# configure terminal
SEFOS(config)# set port-channel enable
SEFOS(config)# interface port-channel 100
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS(config)# interface extreme-ethernet 0/3
SEFOS(config-if)# channel-group 100 mode active
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS(config)# interface extreme-ethernet 0/4
SEFOS(config-if)# channel-group 100 mode active
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS(config)# interface extreme-ethernet 0/5
SEFOS(config-if)# channel-group 100 mode active
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
SEFOS(config)# interface extreme-ethernet 0/6
SEFOS(config-if)# channel-group 100 mode active
SEFOS(config-if)# no shutdown
SEFOS(config-if)# exit
3. Repeat the previous step on Switch 2.
4. Verify the VLAN membership:
SEFOS# show vlan
Vlan ID

: 1

Member Ports

: Ex0/1, Ex0/2, Ex0/7, Ex0/8, Ex0/9, Ex0/10


Ex0/11, Ex0/12, Ex0/13, Ex0/14, Ex0/15, Ex0/16
Ex0/17, Ex0/18, Ex0/19, Ex0/20, Ex0/21, Ex0/22

41

Ex0/23, Ex0/24, po100


Untagged Ports

: Ex0/1, Ex0/2, Ex0/7, Ex0/8, Ex0/9, Ex0/10


Ex0/11, Ex0/12, Ex0/13, Ex0/14, Ex0/15, Ex0/16
Ex0/17, Ex0/18, Ex0/19, Ex0/20, Ex0/21, Ex0/22
Ex0/23, Ex0/24, po100

Forbidden Ports

: None

Name

Status

: Permanent

5. Verify the STP port status on Switch 1:


SEFOS# show spanning-tree
Root Id

Priority

32768

Address

00:14:4f:6c:63:0f

Cost

Port

0 [0]

This bridge is the root


Max age 20 Sec, forward delay 15 Sec
MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id

Priority
Address

32768
00:14:4f:6c:63:0f

Max age is 20 sec, forward delay is 15 sec


Dynamic Path Cost is Disabled
Name

Role

State

Cost

Prio

Type

----

----

-----

----

----

------

Ex0/15

Designated

Forwarding

2000

128

P2P

Ex0/16

Designated

Forwarding

2000

128

P2P

po100

Designated

Discarding

1900

128

P2P

6. Ping continuously from Blade Server A to Blade Server C.


There should not be any data loss during traffic forwarding.
7. Verify the port channel summary:
SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled

42 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Port-channel Module Oper Status is enabled


Port-channel System Identifier is

00:14:4f:6c:63:0f

Flags:
D - down

P - in port-channel

I - stand-alone

H - Hot-standby (LACP only)

Number of channel-groups in use: 1


Number of aggregators: 1
Group

Port-channel

Protocol

Ports

------------------------------------------------------------------------100

Po100(P)

LACP

Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)

8. Shut down the port extreme-ethernet 0/3 and verify the port channel summary:
SEFOS# configure terminal
SEFOS(config)# int ext 0/3
SEFOS(config-if)# shut
SEFOS(config-if)# end
SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is

00:14:4f:6c:63:0f

Flags:
D - down

P - in port-channel

I - stand-alone

H - Hot-standby (LACP only)

Number of channel-groups in use: 1


Number of aggregators: 1
Group

Port-channel

Protocol

Ports

------------------------------------------------------------------------100

Po100(P)

LACP

Ex0/3(D),Ex0/4(P),Ex0/5(P),Ex0/6(P)

If traffic flows continuously through the port channel group, then Link Aggregation was configured properly.

43

Saving Configurations
This procedure describes how to save configurations even after restarting the switch.
See Figure 3 for the topology for this procedure.
1. Follow the instructions in the Getting Started section, if you have not already done so.
2. Save the configurations for Switch 1 and Switch 2:
SEFOS# write startup-config
This may take a few seconds. You will see the message [OK] when the configurations have been successfully
saved.
3. Exit the SEFOS session:
SEFOS# exit
Connection closed by foreign host.
cd: The session with /NEM/fs_cli has ended.

4. At the ILOM prompt, change directories to the /SP directory:


-> cd /SP
5. Reset the service processor:
-> reset
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP
Broadcast message from root (Wed Feb 11 23:13:07 1970):
The system is going down for reboot NOW!

6. Log in when the system comes back online.


The default password is changeme.
[root]# ssh my-nem -l root
root@my-nem's password:
Sun(TM) Integrated Lights Out Manager (Fabric Component Edition)
Version 3.0.0.0
Copyright 2009 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Warning: password is set to factory default.

7. Connect to the Sun Ethernet Fabric Operating System again:


-> cd /NEM/fs_cli
44 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

cd: Connecting to Fabric Switch CLI as user fs_cli_usr

8. Verify the port channel summary:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is

00:14:4f:6c:63:0f

Flags:
D - down

P - in port-channel

I - stand-alone

H - Hot-standby (LACP only)

Number of channel-groups in use: 1


Number of aggregators: 1
Group

Port-channel

Protocol

Ports

------------------------------------------------------------------------100

Po100(P)

LACP

Ex0/3(D),Ex0/4(P),Ex0/5(P),Ex0/6(P)

Once Switch 1 and Switch 2 are both restarted, the Sun Ethernet Fabric Operating System will come up with all the
previous configurations.

45

Topology - Routing Feature


The following figure shows a sample topology for the routing feature.

Figure 4. Sample Topology For the Routing Feature

Configuring Static Unicast Route Entries


This procedure describes how to configure the static route entries in SEFOS running on switched NEMs. This
procedure will verify that the traffic between hosts, which are on two different networks, is reachable by configuring
the required static route in SEFOS.
See Figure 4 for the topology for this procedure.
1. Follow the instructions in the Getting Started section, if you have not already done so.
2. Configure Switch 1 based on the topology shown in Figure 4:
SEFOS# configure terminal
SEFOS(config)# set gvrp disable
SEFOS(config)# vlan 10
SEFOS(config-vlan)# exit
SEFOS(config)# interface range extreme-ethernet 0/1-2
SEFOS(config-if-range)# switchport access vlan 10
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface vlan 10
SEFOS(config-if)# shutdown

46 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

SEFOS(config-if)# ip address 10.0.0.201 255.255.255.0


SEFOS(config-if)# no shutdown
SEFOS(config-if)# end
3. Ping from Blade Server C to Blade Server A:
SEFOS# ping 10.0.0.169
Reply
Reply
Reply

Received
Received
Received

From
From
From

:10.0.0.169,
:10.0.0.169,
:10.0.0.169,

TimeTaken
TimeTaken
TimeTaken

:
:
:

20
20
20

--10.0.0.169
Ping
Statistics
3 Packets Transmitted, 3 Packets Received, 0% Packets Loss

msecs
msecs
msecs
---

4. Ping from Blade Server A to Blade Server C:


SEFOS# ping 10.0.0.152
Reply
Received
From
:10.0.0.152,
TimeTaken
Reply
Received
From
:10.0.0.152,
TimeTaken
Reply Received From :10.0.0.152, TimeTaken : 20 msecs

:
:

20
20

msecs
msecs

5. Configure Switch 2 based on the topology shown in Figure 4:


SEFOS# configure terminal
SEFOS(config)# set gvrp disable
SEFOS(config)# interface vlan 1
SEFOS(config-if)# no ip address
SEFOS(config-if)# exit
SEFOS(config)# vlan 40
SEFOS(config-vlan)# exit
SEFOS(config)# interface range extreme-ethernet 0/1-2
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# switchport access vlan 40
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface vlan 40
SEFOS(config-if)# shutdown
SEFOS(config-if)# ip address 40.0.0.200 255.255.255.0
SEFOS(config-if)# no shutdown
SEFOS(config-if)# end
SEFOS# ping 40.0.0.110

47

Reply Received From :40.0.0.110, TimeTaken : 20 msecs


Reply Received From :40.0.0.110, TimeTaken : 20 msecs
Reply Received From :40.0.0.110, TimeTaken : 20 msecs
--- 40.0.0.110 Ping Statistics --3 Packets Transmitted, 3 Packets Received, 0% Packets Loss

6. Create a 10GbE link between the two switches. Use one of the P3 10GbE links from the first QSFP ports
as the interconnect. Also, create and configure the VLAN interfaces on the connection link.
Run the following commands on Switch 1:
SEFOS(config)# interface range extreme-ethernet 0/3-6
SEFOS(config-if-range)# switchport access vlan 20
You should see the following message:
% Access VLAN does not exist. Creating vlan

SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface vlan 20
SEFOS(config-if)# shutdown
SEFOS(config-if)# ip address 20.0.0.201 255.255.255.0
SEFOS(config-if)# no shutdown
SEFOS(config-if)# end
SEFOS#
Run the following commands on Switch 2:
SEFOS(config)# interface range extreme-ethernet 0/3-6
SEFOS(config-if-range)# switchport access vlan 20
You should see the following message:
% Access VLAN does not exist. Creating vlan

SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface vlan 20
SEFOS(config-if)# shutdown
SEFOS(config-if)# ip address 20.0.0.200 255.255.255.0
SEFOS(config-if)# no shutdown
SEFOS(config-if)# end
SEFOS#
You can now use this configuration as a starting point for different router configurations.
7. Save the configuration information so that the switch will start with the saved configuration on reboot.
48 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Run the following command on Switch 1:


SEFOS# write startup-config
Building configuration
[OK]

Run the following command on Switch 2:


SEFOS# write startup-config
Building configuration
[OK]

8. Verify that the existing routes are available in Switch 1.


Run the following command on Switch 1:
SEFOS# show ip route
Codes: C - connected, S - static, R - rip, B - bgp, O - ospf
C 10.0.0.0/24 is directly connected, vlan10
C 20.0.0.0/24 is directly connected, vlan20

Run the following command on Switch 2:


SEFOS# show ip route
Codes: C - connected, S - static, R - rip, B - bgp, O - ospf
C 20.0.0.0/24 is directly connected, vlan20
C 40.0.0.0/24 is directly connected, vlan40

If there is no known route from Switch 1 to Blade Server B, the ping from Blade Server A
to Blade Server B will fail.

9. Configure the static route in Switch 1:


SEFOS# configure terminal
SEFOS(config)# ip route 40.0.0.0 255.255.255.0 20.0.0.200
SEFOS(config)# end
10. Configure the static route in Switch 2:
SEFOS# configure terminal
SEFOS(config)# ip route 10.0.0.0 255.255.255.0 20.0.0.201
SEFOS(config)# end

49

11. Verify that the routes are known to Switch 1 and Switch 2.
Run the following command on Switch 1:
SEFOS# show ip route
Codes: C - connected, S - static, R - rip, B - bgp, O - ospf
C 10.0.0.0/24 is directly connected, vlan10
C 20.0.0.0/24 is directly connected, vlan20
S 40.0.0.0/24 [-1] via 20.0.0.200

Run the following command on Switch 2:


SEFOS# show ip route
Codes: C - connected, S - static, R - rip, B - bgp, O - ospf
S 10.0.0.0/24 [-1] via 20.0.0.201
C 20.0.0.0/24 is directly connected, vlan20
C 40.0.0.0/24 is directly connected, vlan40 iss# show ip route

12. Ping from Blade Server A to Blade Server B.


Run the following command on Switch 1:
SEFOS# ping 40.0.0.200
Reply Received From :40.0.0.200, TimeTaken : 20 msecs
Reply Received From :40.0.0.200, TimeTaken : 20 msecs
Reply Received From :40.0.0.200, TimeTaken : 20 msecs
--- 40.0.0.200 Ping Statistics --3 Packets Transmitted, 3 Packets Received, 0% Packets Loss

Run the following command on Switch 2:


SEFOS# ping 10.0.0.201
Reply Received From :10.0.0.201, TimeTaken : 30 msecs
Reply Received From :10.0.0.201, TimeTaken : 20 msecs
Reply Received From :10.0.0.201, TimeTaken : 20 msecs
--- 10.0.0.201 Ping Statistics --3 Packets Transmitted, 3 Packets Received, 0% Packets Loss

Add static routes on Blade Server A:


# route add -net 20.0.0.0 netmask 255.255.255.0 gw 10.0.0.201
# route add -net 40.0.0.0 netmask 255.255.255.0 gw 10.0.0.201
Ping the VLAN 20 interface on Switch 1:
# ping 20.0.0.201
PING 20.0.0.201 (20.0.0.201) 56(84) bytes of data.
64 bytes from 20.0.0.201: icmp_seq=0 ttl=64 time=0.910 ms

50 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

64 bytes from 20.0.0.201: icmp_seq=1 ttl=64 time=0.722 ms


--- 20.0.0.201 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.722/0.816/0.910/0.094 ms, pipe 2

Ping the VLAN 20 interface on Switch 2:


# ping 20.0.0.200
PING 20.0.0.200 (20.0.0.200) 56(84) bytes of data.
64 bytes from 20.0.0.200: icmp_seq=1 ttl=63 time=0.693 ms
64 bytes from 20.0.0.200: icmp_seq=2 ttl=63 time=0.693 ms
64 bytes from 20.0.0.200: icmp_seq=3 ttl=63 time=0.821 ms
--- 20.0.0.200 ping statistics --4 packets transmitted, 3 received, 25% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.693/0.735/0.821/0.067 ms, pipe 2

Ping the VLAN 40 interface on Switch 2:


# ping 40.0.0.200
PING 40.0.0.200 (40.0.0.200) 56(84) bytes of data.
64 bytes from 40.0.0.200: icmp_seq=0 ttl=63 time=0.881 ms
64 bytes from 40.0.0.200: icmp_seq=1 ttl=63 time=0.774 ms
--- 40.0.0.200 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.774/0.827/0.881/0.060 ms, pipe 2

Add static routes on Blade Server B:


# route add -net 20.0.0.0 netmask 255.255.255.0 gw 40.0.0.200
# route add -net 10.0.0.0 netmask 255.255.255.0 gw 40.0.0.200
Now that Blade Server A is able to communicate to the VLAN 40 interface on Switch 2, ping Blade
Server B on Switch 2.
# ping 40.0.0.110
PING 40.0.0.110 (40.0.0.110) 56(84) bytes of data.
64 bytes from 40.0.0.110: icmp_seq=0 ttl=62 time=0.100
64 bytes from 40.0.0.110: icmp_seq=1 ttl=62 time=0.085
64 bytes from 40.0.0.110: icmp_seq=2 ttl=62 time=0.215
64 bytes from 40.0.0.110: icmp_seq=3 ttl=62 time=0.127

ms
ms
ms
ms

From Blade Server B, ping the VLAN 20 interface on Switch 2:


# ping 20.0.0.200
PING 20.0.0.200 (20.0.0.200) 56(84) bytes of data.
64 bytes from 20.0.0.200: icmp_seq=1 ttl=64 time=6.23 ms
64 bytes from 20.0.0.200: icmp_seq=2 ttl=64 time=0.638 ms

51

--- 20.0.0.200 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 1004ms
rtt min/avg/max/mdev = 0.638/3.434/6.230/2.796 ms

From Blade Server B, ping the VLAN 20 interface on Switch 1:


# ping 20.0.0.201
PING 20.0.0.201 (20.0.0.201) 56(84) bytes of data.
64 bytes from 20.0.0.201: icmp_seq=1 ttl=63 time=0.597 ms
64 bytes from 20.0.0.201: icmp_seq=2 ttl=63 time=0.613 ms
--- 20.0.0.201 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.597/0.605/0.613/0.008 ms

From Blade Server B, ping the VLAN 10 interface on Switch 1:


# ping 10.0.0.201
PING 10.0.0.201 (10.0.0.201) 56(84) bytes of data.
64 bytes from 10.0.0.201: icmp_seq=1 ttl=63 time=0.648 ms
64 bytes from 10.0.0.201: icmp_seq=2 ttl=63 time=0.697 ms
--- 10.0.0.201 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.648/0.672/0.697/0.035 ms

From Blade Server B, ping Blade Server A:


# ping 10.0.0.169
PING 10.0.0.169 (10.0.0.169) 56(84) bytes of data.
64 bytes from 10.0.0.169: icmp_seq=1 ttl=62 time=0.176
64 bytes from 10.0.0.169: icmp_seq=2 ttl=62 time=0.173
64 bytes from 10.0.0.169: icmp_seq=3 ttl=62 time=0.157
64 bytes from 10.0.0.169: icmp_seq=4 ttl=62 time=0.074

ms
ms
ms
ms

To reach Blade Server A from Blade Server B, packets should be routed between VLAN 40 and VLAN 20, and
between VLAN 20 and VLAN 10. You can now test the basic routing functionality of SEFOS now that the static
routes have been configured.
Follow these instructions to remove the static routes on Switch 1 and Switch 2:
1. Remove the static route on Switch 1:
SEFOS# configure terminal
SEFOS(config)# no ip route 40.0.0.0 255.0.0.0 20.0.0.200
SEFOS(config)# end
2. Remove the static route on Switch 2:
SEFOS# configure terminal

52 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

SEFOS(config)# no ip route 10.0.0.0 255.0.0.0 20.0.0.201


SEFOS(config)# end

Configuring Dynamic Routing - RIP


This procedure describes how to configure dynamic routing using the redistribution feature of RIP protocol in
SEFOS. This procedure verifies that accessibility between the two hosts are established after learning the inbetween
routes dynamically through the RIP protocol.
See Figure 4 for the topology for this procedure.
Enable and configure RIP on NEM 1 and NEM 2.
1. Verify that Switch 1 and Switch 2 still have the configuration information that you saved in the section
Configuring Static Unicast Route Entries.
The basic VLAN configuration for dynamic routing is similar to the basic VLAN configuration for static routing. If
the configuration information was not saved on Switch 1 and Switch 2, follow the procedures in Steps 1 6 in
Configuring Static Unicast Route Entries, then return here.
2. Enable RIP protocol on Switch 1:
SEFOS# configure terminal
SEFOS(config)# router rip
SEFOS(config-router)# neighbor 20.0.0.200
SEFOS(config-router)# network 20.0.0.201
SEFOS(config-router)# network 10.0.0.201
SEFOS(config-router)# redistribute all
SEFOS(config-router)# end
3. Enable RIP protocol on Switch 2 using the following commands. If Switch 2 is a third-party switch, verify
that RIP protocol is enabled on Switch 2 and that related configurations are applied accordingly.
SEFOS# configure terminal
SEFOS(config)# router rip
SEFOS(config-router)#

neighbor 20.0.0.201

SEFOS(config-router)# network 20.0.0.200


SEFOS(config-router)# network 40.0.0.200
SEFOS(config-router)# redistribute all
SEFOS(config-router)# end

4. Check the routing database on Switch 1:


SEFOS# show ip rip database

53

Vrf default
10.0.0.0/8
10.0.0.0/24
20.0.0.0/8
20.0.0.0/24
40.0.0.0/8
40.0.0.0/8

[1]
[1]
[1]
[1]
[2]
[2]

auto-summary
directly connected, vlan10
auto-summary
directly connected, vlan20
auto-summary
via 20.0.0.200, vlan20

5. Check the routing database on Switch 2:


SEFOS# show ip rip database
Vrf default
10.0.0.0/8
10.0.0.0/8
20.0.0.0/8
20.0.0.0/24
40.0.0.0/8
40.0.0.0/24

[2]
[2]
[1]
[1]
[1]
[1]

auto-summary
via 20.0.0.201, vlan20
auto-summary
directly connected, vlan20
auto-summary
directly connected, vlan40

6. Ping the VLAN interface on VLAN 10 on Switch 1 from Blade Server B:


# ping 10.0.0.201
PING 10.0.0.201 (10.0.0.201) 56(84) bytes of data.
64 bytes from 10.0.0.201: icmp_seq=1 ttl=63 time=1.64 ms
....

7. Ping Blade Server A from Blade Server B:


# ping 10.0.0.169
PING 10.0.0.169 (10.0.0.169) 56(84) bytes of data.
64 bytes from 10.0.0.169: icmp_seq=1 ttl=62 time=2.35 ms

8. Ping Blade Server B from Blade Server A:


# ping 40.0.0.110
PING 40.0.0.110 (40.0.0.110) 56(84) bytes of data.
64 bytes from 40.0.0.110: icmp_seq=1 ttl=62 time=3.41
64 bytes from 40.0.0.110: icmp_seq=2 ttl=62 time=2.32
64 bytes from 40.0.0.110: icmp_seq=3 ttl=62 time=2.10
64 bytes from 40.0.0.110: icmp_seq=4 ttl=62 time=2.10

ms
ms
ms
ms

You can now test the basic routing functionality of SEFOS now that SEFOS is able to dynamically learn the routing
entries using the redistribution feature of RIP.
Follow these instructions to disable RIP on Switch 1 and Switch 2:
1. Disable RIP on Switch 1:
SEFOS# configure terminal
SEFPS(config)# no router rip
54 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

SEFOS(config)# end
2. Disable RIP on Switch 2.
If Switch 2 is a third-party switch, follow the instructions that came with the switch to disable RIP. Otherwise,
follow the instructions given above to disable RIP on Switch 2.

Configuring Dynamic Routing - OSPF


This procedure describes how to configure dynamic routing using the redistribution feature of the OSPF protocol in
SEFOS. This procedure verifies that the reachability between two hosts is established after learning the inbetween
routes dynamically through the OSPF protocol.
See Figure 4 for the topology for this procedure.
Enable and configure RIP on NEM 1 and NEM 2.
1. Verify that Switch 1 and Switch 2 still have the configuration information that you saved in the section
Configuring Static Unicast Route Entries.
The basic VLAN configuration for dynamic routing is similar to the basic VLAN configuration for static routing. If
the configuration information was not saved on Switch 1 and Switch 2, follow the procedures in Steps 1 6 in
Configuring Static Unicast Route Entries, then return here. If Switch 1 or Switch 2 is a third-party switch, configure
the basic VLAN and the dynamic routing as recommended by the switch's manufacturer.
2. Enable the OSPF protocol on Switch 1:
SEFOS# configure terminal
SEFOS(config)# router ospf
SEFOS(config-router)# asBR router
SEFOS(config-router)# network

20.0.0.201 area 0.0.0.0

SEFOS(config-router)# router-id 20.0.0.201


SEFOS(config-router)# network

10.0.0.201 area 0.0.0.0

SEFOS(config-router)# router-id 10.0.0.201


SEFOS(config-router)# redistribute all
SEFOS(config-router)# end
3. Enable the OSPF protocol on Switch 2 using the following commands. If Switch 2 is a third-party switch,
verify that the OSPF protocol is enabled on Switch 2 and that related configurations are applied
accordingly.
SEFOS# configure terminal
SEFOS(config)# router ospf
SEFOS(config-router)# asBR router
SEFOS(config-router)# network 20.0.0.200 area 0.0.0.0
SEFOS(config-router)# router-id 20.0.0.200
SEFOS(config-router)# network 40.0.0.200 area 0.0.0.0
55

SEFOS(config-router)# router-id 40.0.0.200


SEFOS(config-router)# redistribute all
SEFOS(config-router)# end
4. Check the neighbor router status of Switch 1:
SEFOS# show ip ospf neighbor
Vrf

default

Neighbor-ID Pri
----------- --40.0.0.200 1

State
----FULL/DR

DeadTime
39

Address
Interface
-----------------------20.0.0.200 vlan20

SEFOS# show ip route


Codes: C - connected, S - static, R - rip, B - bgp, O ospf
C 10.0.0.0/24 is directly connected, vlan10
C 20.0.0.0/24 is directly connected, vlan20
O 40.0.0.0/24 [2] via 20.0.0.200

5. Verify the neighbor router status of Switch 2 using the following commands. If Switch 2 is a third-party
switch, use appropriate command to verify the neighbor router status.
SEFOS# show ip ospf neighbor
Vrf default
Neighbor-ID Pri
----------- --10.0.0.201
1

State
----FULL/BACKUP

DeadTime
-------32

Address Interface
------- --------20.0.0.201 vlan20

SEFOS# show ip route


Codes: C - connected, S - static, R - rip, B - bgp, O - ospf
O 10.0.0.0/24 [2] via 20.0.0.201
C 20.0.0.0/24 is directly connected, vlan20
C 40.0.0.0/24 is directly connected, vlan40

The connectivity between Blade Server A and Blade Server B can be verified using ping. Follow these procedures to
disable OSPF on Switch 1 and Switch 2:
1. Disable OSPF on Switch 1:
SEFOS# configure terminal
SEFOS(config)# no router ospf
SEFOS(config)# end

56 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

2. Disable OSPF on Swich 2.


If Switch 2 is a third-party switch, follow the instructions that came with the switch to disable OSPF. Otherwise,
follow the instructions given above to disable OSPF on Switch 2.

Topology SFP+ LAG and Backplane Portface Connection


The following figure shows a sample topology for the SFP+ LAG and backplane portface connection.

Figure 5. Sample Topology For the SFP+ LAG and Backplane Portface Connection

Configuring Basic SFP+ LAG and Backplane


This procedure describes how to provide SFP+ two-port 10GbE trunk uplinks using a single Sun Blade 6000
Ethernet Switched NEM 24p 10GbE installed in a Sun Blade 6000 Modular System with up to ten blade servers.
The blade server ports (backplane ports 15-24) will use portfast spanning tree for fast convergence. Note that you
can also disable the spanning tree on these backplane ports.
See Figure 5 for the topology for this procedure.
1. Enable a trunk on SFP+ ports P1 and P2 on the Sun Blade 6000 Ethernet Switched NEM 24p 10GbE.
In this procedure, the trunk is connected to another layer 2 switch, labeled RSW1. A remote server is also connected
to the RSW1 layer 2 switch.
SEFOS# configure terminal
SEFOS(config)# set port-channel enable
SEFOS(config)# interface port-channel 12
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/1-2
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# channel-group 12 mode active
57

SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# end
SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:67:0f
Flags:
D down

P - in port-channel

I stand-alone

H - Hot-standby (LACP only)

Number of channel-groups in use: 1


Number of aggregators: 1
Group Port-channel

Protocol

Ports

------------------------------------------------------------------------12

Po12(P)

LACP

Ex0/1(P),Ex0/2(P)

SEFOS# configure terminal


SEFOS(config)# interface range extreme-ethernet 0/15-24
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# spanning-tree portfast
Warning: portfast should only be enabled on ports connected to a single host.
Connecting hubs, concentrators,switches, bridges, etc... to this interface when
portfast is enabled can cause temporary bridging loops.
Use with CAUTION

SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# end
SEFOS# show spanning-tree
Root Id

Priority

32768

Address

00:14:4f:6c:63:0f

Cost

1900

Port

25 [po12]

Max age 20 Sec, forward delay 15 Sec

MST00

58 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Spanning tree Protocol has been enabled


MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id

Priority

32768

Address

00:14:4f:6c:67:0f

Max age is 20 sec, forward delay is 15 sec


Dynamic Path Cost is Disabled
Name

Role

State

Cost

Prio

Type

----

----

-----

----

----

------

Ex0/15

Designated Forwarding 2000

128

P2P

Ex0/17

Designated Forwarding 2000

128

P2P

Ex0/19

Designated Forwarding 2000

128

P2P

Ex0/24

Designated Forwarding 2000

128

P2P

po12

Root

128

P2P

Forwarding 1900

2. Disable spanning tree on these ports, if necessary.


SEFOS# configure terminal
SEFOS(config)# interface range extreme-ethernet 0/15-24
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# spanning-tree disable
SEFOS(config-if-range)# end
SEFOS# show spanning-tree
Root Id

Priority

32768

Address

00:14:4f:6c:63:0f

Cost

1900

Port

25 [po12]

Max age 20 Sec, forward delay 15 Sec


MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id

Priority

32768

Address

00:14:4f:6c:67:0f

Max age is 20 sec, forward delay is 15 sec

59

Dynamic Path Cost is Disabled


Name

Role

State

Cost

Prio

Type

----

----

-----

----

----

------

Ex0/15

Disabled

Forwarding

2000

128

P2P

Ex0/17

Disabled

Forwarding

2000

128

P2P

Ex0/19

Disabled

Forwarding

2000

128

P2P

Ex0/24

Disabled

Forwarding

2000

128

P2P

po12

Root

Forwarding

1900

128

P2P

3. Verify that the server that is connected to the remote switch or router is reachable.
[root@nsn57-20 ~]# ping 10.2.2.184
PING 10.2.2.184 (10.2.2.184) 56(84) bytes of data.
64 bytes from 10.2.2.184: icmp_seq=1 ttl=64 time=0.721 ms
--- 10.2.2.184 ping statistics --1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.721/0.721/0.721/0.000 ms

[root@nsn57-20 ~]# ping 10.2.2.46


PING 10.2.2.46 (10.2.2.46) 56(84) bytes of data.
64 bytes from 10.2.2.46: icmp_seq=1 ttl=64 time=0.441 ms
64 bytes from 10.2.2.46: icmp_seq=2 ttl=64 time=0.251 ms
--- 10.2.2.46 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.251/0.346/0.441/0.095 ms

[root@nsn57-20 ~]# ping 10.2.2.44


PING 10.2.2.44 (10.2.2.44) 56(84) bytes of data.
64 bytes from 10.2.2.44: icmp_seq=1 ttl=64 time=1.24 ms
64 bytes from 10.2.2.44: icmp_seq=2 ttl=64 time=0.303 ms
64 bytes from 10.2.2.44: icmp_seq=3 ttl=64 time=0.381 ms
-- 10.2.2.44 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.303/0.641/1.240/0.425 ms

SEFOS#
60 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

SEFOS# show mac-address-table


Vlan

Mac Address

Type

Ports

----

-----------

----

-----

00:14:4f:6c:63:0f

Learnt

po12

00:14:4f:6c:63:10

Learnt

po12

00:14:4f:6c:63:27

Learnt

po12

00:21:28:0b:25:4f

Learnt

Ex0/17

00:21:28:2d:f5:c7

Learnt

Ex0/15

1 00:21:28:2d:f6:27 Learnt
1

00:21:28:2d:f7:47 Learnt

po12
Ex0/24

Total Mac Addresses displayed: 7

Topology Layer 2 Switch LAG and QSFP Splitter SAN Connection


The following figure shows a sample topology for the Layer 2 switch LAG and QSFP splitter SAN connection.

Figure 6. Sample Topology For the Layer 2 Switch LAG and QSFP Splitter SAN Connection

Configuring a Layer 2 Switch LAG and QSFP Splitter SAN Connection


This procedure describes how to use a single Sun Blade 6000 Ethernet Switched NEM 24p 10GbE installed in a Sun
Blade 6000 Modular System with up to ten blade servers to provide:
SFP+ two-port trunk 10GbE uplinks
One QSFP port with a QSFP to SFP+ splitter connected to four 10GbE storage units

61

The blade server ports (backplane ports 15-24) will use portfast spanning tree for fast convergence. Note that you
can also disable the spanning tree on these backplane ports.
See Figure 6 for the topology for this procedure.
1. Enable a trunk on SFP+ ports P1 and P2 on the Sun Blade 6000 Ethernet Switched NEM 24p 10GbE.
In this procedure, the trunk is connected to another layer 2 switch, labeled RSW1. A remote server is also connected
to the RSW1 layer 2 switch.
SEFOS# configure terminal
SEFOS(config)# set port-channel enable
SEFOS(config)# interface port-channel 12
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/1-2
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# channel-group 12 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# end
SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:67:0f
Flags:
D down

P - in port-channel

I stand-alone

H - Hot-standby (LACP only)

Number of channel-groups in use: 1


Number of aggregators: 1
Group Port-channel

Protocol

Ports

------------------------------------------------------------------------12

Po12(P)

LACP

Ex0/1(P),Ex0/2(P)

2. Enable QSFP ports 11 P14 on the Sun Blade 6000 Ethernet Switched NEM 24p 10GbE.
In this procedure, the cable connected to the QSFP ports 11-14 has the following connections:
A single QSFP connector on one end, connected to the QSFP ports 11-14 on the Sun Blade 6000 Ethernet
Switched NEM 24p 10GbE
On the other end, four SFP+ connectors from the QSFP to SFP+ splitter, onnected to four 10GbE ports on the
server, simulating a storage box

62 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

SEFOS# configure terminal


SEFOS(config)# interface range extreme-ethernet 0/11-14
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# spanning-tree portfast
Warning: portfast should only be enabled on ports connected to a single host.
Connecting hubs, concentrators,switches, bridges, etc... to this interface when
portfast is enabled can cause temporary bridging loops.
Use with CAUTION

SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# end
3. Run the following commands on port 15 of the blade server installed in slot 0 in the Sun Blade 6000
Modular System:
[root@nsn57-28 ~]# ifconfig eth2:2 10.2.2.184/24 up
[root@nsn57-28 ~]# ifconfig eth2:3 10.2.3.184/24 up
[root@nsn57-28 ~]# ifconfig eth2:4 10.2.4.184/24 up
[root@nsn57-28 ~]# ifconfig eth2:5 10.2.5.184/24 up
4. Verify the four 10GbE connections on the server simulating the storage box connections.
Addresses 10.2.X.28 are configured on these 10GbE ports.
[root@nsn57-28 ~]# ifconfig eth2
eth2 Link encap:Ethernet HWaddr 00:1B:21:51:26:EC
inet addr:10.2.2.28 Bcast:10.2.2.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe51:26ec/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4776 errors:0 dropped:0 overruns:0 frame:0
TX packets:423 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:507165 (495.2 KiB) TX bytes:28021 (27.3 KiB)

[root@nsn57-28 ~]# ifconfig eth3


eth3 Link encap:Ethernet HWaddr 00:1B:21:51:26:ED
inet addr:10.2.3.28 Bcast:10.2.3.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe51:26ed/64 Scope:Link

63

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1


RX packets:4470 errors:0 dropped:0 overruns:0 frame:0
TX packets:62 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:451968 (441.3 KiB) TX bytes:12345 (12.0 KiB)

[root@nsn57-28 ~]# ifconfig eth4


eth4 Link encap:Ethernet HWaddr 00:1B:21:51:26:BC
inet addr:10.2.4.28 Bcast:10.2.4.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe51:26bc/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4274 errors:0 dropped:0 overruns:0 frame:0
TX packets:43 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:407345 (397.7 KiB) TX bytes:7109 (6.9 KiB)

[root@nsn57-28 ~]# ifconfig eth5


eth5 Link encap:Ethernet HWaddr 00:1B:21:51:26:BD
inet addr:10.2.5.28 Bcast:10.2.5.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe51:26bd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4058 errors:0 dropped:0 overruns:0 frame:0
TX packets:61 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:375170 (366.3 KiB) TX bytes:12095 (11.8 KiB)

5. Ping the four connections to the simulated storage device from the Blade Server in slot 0 in the Sun Blade
6000 Modular System:
[root@nsn57-184 ~]# ping 10.2.2.28
PING 10.2.2.28 (10.2.2.28) 56(84) bytes of data.
64 bytes from 10.2.2.28: icmp_seq=1 ttl=64 time=1.79 ms
64 bytes from 10.2.2.28: icmp_seq=2 ttl=64 time=0.158 ms
--- 10.2.2.28 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.158/0.974/1.790/0.816 ms

64 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

[root@nsn57-184 ~]# ping 10.2.3.28


PING 10.2.3.28 (10.2.3.28) 56(84) bytes of data.
64 bytes from 10.2.3.28: icmp_seq=1 ttl=64 time=0.887 ms
--- 10.2.3.28 ping statistics --1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.887/0.887/0.887/0.000 ms

[root@nsn57-184 ~]# ping 10.2.4.28


PING 10.2.4.28 (10.2.4.28) 56(84) bytes of data.
64 bytes from 10.2.4.28: icmp_seq=1 ttl=64 time=1.06 ms
--- 10.2.4.28 ping statistics --1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.065/1.065/1.065/0.000 ms

[root@nsn57-184 ~]# ping 10.2.5.28


PING 10.2.5.28 (10.2.5.28) 56(84) bytes of data.
64 bytes from 10.2.5.28: icmp_seq=1 ttl=64 time=1.43 ms
--- 10.2.5.28 ping statistics --1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.434/1.434/1.434/0.000 ms

[root@nsn57-184 ~]# ping 10.2.5.28


PING 10.2.5.28 (10.2.5.28) 56(84) bytes of data.
64 bytes from 10.2.5.28: icmp_seq=1 ttl=64 time=1.41 ms
64 bytes from 10.2.5.28: icmp_seq=2 ttl=64 time=0.096 ms
--- 10.2.5.28 ping statistics --2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.096/0.756/1.417/0.661 ms

6. Ping the remote switch or router from the remote server:


[root@nsn57-28 ~]# ping 10.2.2.20
PING 10.2.2.20 (10.2.2.20) 56(84) bytes of data.
64 bytes from 10.2.2.20: icmp_seq=1 ttl=64 time=1.80 ms
64 bytes from 10.2.2.20: icmp_seq=2 ttl=64 time=0.106 ms

65

Topology Switched NEM High Availability Connection


The following figure shows a sample topology for the Switched NEM high availability connection.

Figure 7. Sample Topology For the Switched NEM High Availability Connection

Setting Up a Four NEM/Two Chassis High Availibility Configuration


This example configuration illustrates a High Availability configuration where the 10GbE connections are failed
over between two NEMs, one in slot NEM0 and the other in slot NEM1.
In the example, two Sun Blade 6000 Modular System chassis are used. The first Sun Blade 6000 Modular System
chassis (SBC6000_1) is used to show the sample High Availability configuration using Linux and Solaris OS-based
blade servers. The other Sun Blade 6000 Modular System chassis (SBC6000_2) is used to simulate a remote
switch/router and server resource that needs to be reached from the blade servers on chassis SBC6000_1, and it is
configured in a similar manner. The QSFP ports used for interconnections are configured as a link aggregation.
Each blade server has a dual-port 10GbE FEM card, with each 10GbE interface connected to the NEMs. One of the
FEMs 10GbE interface is connected to the NEM in slot NEM0, and the other FEM interface is connected to the
NEM in slot NEM1.
For Linux hosts, these two FEM interfaces are configured into a single bonding inteface with the High Availability
configuration option. For Solaris hosts, the two FEM interfaces are configured into the same IMPM failover group.
See Figure 7 for the topology for this procedure.

66 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

1. Configure the Blade Servers and QSFP ports on the SBC6000_1 chassis.
Enable bonding on the Linux server:
[root@linux-184 ~]# modprobe bonding arp_interval=1000
arp_ip_target=38.38.38.210 mode=1
[root@linux-184 ~]# ifconfig bond0 38.38.38.184/24 up
Interface)

(monitor

[root@linux-184 ~]# ifenslave bond0 eth eth2


[root@linux-184 ~]# ifconfig bond0:5 5.5.5.184/24 up
Interface)

(test

Enable the QSFP ports P3-6 and P7-10 on the NEM in slot NEM0. Configure the link aggregation
interfaces Po100 and Po200 on these QSFP ports:
SEFOS# configure terminal
SEFOS(config)# set port-channel enable
SEFOS(config)# interface port-channel 100
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface port-channel 200
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/3-6
SEFOS(config-if-range)# channel-group 100 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/7-10
SEFOS(config-if-range)# channel-group 200 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# exit
SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:56:0f
Flags:
D down

P - in port-channel

I stand-alone

H - Hot-standby (LACP only)

Number of channel-groups in use: 2

67

Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(D)
Po200(D)

LACP
LACP

Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)

Enable the QSFP ports P3-6 and P7-10 on the NEM in slot NEM1. Configure the link aggregation
interfaces Po100 and Po200 on these QSFP ports:
SEFOS# configure terminal
SEFOS(config)# set port-channel enable
SEFOS(config)# interface port-channel 100
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface port-channel 200
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/3-6
SEFOS(config-if-range)# channel-group 100 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/7-10
SEFOS(config-if-range)# channel-group 200 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# exit
SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:67:0f
Flags:
D down

P - in port-channel

I stand-alone

H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100

Po100(D)

LACP

Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)

68 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

200

Po200(D)

LACP

Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)

2. Configure the Blade Servers and QSFP ports on the SBC6000_2 chassis.
Enable bonding on the Linux server:
[root@linux-210 ~]# modprobe bonding arp_interval=1000
arp_ip_target=38.38.38.184 mode=1
[root@linux-210 ~]# ifconfig bond0 38.38.38.210/24 up
Interface)

(monitor

[root@linux-210 ~]# ifenslave bond0 eth eth2


[root@linux-210 ~]# ifconfig bond0:5 5.5.5.210/24 up
Interface)

(test

Enable the QSFP ports P3-6 and P7-10 on the NEM in slot NEM0. Configure the link aggregation
interfaces Po100 and Po200 on these QSFP ports:
SEFOS# configure terminal
SEFOS(config)# set port-channel enable
SEFOS(config)# interface port-channel 100
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface port-channel 200
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/3-6
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# channel-group 100 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/7-10
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# channel-group 200 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# exit
SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:7c:63:11
Flags:

69

D down

P - in port-channel

I stand-alone

H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(P)
Po200(P)

LACP
LACP

Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)

Enable the QSFP ports P3-6 and P7-10 on the NEM in slot NEM1. Configure the link aggregation
interfaces Po100 and Po200 on these QSFP ports:
SEFOS# configure terminal
SEFOS(config)# set port-channel enable
SEFOS(config)# interface port-channel 100
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface port-channel 200
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/3-6
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# channel-group 100 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/7-10
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# channel-group 200 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# exit
SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:63:0f
Flags:
D down

P - in port-channel

I stand-alone

H - Hot-standby (LACP only)

70 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
Po100(P)
LACP
Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
200
Po200(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)

3. Check the link aggregation and spanning tree status on each of the four NEMs.
Check the link aggregation and spanning tree status on the NEM in slot NEM0 in the SBC6000_1
chassis:
SEFOS# show etherchannel summary
Port-channel Module
Port-channel Module
Port-channel System
Flags:
D down
I stand-alone

Admin Status is enabled


Oper Status is enabled
Identifier is 00:14:4f:6c:56:0f
P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(P)
Po200(P)

LACP
LACP

Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)

Note that the entries in the Port-channel and Ports columns changed from D (down) to P (in port-channel)
for the Sun Blade 6000 Ethernet Switched NEM 24p 10GbE installed in NEM slot 0 in the first Sun Blade
6000 Modular System.
SEFOS# show spanning-tree
Root Id

Priority 0
Address 00:00:00:00:00:00
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been disabled
Bridge Id
Priority 32768
Address 00:00:00:00:00:00
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name

Role

State

Cost

Prio

Type

71

----

----

-----

----

----

------

Ex0/15

Disabled

Unknown

2000

128

P2P

Ex0/17

Disabled

Unknown

2000

128

P2P

Ex0/19

Disabled

Unknown

2000

128

P2P

Ex0/24

Disabled

Unknown

2000

128

P2P

po100

Disabled

Unknown

1900

128

P2P

po200

Disabled

Unknown

1900

128

P2P

Check the link aggregation and spanning tree status on the NEM in slot NEM1 in the SBC6000_1
chassis:
SEFOS# show etherchannel summary
Port-channel Module
Port-channel Module
Port-channel System
Flags:
D down
I stand-alone

Admin Status is enabled


Oper Status is enabled
Identifier is 00:14:4f:6c:67:0f
P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(P)
Po200(P)

LACP
LACP

Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)

Note that the entries in the Port-channel and Ports columns changed from D (down) to P (in port-channel)
for the NEM installed in slot NEM 1 in the SBC6000_1 chassis.
SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:63:0f
Cost 1900
Port 25 [po100]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:67:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled

72 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Name

Role

State

Cost

Prio

Type

----

----

-----

----

----

------

Ex0/15

Designated

Forwarding

2000

128

P2P

Ex0/17

Designated

Forwarding

2000

128

P2P

Ex0/19

Designated

Forwarding

2000

128

P2P

Ex0/24

Designated

Forwarding

2000

128

P2P

po100

Root

Forwarding

1900

128

P2P

po200

Designated

Forwarding

1900

128

P2P

4. Check the bonding interface status for the Linux Blade Server linux-184 in the SBC6000_1 chassis.
[root@nsn57-184 ~]#

more /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006)


Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth2
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 1000
ARP IP target/s (n.n.n.n form): 38.38.38.210
Slave Interface: eth
MII Status: up
Link Failure Count: 4
Permanent HW addr: 00:21:28:2d:f5:c6
Slave Interface: eth2
MII Status: up
Link Failure Count: 36
Permanent HW addr: 00:21:28:2d:f5:c7

This output confirms that the switched NEM installed in NEM slot 1 in the first Sun Blade 6000 Modular System is
the active path.
5. Unplug the cables from QSFP ports 3-6 and 7-10 on the NEM in slot NEM1 in the SBC6000_1 chassis.
6. Check the interface status on the SBC6000_1 chassis.
Check the link status of the link aggregation interfaces and the spanning tree status on the NEM in slot
NEM1 in the SBC6000_1 chassis. The link aggregation interfaces should be down and the spanning tree
root should move somewhere else.
SEFOS# show spanning-tree

Root Id

Priority 32768
Address 00:14:4f:6c:67:0f

73

Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec
MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id

Priority 32768
Address 00:14:4f:6c:67:0f
Max age is 20 sec, forward delay is 15 sec

Name

Dynamic Path Cost is Disabled


Role
State
Cost

Prio

Type

----

----

-----

----

----

------

Ex0/15

Designated

Forwarding

2000

128

P2P

Ex0/17

Designated

Forwarding

2000

128

P2P

Ex0/19

Designated

Forwarding

2000

128

P2P

Ex0/24

Designated

Forwarding

2000

128

P2P

SEFOS# show etherchannel summary


Port-channel Module
Port-channel Module
Port-channel System
Flags:
D down
I stand-alone

Admin Status is enabled


Oper Status is enabled
Identifier is 00:14:4f:6c:67:0f
P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(D)
Po200(D)

LACP
LACP

Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)

Check the link status of the link aggregation interfaces and the spanning tree status on the NEM in slot
NEM0 in the SBC6000_1 chassis:
SEFOS# show spanning-tree
Root Id

Priority 0
Address 00:00:00:00:00:00
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

74 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

MST00
Spanning tree Protocol has been disabled
Bridge Id
Priority 32768
Address 00:00:00:00:00:00

Name

Max age is 20 sec, forward delay is 15 sec


Dynamic Path Cost is Disabled
Role
State
Cost
Prio

Type

----

----

-----

----

----

------

Ex0/15

Disabled

Unknown

2000

128

P2P

Ex0/17

Disabled

Unknown

2000

128

P2P

Ex0/19

Disabled

Unknown

2000

128

P2P

Ex0/24

Disabled

Unknown

2000

128

P2P

po10

Disabled

Unknown

1900

128

P2P

po200

Disabled

Unknown

1900

128

P2P

SEFOS# show etherchannel summary


Port-channel Module
Port-channel Module
Port-channel System
Flags:
D down
I stand-alone

Admin Status is enabled


Oper Status is enabled
Identifier is 00:14:4f:6c:56:0f
P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(P)
Po200(P)

LACP
LACP

Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)

Check the bonding status on the the linux-184 Blade Server in the SBC6000_1 chassis. The active
bonding interface should be changed from eth2 to eth. The 10GbE interface eth corresponds to
the NEM in slot NEM0.
[root@linux-184 ~]#

more /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006)


Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 1000
ARP IP target/s (n.n.n.n form): 38.38.38.210
Slave Interface: eth
MII Status: up

75

Link Failure Count: 4


Permanent HW addr: 00:21:28:2d:f5:c6
Slave Interface: eth2
MII Status: down
Link Failure Count: 37
Permanent HW addr: 00:21:28:2d:f5:c7

7. Check the bonding interface status for the Blade Server in the SBC6000_2 chassis.
[root@nsn57-20 ~]#

more /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)


Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth4
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 1000
ARP IP target/s (n.n.n.n form): 38.38.38.184
Slave Interface: eth4
MII Status: up
Link Failure Count: 27
Permanent HW addr: 00:21:28:2d:f6:26
Slave Interface: eth5
MII Status: up
Link Failure Count: 28
Permanent HW addr: 00:21:28:2d:f6:27

8. Check the interface status on the SBC6000_2 chassis.


Check the link status of the link aggregation interfaces and the spanning tree status on the NEM in slot
NEM1 in the SBC6000_2 chassis:
SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:63:0f
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:63:0f

Name

Role

Max age is 20 sec, forward delay is 15 sec


Dynamic Path Cost is Disabled
State
Cost
Prio
Type

76 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

----

----

-----

----

----

------

Ex0/1

Designated

Forwarding

2000

128

P2P

Ex0/16

Designated

Forwarding

2000

128

P2P

Ex0/18

Designated

Forwarding

2000

128

P2P

Ex0/22

Designated

Forwarding

2000

128

P2P

po200

Designated

Forwarding

1900

128

P2P

SEFOS# show etherchannel summary


Port-channel Module
Port-channel Module
Port-channel System
Flags:
D down
I stand-alone

Admin Status is enabled


Oper Status is enabled
Identifier is 00:14:4f:6c:63:0f
P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(D)
Po200(P)

LACP
LACP

Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)

Check the link status of the link aggregation interfaces and the spanning tree status on the NEM in slot
NEM0 in the SBC6000_2 chassis:
SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:63:0f
Cost 1900
Port 25 [po100]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the mstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:7c:63:11
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
State
Cost
Prio
Type

Name

Role

----

----

-----

----

----

------

Ex0/16

Designated

Forwarding

2000

128

P2P

Ex0/18

Designated

Forwarding

2000

128

P2P

Ex0/22

Designated

Forwarding

2000

128

P2P

po100

Root

Forwarding

1900

128

P2P

77

SEFOS# show etherchannel summary


Port-channel Module
Port-channel Module
Port-channel System
Flags:
D down
I stand-alone

Admin Status is enabled


Oper Status is enabled
Identifier is 00:14:4f:7c:63:11
P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(P)
Po200(D)

LACP
LACP

Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)

9. Unplug the cables from QSFP ports 3-6 and 7-10 on the NEM in slot NEM0 in the SBC6000_2 chassis..
10. Check the bonding status on the Blade Server in the SBC_60002 chassis.
[root@nsn57-20 ~]#

more /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)


Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth5
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 1000
ARP IP target/s (n.n.n.n form): 38.38.38.184
Slave Interface: eth4
MII Status: down
Link Failure Count: 28
Permanent HW addr: 00:21:28:2d:f6:26
Slave Interface: eth5
MII Status: up
Link Failure Count: 28
Permanent HW addr: 00:21:28:2d:f6:27

11. Check the interface status on the SBC6000_2 chassis.


Check the link status of the link aggregation interfaces and the spanning tree status on the NEM in slot
NEM0 in the SBC6000_2 chassis:
SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:7c:63:11
Cost 0
Port 0 [0]
This bridge is the root

78 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Max age 20 Sec, forward delay 15 Sec


MST00
Spanning tree Protocol has been enabled
MST00 is executing the mstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:7c:63:11
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
State
Cost
Prio
Type

Name

Role

----

----

-----

----

----

------

Ex0/16

Designated

Forwarding

2000

128

P2P

Ex0/18

Designated

Forwarding

2000

128

P2P

Ex0/22

Designated

Forwarding

2000

128

P2P

SEFOS# show etherchannel summary


Port-channel Module
Port-channel Module
Port-channel System
Flags:
D down
I stand-alone

Admin Status is enabled


Oper Status is enabled
Identifier is 00:14:4f:7c:63:11
P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(D)
Po200(D)

LACP
LACP

Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)

Check the link status of the link aggregation interfaces and the spanning tree status on the NEM in slot
NEM1 in the SBC6000_2 chassis:
SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:63:0f
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:63:0f

79

Max age is 20 sec, forward delay is 15 sec


Dynamic Path Cost is Disabled
State
Cost
Prio
Type

Name

Role

----

----

-----

----

----

------

Ex0/1

Designated

Forwarding

2000

128

P2P

Ex0/16

Designated

Forwarding

2000

128

P2P

Ex0/18

Designated

Forwarding

2000

128

P2P

Ex0/22

Designated

Forwarding

2000

128

P2P

po200

Designated

Forwarding

1900

128

P2P

SEFOS# show etherchannel summary


Port-channel Module
Port-channel Module
Port-channel System
Flags:
D down
I stand-alone

Admin Status is enabled


Oper Status is enabled
Identifier is 00:14:4f:6c:63:0f
P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(D)
Po200(P)

LACP
LACP

Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)

12. Check the interface status on the SBC6000_1 chassis.


Check the bonding interface status on the for the Linux Blade Server linux-184 in the SBC6000_1
chassis.
[root@nsn57-184 ~]#

more /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006)


Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
ARP Polling Interval (ms): 1000
ARP IP target/s (n.n.n.n form): 38.38.38.210
Slave Interface: eth
MII Status: up
Link Failure Count: 4
Permanent HW addr: 00:21:28:2d:f5:c6
Slave Interface: eth2
MII Status: down

80 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Link Failure Count: 37


Permanent HW addr: 00:21:28:2d:f5:c7

Check the link aggregation and spanning tree status on the NEM in slot NEM0 in the SBC6000_1
chassis:
SEFOS# show spanning-tree
Root Id

Priority 0
Address 00:00:00:00:00:00
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been disabled
Bridge Id
Priority 32768
Address 00:00:00:00:00:00
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
State
Cost
Prio
Type

Name

Role

----

----

-----

----

----

------

Ex0/15

Disabled

Unknown

2000

128

P2P

Ex0/17

Disabled

Unknown

2000

128

P2P

Ex0/19

Disabled

Unknown

2000

128

P2P

Ex0/24

Disabled

Unknown

2000

128

P2P

po200

Disabled

Unknown

1900

128

P2P

SEFOS# show etherchannel summary


Port-channel Module
Port-channel Module
Port-channel System
Flags:
D down
I stand-alone

Admin Status is enabled


Oper Status is enabled
Identifier is 00:14:4f:6c:56:0f
P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(D)
Po200(P)

LACP
LACP

Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)

81

Check the link aggregation and spanning tree status on the NEM in slot NEM1 in the SBC6000_1
chassis:
SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:67:0f
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:67:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
State
Cost
Prio
Type

Name

Role

----

----

-----

----

----

------

Ex0/15

Designated

Forwarding

2000

128

P2P

Ex0/17

Designated

Forwarding

2000

128

P2P

Ex0/19

Designated

Forwarding

2000

128

P2P

Ex0/24

Designated

Forwarding

2000

128

P2P

SEFOS# show etherchannel summary


Port-channel Module
Port-channel Module
Port-channel System
Flags:
D down
I stand-alone

Admin Status is enabled


Oper Status is enabled
Identifier is 00:14:4f:6c:67:0f
P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 2


Number of aggregators: 2
Group Port-channel Protocol
Ports
------------------------------------------------------------------------100
200

Po100(D)
Po200(D)

LACP
LACP

Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)

82 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

13. Configure and verify High Availability on the Solaris Blade server on the SBC6000_1 chassis.
These steps will reestablish a connection on the unplugged QSFP cable and will verify High Availability on the
Linux Blade Server. The Solaris Blade Server has two 10GbE interfaces, just as the Linux Blade Server has.
Interface ixgbe0 is connected to the NEM in slot NEM0 and interface ixgbe1 is connected to the NEM in slot NEM1
in the SBC6000_1 chassis.
Configure and verify High Availability on the ixgbe0 10GbE interface:
[root@Solaris-190
[root@Solaris-190
broadcast + group
-failover netmask

~]# ifconfig ixgbe0 plumb


~]# ifconfig ixgbe0 38.38.38.190 netmask +
testgroup1 up \ addif 38.38.38.192 deprecated
+ broadcast + up

Configure and verify High Availability on the ixgbe1 10GbE interface:


[root@Solaris-190
[root@Solaris-190
broadcast + group
-failover netmask

~]# ifconfig ixgbe1 plumb


~]# ifconfig ixgbe1 38.38.38.191 netmask +
testgroup1 up \ addif 38.38.38.193 deprecated
+ broadcast + up

14. Add the target for probe-based monitoring:


[root@nsn57-184 ~]# route add -host 38.38.38.210 38.38.38.210 -static
15. Ping the target Blade Server in the SBC6000_2 chassis.
The ping should succeed.
16. Remove the NEM from slot NEM0 in the SBC6000_1 chassis.
Refer to the Sun Blade 6000 Ethernet Switched NEM 24p 10GbE User's Guide for those instructions, then return
here. Removing the switched NEM from a running system will simulate having a failed switched NEM in that slot.
17. Monitor the console output of the Solaris Blade Server in the SBC6000_1 chassis:
Feb 25 18:29:36 Solaris-190 in.mpathd[858]: [ID 594170 daemon.error] NIC
failure detected on ixgbe0 of group testgroup1
Feb 25 18:29:36 Solaris-190 in.mpathd[858]: [ID 832587 daemon.error]
Successfully failed over from NIC ixgbe0 to NIC ixgbe1

Note that the ixgbe1 10 GbE interface has become the active IPMP interface.
18. Re-insert the NEM into slot NEM0 in the SBC6000_1 chassis.
19. Monitor the console output of the Solaris Blade Server in the SBC6000_1 chassis:
Feb 25 18:31:56 Solaris-190 in.mpathd[858]: [ID 299542 daemon.error] NIC
repair detected on ixgbe0 of group testgroup1
Feb 25 18:31:56 Solaris-190 in.mpathd[858]: [ID 620804 daemon.error]
Successfully failed back to NIC ixgbe0

83

Note that the active link has failed back over to the ixgbe0 10GbE interface.
20. Disable the 10GbE interface corresponding to the ixgbe0 link on the NEM in slot NEM0 in the
SBC6000_1 chassis:
SEFOS# configure terminal
SEFOS(config)# interface extreme-ethernet 0/19
SEFOS(config-if)# shutdown
SEFOS(config-if)# end
SEFOS#
21. Monitor the console output of the Solaris Blade Server in the SBC6000_1 chassis:
Feb 25 18:35:07 Solaris-190 ixgbe: [ID 339653 kern.notice] NOTICE: ixgbe0:
xcvr addr:0x00 - link is down
Feb 25 18:35:07 Solaris-190 in.mpathd[858]: [ID 215189 daemon.error] The
link has gone down on ixgbe0
Feb 25 18:35:07 Solaris-190 in.mpathd[858]: [ID 594170 daemon.error] NIC
failure detected on ixgbe0 of group testgroup1
Feb 25 18:35:07 Solaris-190 in.mpathd[858]: [ID 832587 daemon.error]
Successfully failed over from NIC ixgbe0 to NIC ixgbe1
Feb 25 18:35:07 Solaris-190 ixgbe: [ID 752849 kern.warning] WARNING: ixgbe0
: portn 0

Note that the active link has failed over to the ixgbe1 10GbE interface.
22. Enable the 10GbE interface corresponding to the ixgbe0 link on the NEM in slot NEM0 in the
SBC6000_1 chassis:
SEFOS# configure terminal
SEFOS(config)# interface extreme-ethernet 0/19
SEFOS(config-if)# no shutdown
SEFOS(config-if)# end
SEFOS#
23. Monitor the console output of the Solaris Blade Server in the SBC6000_1 chassis:
Link ixgbe0 recovered
Feb 25 18:35:57 Solaris-190 ixgbe: [ID 339653 kern.notice] NOTICE: ixgbe0:
xcvr addr:0x00 - link is up 10000 Mbps full duplex
Feb 25 18:35:57 Solaris-190 in.mpathd[858]: [ID 820239 daemon.error] The
link has come up on ixgbe0
Feb 25 18:35:58 Solaris-190 in.routed[350]: [ID 464608 daemon.error] route
38.38.38.192 --> 0.0.0.0 nexthop is not directly connected
Feb 25 18:36:13 Solaris-190 in.mpathd[858]: [ID 299542 daemon.error] NIC
repair detected on ixgbe0 of group testgroup1
Feb 25 18:36:13 Solaris-190 in.mpathd[858]: [ID 620804 daemon.error]
Successfully failed back to NIC ixgbe0

Note that the active link has failed back over to the ixgbe0 10GbE interface.

84 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Topology Four Switched NEMs, High Availability Connection


The following figure shows a sample topology for four Switched NEMs, high availability connection.

Figure 8. Sample Topology For Four Switched NEMs, High Availability Connection

Setting Up a Four NEM/Four Chassis High Availibility Configuration


In this example, the configuration consists of a cluster of servers within four Sun Blade 6000 Modular System
chassis. Each Sun Blade 6000 Modular System chassis has one or two NEMs (Sun Blade 6000 Ethernet Switched
NEM 24p 10GbEs from Oracle) installed. The NEMs are interconnected in the topology shown in Figure 8 using the
QSFP ports. The four ports on each QSFP port are configured as a link aggregation.
For the purposes of this exercise, the following components will use the following Ethernet addresses:
NEM A, installed in Chassis A: Ethernet address 1.1.1.184
NEM B, installed in Chassis B: Ethernet address 1.1.1.190
NEM C, installed in Chassis C: Ethernet address 1.1.1.156
NEM D, installed in Chassis D: Ethernet address 1.1.1.20
See Figure 8 for the topology for this procedure.

85

Setting Up QSFP Ports Into a Link Aggregation


This procedure describes how to configure the QSFP ports as a link aggregation. Note that you can copy and paste
this information to a SEFOS console or you can use this information to create a script.
SEFOS# configure terminal
SEFOS(config)# set port-channel enable
SEFOS(config)# interface port-channel 12
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/1-2
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# channel-group 12 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface port-channel 36
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/3-6
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# channel-group 36 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface port-channel 710
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/7-10
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# channel-group 710 mode active
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface port-channel 1114
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/11-14
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# channel-group 1114 mode active

86 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config)# interface range extreme-ethernet 0/15-24
SEFOS(config-if-range)# shutdown
SEFOS(config-if-range)# spanning-tree portfast
SEFOS(config-if-range)# no shutdown
SEFOS(config-if-range)# exit
SEFOS(config-if-range)# end

Checking the Connectivity Status on the NEMs


1. Check the connectivity status on NEM A:
SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 1900
Port 26 [po36]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:67:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
State
Cost
Prio
Type

Name

Role

----

----

-----

----

----

------

Ex0/15
Ex0/17
Ex0/19
Ex0/21
Ex0/22
Ex0/23
Ex0/24
po36
po710
po1114

Designated
Designated
Designated
Designated
Designated
Designated
Designated
Root
Alternate
Designated

Forwarding
Forwarding
Forwarding
Forwarding
Forwarding
Forwarding
Forwarding
Forwarding
Discarding
Forwarding

2000
2000
2000
2000
2000
2000
2000
1900
1900
1900

128
128
128
128
128
128
128
128
128
128

P2P
P2P
P2P
P2P
P2P
P2P
P2P
P2P
P2P
P2P

SEFOS# show etherchannel summary


Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:67:0f

87

Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(P)
LACP
Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
710
Po710(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

2. Check the connectivity status on NEM B:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:56:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(P)
LACP
Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
710
Po710(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

SEFOS# show spanning-tree


Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:56:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/17
Designated
Forwarding
2000
128
P2P

88 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Ex0/19
Ex0/21
Ex0/22
Ex0/23
Ex0/24
po36
po710
po1114

Designated
Designated
Designated
Designated
Designated
Designated
Designated
Designated

Forwarding
Forwarding
Forwarding
Forwarding
Forwarding
Forwarding
Forwarding
Forwarding

2000
2000
2000
2000
2000
1900
1900
1900

128
128
128
128
128
128
128
128

P2P
P2P
P2P
P2P
P2P
P2P
P2P
P2P

3. Check the connectivity status on NEM C:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:63:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(P)
LACP
Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
710
Po710(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

SEFOS# show spanning-tree


Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 1900
Port 28 [po1114]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:63:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/16
Designated
Forwarding
2000
128
P2P
Ex0/18
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
po36
Designated
Forwarding
1900
128
P2P
po710
Designated
Forwarding
1900
128
P2P
po1114
Root
Forwarding
1900
128
P2P

89

4. Check the connectivity status on NEM D:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:7c:63:11
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(P)
LACP
Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
710
Po710(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

SEFOS# show spanning-tree


Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 1900
Port 27 [po710]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:7c:63:11
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/16
Designated
Forwarding
2000
128
P2P
Ex0/18
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
po36
Alternate
Discarding
1900
128
P2P
po710
Root
Forwarding
1900
128
P2P
po1114
Alternate
Discarding
1900
128
P2P

The spanning tree and link aggregation status indicates that there is full mesh connectivity between the NEMs and
the Sun Blade 6000 Modular System chassis.
5. Ping the other chassis from Chassis B (1.1.1.190):
# ping 1.1.1.184; ping 1.1.1.156; ping 1.1.1.20
1.1.1.184 is alive
1.1.1.156 is alive
1.1.1.20 is alive

90 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

The following steps will verify that redundant paths exist between the multiple chassis.
6. Disconnect the QSFP cable connecting NEM A in Chassis A to NEM B in Chassis B (QSFP ports 3-6).
7. Check the connectivity on NEM A installed in Chassis A:
SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:67:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(D)
LACP
Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
710
Po710(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

Note that trunk link Po36 is now down.


SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 3800
Port 27 [po710]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:67:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/17
Designated
Forwarding
2000
128
P2P
Ex0/19
Designated
Forwarding
2000
128
P2P
Ex0/21
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
Ex0/23
Designated
Forwarding
2000
128
P2P
Ex0/24
Designated
Forwarding
2000
128
P2P
po710
Root
Forwarding
1900
128
P2P
po1114
Alternate
Discarding
1900
128
P2P

91

8. Enter the following commands for NEM B installed in Chassis B:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:56:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(D)
LACP
Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
710
Po710(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

Note that trunk link Po36 is now down.


SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:56:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/17
Designated
Forwarding
2000
128
P2P
Ex0/19
Designated
Forwarding
2000
128
P2P
Ex0/21
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
Ex0/23
Designated
Forwarding
2000
128
P2P
Ex0/24
Designated
Forwarding
2000
128
P2P
po710
Designated
Forwarding
1900
128
P2P
po1114
Designated
Forwarding
1900
128
P2P

9. Enter the following commands for NEM C installed in Chassis C:


SEFOS# show etherchannel summary

92 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Port-channel Module Admin Status is enabled


Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:63:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(P)
LACP
Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
710
Po710(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

SEFOS# show spanning-tree


Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 1900
Port 28 [po1114]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:63:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/16
Designated
Forwarding
2000
128
P2P
Ex0/18
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
po36
Designated
Forwarding
1900
128
P2P
po710
Designated
Forwarding
1900
128
P2P
po1114
Root
Forwarding
1900
128
P2P

10. Enter the following commands for NEM D installed in Chassis D:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:7c:63:11
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4

93

Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(P)
LACP
Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
710
Po710(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

SEFOS# show spanning-tree


Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 1900
Port 27 [po710]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:7c:63:11
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/16
Designated
Forwarding
2000
128
P2P
Ex0/18
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
po36
Alternate
Discarding
1900
128
P2P
po710
Root
Forwarding
1900
128
P2P
po1114
Designated
Forwarding
1900
128
P2P

11. Ping the other chassis from Chassis B (1.1.1.190):


# ping 1.1.1.184; ping 1.1.1.156; ping 1.1.1.20
1.1.1.184 is alive
1.1.1.156 is alive
1.1.1.20 is alive

This step in the procedure verifies that even though there is a link failure of the link aggregation between the chassis
(in this case, between NEM A and NEM B, QSFP ports 3-6), there is still network connectivity between the two
NEMs.
The following steps will introduce an additional link failure to verify that redundant paths continue to exist between
the multiple chassis.
12. Disconnect the QSFP cable connecting NEM B in Chassis B to NEM D in Chassis D (QSFP ports 7-11).
13. Check the connectivity on NEM A installed in Chassis A:
SEFOS# show etherchannel summary

94 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

Port-channel Module Admin Status is enabled


Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:67:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(D)
LACP
Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
710
Po710(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

SEFOS# show spanning-tree


Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 3800
Port 27 [po710]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:67:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/17
Designated
Forwarding
2000
128
P2P
Ex0/19
Designated
Forwarding
2000
128
P2P
Ex0/21
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
Ex0/23
Designated
Forwarding
2000
128
P2P
Ex0/24
Designated
Forwarding
2000
128
P2P
po710
Root
Forwarding
1900
128
P2P
po1114
Designated
Forwarding
1900
128
P2P

14. Check the connectivity on NEM B installed in Chassis B:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:56:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

95

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(D)
LACP
Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
710
Po710(D)
LACP
Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

Note that trunk link Po710 is now down.


SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:56:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/17
Designated
Forwarding
2000
128
P2P
Ex0/19
Designated
Forwarding
2000
128
P2P
Ex0/21
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
Ex0/23
Designated
Forwarding
2000
128
P2P
Ex0/24
Designated
Forwarding
2000
128
P2P
po1114
Designated
Forwarding
1900
128
P2P

15. Check the connectivity on NEM C installed in Chassis C:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:63:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
-------------------------------------------------------------------------

96 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

12
36
710
1114

Po12(D)
Po36(P)
Po710(P)
Po1114(P)

LACP
LACP
LACP
LACP

Ex0/1(D),Ex0/2(D)
Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

SEFOS# show spanning-tree


Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 1900
Port 28 [po1114]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:63:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/16
Designated
Forwarding
2000
128
P2P
Ex0/18
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
po36
Designated
Forwarding
1900
128
P2P
po710
Designated
Forwarding
1900
128
P2P
po1114
Root
Forwarding
1900
128
P2P

16. Check the connectivity on NEM D installed in Chassis D:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:7c:63:11
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(P)
LACP
Ex0/3(P),Ex0/4(P),Ex0/5(P),Ex0/6(P)
710
Po710(D)
LACP
Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

Note that trunk link Po710 is now down.


SEFOS# show spanning-tree
97

Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 3800
Port 26 [po36]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:7c:63:11
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/16
Designated
Forwarding
2000
128
P2P
Ex0/18
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
po36
Root
Forwarding
1900
128
P2P
po1114
Alternate
Discarding
1900
128
P2P

17. Ping the other chassis from Chassis B (1.1.1.190):


# ping 1.1.1.184; ping 1.1.1.156; ping 1.1.1.20
1.1.1.184 is alive
1.1.1.156 is alive
1.1.1.20 is alive

This step in the procedure verifies that even though there are now two link failures of the link aggregation between
the chassis (in this case, between NEM A and NEM B, ports 3-6, and from between NEM B and NEM D, ports 711), there is still network connectivity between the chassis.
The following steps will introduce another link failure to verify that redundant paths continue to exist between the
multiple chassis.
18. Disconnect the QSFP cable connecting NEM C in Chassis C to NEM D in Chassis D (QSFP ports 3-6).
19. Check the connectivity on NEM A installed in Chassis A:
SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:67:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)

98 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

36
710
1114

Po36(D)
Po710(P)
Po1114(P)

LACP
LACP
LACP

Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

SEFOS# show spanning-tree


Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 3800
Port 27 [po710]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:67:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/17
Designated
Forwarding
2000
128
P2P
Ex0/19
Designated
Forwarding
2000
128
P2P
Ex0/21
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
Ex0/23
Designated
Forwarding
2000
128
P2P
Ex0/24
Designated
Forwarding
2000
128
P2P
po710
Root
Forwarding
1900
128
P2P
po1114
Designated
Forwarding
1900
128
P2P

20. Check the connectivity on NEM B installed in Chassis B:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:56:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(D)
LACP
Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
710
Po710(D)
LACP
Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

SEFOS# show spanning-tree

99

Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:56:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/17
Designated
Forwarding
2000
128
P2P
Ex0/19
Designated
Forwarding
2000
128
P2P
Ex0/21
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
Ex0/23
Designated
Forwarding
2000
128
P2P
Ex0/24
Designated
Forwarding
2000
128
P2P
po1114
Designated
Forwarding
1900
128
P2P

21. Check the connectivity on NEM C installed in Chassis C:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:63:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(D)
LACP
Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
710
Po710(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

Note that trunk link Po36 is now down.


SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 1900
Port 28 [po1114]
Max age 20 Sec, forward delay 15 Sec

100 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:63:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/16
Designated
Forwarding
2000
128
P2P
Ex0/18
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
po710
Designated
Forwarding
1900
128
P2P
po1114
Root
Forwarding
1900
128
P2P

22. Check the connectivity on NEM D installed in Chassis D:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:56:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(D)
LACP
Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
710
Po710(D)
LACP
Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

Note that trunk link Po36 is now down.


SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768

101

Name
---Ex0/15
Ex0/17
Ex0/19
Ex0/21
Ex0/22
Ex0/23
Ex0/24
po1114

Address 00:14:4f:6c:56:0f
Max age is 20 sec, forward delay
Dynamic Path Cost is Disabled
Role
State
Cost
----------Designated
Forwarding
2000
Designated
Forwarding
2000
Designated
Forwarding
2000
Designated
Forwarding
2000
Designated
Forwarding
2000
Designated
Forwarding
2000
Designated
Forwarding
2000
Designated
Forwarding
1900

is 15 sec
Prio
---128
128
128
128
128
128
128
128

Type
-----P2P
P2P
P2P
P2P
P2P
P2P
P2P
P2P

23. Ping the other chassis from Chassis B (1.1.1.190):


# ping 1.1.1.184; ping 1.1.1.156; ping 1.1.1.20
1.1.1.184 is alive
1.1.1.156 is alive
1.1.1.20 is alive

This step in the procedure verifies that even though there are now three link failures of the link aggregation between
the chassis, there is still network connectivity between the chassis:
QSFP cables are disconnected between NEM A and NEM B, ports 3-6
QSFP cables are disconnected between NEM B and NEM D, ports 7-11
QSFP cables are disconnected between NEM C and NEM D, ports 3-6
The following steps will introduce another link failure to verify that even if more trunks fail, there could still be
limited connection between some of the NEMs.
24. Disconnect the QSFP cable connecting NEM A in Chassis A to NEM D in Chassis D (QSFP ports 11-14).
NEM A, NEM B and NEM C can still communicate with each other with this simulated trunk failure.
25. Check the connectivity on NEM A installed in Chassis A:
SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:67:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
-------------------------------------------------------------------------

102 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

12
36
710
1114

Po12(D)
Po36(D)
Po710(P)
Po1114(D)

LACP
LACP
LACP
LACP

Ex0/1(D),Ex0/2(D)
Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
Ex0/11(D),Ex0/12(D),Ex0/13(D),Ex0/14(D)

Note that trunk link Po1114 is now down.


SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 3800
Port 27 [po710]
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:67:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/17
Designated
Forwarding
2000
128
P2P
Ex0/19
Designated
Forwarding
2000
128
P2P
Ex0/21
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
Ex0/23
Designated
Forwarding
2000
128
P2P
Ex0/24
Designated
Forwarding
2000
128
P2P
po710
Root
Forwarding
1900
128
P2P

26. Check the connectivity on NEM B installed in Chassis B:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:56:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(D)
LACP
Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
710
Po710(D)
LACP
Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

103

SEFOS# show spanning-tree


Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:56:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/17
Designated
Forwarding
2000
128
P2P
Ex0/19
Designated
Forwarding
2000
128
P2P
Ex0/21
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
Ex0/23
Designated
Forwarding
2000
128
P2P
Ex0/24
Designated
Forwarding
2000
128
P2P
po1114
Designated
Forwarding
1900
128
P2P

27. Check the connectivity on NEM C installed in Chassis C:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:6c:63:0f
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(D)
LACP
Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
710
Po710(P)
LACP
Ex0/7(P),Ex0/8(P),Ex0/9(P),Ex0/10(P)
1114
Po1114(P)
LACP
Ex0/11(P),Ex0/12(P),Ex0/13(P),Ex0/14(P)

SEFOS# show spanning-tree


Root Id

Priority 32768
Address 00:14:4f:6c:56:0f
Cost 1900
Port 28 [po1114]
Max age 20 Sec, forward delay 15 Sec

104 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:6c:63:0f
Max age is 20 sec, forward delay is 15 sec
Dynamic Path Cost is Disabled
Name
Role
State
Cost
Prio
Type
---------------------Ex0/15
Designated
Forwarding
2000
128
P2P
Ex0/16
Designated
Forwarding
2000
128
P2P
Ex0/18
Designated
Forwarding
2000
128
P2P
Ex0/22
Designated
Forwarding
2000
128
P2P
po710
Designated
Forwarding
1900
128
P2P
po1114
Root
Forwarding
1900
128
P2P

28. Check the connectivity on NEM D installed in Chassis D:


SEFOS# show etherchannel summary
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is 00:14:4f:7c:63:11
Flags:
D down
I stand-alone

P - in port-channel
H - Hot-standby (LACP only)

Number of channel-groups in use: 4


Number of aggregators: 4
Group Port-channel
Protocol
Ports
------------------------------------------------------------------------12
Po12(D)
LACP
Ex0/1(D),Ex0/2(D)
36
Po36(D)
LACP
Ex0/3(D),Ex0/4(D),Ex0/5(D),Ex0/6(D)
710
Po710(D)
LACP
Ex0/7(D),Ex0/8(D),Ex0/9(D),Ex0/10(D)
1114
Po1114(D)
LACP
Ex0/11(D),Ex0/12(D),Ex0/13(D),Ex0/14(D)

Note that trunk link Po1114 is now down.


SEFOS# show spanning-tree
Root Id

Priority 32768
Address 00:14:4f:7c:63:11
Cost 0
Port 0 [0]
This bridge is the root
Max age 20 Sec, forward delay 15 Sec

MST00
Spanning tree Protocol has been enabled
MST00 is executing the rstp compatible Multiple Spanning Tree Protocol
Bridge Id
Priority 32768
Address 00:14:4f:7c:63:11

105

Name
---Ex0/15
Ex0/16
Ex0/18
Ex0/22

Max age is 20 sec, forward delay


Dynamic Path Cost is Disabled
Role
State
Cost
----------Designated
Forwarding
2000
Designated
Forwarding
2000
Designated
Forwarding
2000
Designated
Forwarding
2000

is 15 sec
Prio
---128
128
128
128

Type
-----P2P
P2P
P2P
P2P

29. Ping the other chassis from Chassis B (1.1.1.190):


# ping 1.1.1.184; ping 1.1.1.156; ping 1.1.1.20
1.1.1.184 is alive
1.1.1.156 is alive
no answer from 1.1.1.20

This step in the procedure verifies that even though there are several failures of the link aggregation between the
chassis, there is still network connectivity between the two or three of the NEMs installed in the chassis.

106 Sun Blade 6000 Ethernet Switched NEM 24p 10GbE Software Configuration Guide October 2011