OpenStack Cumulus Linux Validated Design Guide

OpenStack and Cumulus Linux
Validated Design Guide
Deploying OpenStack with Network Switches Running Cumulus Linux
OPENSTACK AND CUMULU S LINUX: VALIDATED DESIGN GUI DE
Contents
Contents ..................................................................................................................................................................................................... 2
OpenStack with Cumulus Linux ............................................................................................................................................................... 4
Objective................................................................................................................................................................................................. 4
Enabling Choice of Hardware in the Data Center ............................................................................................................................. 4
Combined Solution Using OpenStack and Cumulus Linux .............................................................................................................. 4
Driving Towards Operational Efficiencies .......................................................................................................................................... 5
Intended Audience for Network Design and Build ............................................................................................................................ 6
OpenStack Network Architecture in a PoC or Small Test/Dev Environment ..................................................................................... 6
Network Architecture and Design Considerations ............................................................................................................................ 6
OpenStack Network Architecture in a Cloud Data Center ................................................................................................................... 8
Network Architecture ............................................................................................................................................................................ 8
Scaling Out ............................................................................................................................................................................................. 9
Out-of-Band Management ......................................................................................................................................................................10
Building an OpenStack Cloud with Cumulus Linux .............................................................................................................................11
Minimum Hardware Requirements ..................................................................................................................................................11
Network Assumptions and Numbering ............................................................................................................................................12
Build Steps -- Automated....................................................................................................................................................................15
Behind the Scenes of the Automated Build Out ........................................................................................................................ 16
Build Steps -- Manual .........................................................................................................................................................................17
1. Set Up Physical Network ........................................................................................................................................................... 18
2. Basic Physical Network Configuration .................................................................................................................................... 18
3. Verify Connectivity ..................................................................................................................................................................... 21
4. Set Up Physical Servers ............................................................................................................................................................ 21
5. Configure Spine Switches ......................................................................................................................................................... 22
6. Configure Each Pair of Leaf Switches ..................................................................................................................................... 24
7. Configure the OpenStack Controller ....................................................................................................................................... 27
8. Configure Each Compute Node ............................................................................................................................................... 30
9. Create Tenant Networks ........................................................................................................................................................... 33
10. Start VMs Using the OpenStack Horizon Web UI ................................................................................................................ 33
Conclusion ................................................................................................................................................................................................34
Summary ..............................................................................................................................................................................................34
References ...........................................................................................................................................................................................34
Appendix A: Example /etc/network/interfaces Configurations ........................................................................................................36
CONTENTS
leaf01 ...................................................................................................................................................................................................36
leaf02 ...................................................................................................................................................................................................39
leaf03 ...................................................................................................................................................................................................42
leaf04 ...................................................................................................................................................................................................45
spine01 ................................................................................................................................................................................................48
spine02 ................................................................................................................................................................................................50
Appendix B: Network Setup Checklist ..................................................................................................................................................52
Version 1.0.3
June 26, 2015
About Cumulus Networks

Unleash the power of Open Networking with Cumulus Networks. Founded by veteran networking engineers from Cisco and
VMware, Cumulus Networks makes the first Linux operating system for networking hardware and fills a critical gap in
realizing the true promise of the software-defined data center. Just as Linux completely transformed the economics and
innovation on the server side of the data center, Cumulus Linux is doing the same for the network. It is radically reducing
the costs and complexities of operating modern data center networks for service providers and businesses of all sizes.
Cumulus Networks has received venture funding from Andreessen Horowitz, Battery Ventures, Sequoia Capital, Peter
Wagner and four of the original VMware founders. For more information visit cumulusnetworks.com or @cumulusnetworks.
2015 Cumulus Networks. CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the Marks) are trademarks and service marks
of Cumulus Networks, Inc. in the U.S. and other countries. You are not permitted to use the Marks without the prior written consent of Cumulus Networks. The
registered trademark Linux is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. All
other marks are used under fair use or license from their respective owners.
The OpenStack Word Mark and OpenStack Logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundaiton,
in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the
OpenStack Foundation, or the OpenStack community.
www.cumulusnetworks.com
OpenStack with Cumulus Linux

Objective
This Validated Design Guide presents a design and implementation approach for deploying OpenStack with network
switches running Cumulus Linux. Specific steps are included for both a fully automated install of both switches and servers,
as well as the manual steps to install and configure by hand.
Enabling Choice of Hardware in the Data Center

Cloud-oriented infrastructure designs revolutionized how server applications are delivered in the data center. They reduce
CapEx costs by commoditizing server hardware platforms and OpEx costs by automating and orchestrating infrastructure
deployment and management.
The same benefits of choice of commodity hardware and automation are available to networking in the data center. With
Cumulus Linux, network administrators now have a multi-platform network OS that provides freedom of choice with
network switch hardware. Because Cumulus Linux is Linux, data center administrators have access to a rich ecosystem of
existing Linux automation tools and now the ability for converged deployment, administration and monitoring of compute
servers and network switches.
OpenStack is a cloud platform for enterprise and commercial IT environments. Widely deployed in private and public cloud
applications, OpenStack offers a rich variety of components that can be combined to build a tailored cloud solution.
OpenStack enables data center architects to use commodity server hardware to build infrastructure environments that
deliver the agility and easy scaling promised by the cloud. The cloud allows infrastructure consumers to request and utilize
capacity in seconds rather than hours or days, providing you with radical CapEx and OpEx savings while delivering rapid,
self-service deployment of capacity for IT consumers.
Cumulus Networks believes the same design principles should hold true for networking. A network device can be
configured at first boot, so an administrator can quickly replace failed equipment instead of spending valuable time and
resources troubleshooting hardware. This enables new support models to be leveraged to drive down operational costs.
Imagine managing your own set of hot spare switches, guaranteeing that a replacement will always be available instead of
paying for ongoing support for every device. This is the same model currently used by most organizations for managing
large fleets of servers.
Additionally, Cumulus Linux can help you achieve the same CapEx and OpEx efficiencies for your networks by enabling an
open market approach for switching platforms, and by offering a radically simple automated lifecycle management
framework built on the industrys best open source tools. By using bare metal servers and network switches, you can
achieve cost savings that would be impossible just a few years ago.
Combined Solution Using OpenStack and Cumulus Linux

Both Cumulus Linux and Linux/OpenStack are software solutions run on top of bare metal hardware. Because both
solutions are hardware-agnostic, customers can select their chosen platform from a wide array of suppliers who often
employ highly competitive pricing models. The software defines the performance and behavior of the environment and
allows the administrator to exercise version control and programmatic approaches that are already in use by DevOps
teams.
Refer to the Cumulus Linux Hardware Compatibility List (HCL) at cumulusnetworks.com/hcl for a list of hardware vendors
and their supported model numbers, descriptions, switch silicon, and CPU type.
OPENSTACK WITH CUMUL US LINUX
Figure 1. OpenStack and Cumulus Linux
Driving Towards Operational Efficiencies

OpenStack enables the building of cloud environments using commodity off-the-shelf servers combined with standard Linux
virtualization, monitoring, and management technologies. Cloud users can request resources (VMs, storage, network) using
APIs and self-service Web interfaces, and those resources will be allocated and delivered without human intervention. The
hardware in the cloud is thus homogenous, and users neither know nor care where their resources are physically allocated.
Operators monitor aggregate resource utilization, so management is done at the level of a capacity planning exercise,
rather than worrying about individual workloads and users.
OpenStack comprises subcomponents that work together to deliver a cloud. The major components are:
1.
2.
3.
4.
5.
6.
Nova, which manages VMs and their virtual network connectivity.

Glance, which manages OS images.
Cinder, which manages VM storage.
Keystone, which provides authentication and authorization services.
Horizon, a Web-based UI
Neutron, an alternative virtual network provider. It is not frequently used in production at this time, so is not
covered in this guide.
Cumulus Linux complements OpenStack by delivering the same automated, self-service operational model to the network.
And since the underlying operating system is the same on the OpenStack nodes and the switches, the same automation,
monitoring and management tools can be used, greatly simplifying provisioning and operations.
Cumulus Linux offers powerful automation capabilities, by way of technologies such as ONIE, zero touch provisioning, PXE
and Puppet. The combination of bare metal hardware with a consistent Linux platform enables you to leverage automation
to deploy servers and networks together. Thus, you can use a unified set of tools to automate the installation and
configuration of both switches and servers. You can use a common automation framework that uses a simple config file to
install and configure an entire pod of switches and call OpenStack to install and configure the servers, all without any
human intervention.
Intended Audience for Network Design and Build

The rest of this document is aimed at the data center architect or administrator and is interested in evaluating a Proof of
Concept (PoC) or deploying a production cloud using Cumulus Linux and OpenStack.
The implementer is expected to have basic knowledge of Linux commands, logging in, navigating the file system and editing
files. Basic understanding of Layer 2 networking is assumed, such as interfaces, bonds (also known as LAGs), and bridges.
If you are using this guide to help you with setting up your OpenStack and Cumulus Linux environment, we assume you
have Cumulus Linux installed and licensed on switches from the Cumulus Linux HCL. Additional information on Cumulus
Linux software, licensing, and supported hardware may be found on cumulusnetworks.com or by contacting
sales@cumulusnetworks.com. This guide references the Icehouse release of OpenStack.
OpenStack Network Architecture in a PoC or

Small Test/Dev Environment
Network Architecture and Design Considerations
Figure 2 shows the network design of a typical Proof of Concept (PoC) or small test/dev environment running OpenStack.
Figure 2. PoC or Test/Dev OpenStack Environment

Figure 3 below details the connectivity for the hypervisor.
OPENSTACK NETWORK AR CHITECTURE IN A POC OR SMALL TEST/DEV EN VIRONMENT
Figure 3. Hypervisor Host Detail

The network architecture for an OpenStack PoC follows a simplified Top of Rack (ToR) access-tier-only design, all within
Layer 2, while the single services rack provides a gateway to the rest of the network, and also contains all the hypervisor
hosts. The services rack contains the OpenStack controller, and can optionally contain any load balancers, firewalls, and
other network services. Using Layer 2 fits well for OpenStack Icehouse, since it allows the use of nova-net or Neutron.
This design guide focuses on nova-net.
For optimal network performance, 10G switches are used for the ToR/access switches.
The network design employs multi-Chassis Link Aggregation (MLAG, the implementation of MLAG for Cumulus Linux) for
host path redundancy and link aggregation for network traffic optimization. The switches are paired into a single logical
switch for MLAG, with a peer LACP bond link between pair members. No breakout cables are used in this design.
A single OpenStack controller instance is assumed in this design.
Connectivity to external networks is assumed to be via a pair of links to routers, with a single upstream default route. These
links are connected to the leaf switches in the services rack, since it contains the controller. This guide assumes the
routers have been configured with VRRP or some other router redundancy protocol. If there is only one upstream router
link; connect it to either of the leaf switches in the services rack.
OpenStack Network Architecture in a Cloud

Data Center
Network Architecture
The network design of a typical cloud data center running OpenStack is shown in Figure 4.
Figure 4. Enterprise Data Center Network OpenStack Environment

The network architecture for an OpenStack data center follows the traditional hierarchical core, aggregation switch (also
known as spine), and access switch (also known as leaf) tiers, all within Layer 2, while a single services rack provides a
gateway to the rest of the network. The services rack contains the OpenStack controller, and can optionally contain any
load balancers, firewalls and other network services. Using Layer 2 fits well for OpenStack Icehouse, since it allows the use
of nova-net or Neutron. This design guide focuses on nova-net.
For optimal network performance, 40G switches are used for aggregation switches, and 10G switches are used for access
switches.
The network design employs MLAG for host and network path redundancy and link aggregation for network traffic
optimization. Switches are paired into logical switches for MLAG, with a peer LACP bond link between pair members. No
breakout cables are used in this design.
A single OpenStack controller instance is assumed in this design.
Connectivity to external networks is assumed to be via a pair of links to routers, with a single upstream default route. These
links are connected to the leaf switches in the services rack, which is the one that contains the controller. This guide
assumes the routers have been configured with VRRP or some other router redundancy protocol. If there is only one
upstream router link, connect it to either of the leaf switches in the services rack.
OPENSTACK NETWORK AR CHITECTURE IN A CLOU D DATA CENTER
Scaling Out
Scaling out the architecture involves adding more hosts to the access switch pairs, and then adding more access switches
in pairs as needed, as shown in Figure 5.
Figure 5. Adding Additional Switches

Once the limit for the aggregation switch pair has been reached, an additional network pod of aggregation/access switch
tiers may be added as shown in Figure 6. Each new pod has its own services rack and OpenStack controller.
Figure 6. Adding Network Pods/OpenStack Clusters
Out-of-Band Management
An important supplement to the high capacity production data network is the management network used to administer
infrastructure elements, such as network switches, physical servers, and storage systems. The architecture of these
networks vary considerably based on their intended use, the elements themselves, and access isolation requirements.
This solution guide assumes that a single Layer 2 domain is used to administer the network switches and management
interfaces on the controller and hypervisor hosts. These operations include imaging the elements, configuring them, and
monitoring the running system. This network is expected to host both DHCP and HTTP servers, such as isc-dhcp and
apache2, as well as provide DNS reverse and forward resolution. In general, these networks provide some means to
connect to the corporate network, typically a connection through a router or jump host.
Figure 7 below shows the logical and, where possible, the physical connections of each element as well as the services
required to realize this deployment.
Figure 7. Out-of-Band Management
10
BUILDING AN OPENSTAC K CLOUD WITH CUMULUS LINUX
Building an OpenStack Cloud with Cumulus

Linux
Minimum Hardware Requirements
For PoC, test/dev:
3x x86 servers, each with 2x 10G NICs + 1x 1G NIC

2x 48 port 10G switches, with 40G uplinks
Note that this design may be scaled up to 47 hypervisor nodes.

For a cloud data center:
5x x86 servers, each with 2x 10G NICs + 1x 1G NIC

4x 48 port 10G leaf switches, with 40G uplinks
2x 32 port 40G spine switches
Note that this design may be scaled up to 1535 hypervisor nodes. If required, additional OpenStack clusters may be
configured and connected to the core/external routers. OpenStack scalability limits will be hit before full scale is achieved.
11
Network Assumptions and Numbering

The network design for the full cloud deployment (6 switches, 5 servers) is shown in Figure 8 below. The PoC subset is just
the first pair of leafs and no spine switches. The implementation does not assume use of IPMI, as it is intended to
demonstrate the most generic network as possible.
Figure 8. Cloud Data Center Network Topology
Note that the peer bonds for MLAG support are always the last two interfaces on each switch. For spines, they are swp31
and swp32. For leafs, they are swp51 and swp52. The next to last two interfaces on each leaf are for the uplinks to
spine01 and spine02.
Also note that the same subnet is used for every MLAG peer pair. This is safe because the addresses are only used on the
link between the pairs. Routing protocols will not distribute these routes because they are part of the link-local
169.254.0.0/16 subnet.
The details for the switches, hosts, and logical interfaces are as follows:
leaf01
connected to
12
Logical Interface
Description
Physical Interfaces
leaf02
peerlink
peer bond utilized for MLAG traffic
swp51, swp52
leaf02
peerlink.4094
subinterface used for clagd communication
N/A
spine01, spine02
uplink
for MLAG between spine01 and spine02
swp49, swp50
external router
N/A
for accessing the outside network
swp48
leaf01
connected to
Logical Interface
Description
Physical Interfaces
multiple hosts
access ports
connect to hosts
swp1 through swp44
controller01
host01
bond to controller01 for host-to-switch MLAG
swp1
compute01
host02
bond to compute01 for host-to-switch MLAG
swp2
out-of-band
management
N/A
out-of-band management interface
eth0
leaf02
connected to
Logical Interface
Description
Physical Interfaces
leaf01
peerlink
swp51, swp52
leaf01
peerlink.4094
subinterface used for clagd

communication
N/A
spine01, spine02
uplink
for MLAG between spine01 and spine02
swp49, swp50
external router
N/A
for accessing the outside network
swp48
multiple hosts
access ports
connect to hosts
swp1 through swp44
controller01
host01
bond to controller01 for host-to-switch

MLAG
swp1
compute01
host02
bond to compute01 for host-to-switch

MLAG
swp2
out-of-band
management
N/A
eth0
leaf0N
connected to
Logical Interface
Description
Physical Interfaces
Repeat above configurations for each additional pair of leafs, minus the external router interfaces.
spine01
connected to
Logical Interface
Description
Physical Interfaces
spine02
peerlink
swp31, swp32
spine02
peerlink.4094
N/A
multiple leafs
leaf ports
connect to leaf switch pairs
swp1 through swp30
13
spine01
connected to
Logical Interface
Description
Physical Interfaces
leaf01, leaf02
downlink1
bond to another leaf switch pair
swp1, swp2
leaf03, leaf04
downlink2
bond to another leaf switch pair
swp3, swp4
out-of-band
management
N/A
eth0
Description
Physical Interfaces
spine02
connected to
Logical Interface
spine01
peerlink
swp31, swp32
spine01
peerlink.4094
N/A
multiple leafs
leaf ports
connect to leaf switches
swp1 through swp30
leaf01, leaf02
downlink1
bond to another peerlink group
swp1, swp2
leaf03, leaf04
downlink2
bond to another peerlink group
swp3, swp4
out-of-band
management
N/A
eth0
Both the manual and automated installation processes detailed below have some fixed parameters for things like VLAN
ranges and IP addresses. These can be changed in the config.txt file that you will copy onto a USB stick you will use for
automation purposes. If youre following the manual process and want to use different parameters, be careful to modify the
numbers in the configuration to match.
The parameters you are most likely to need to change are the external subnet and default route. Get this information from
whoever configured your access to the outside world (either the Internet or the rest of the data center network).
14
Parameter
Default Setting
OpenStack tenant VLANs
200-2000
OpenStack tenant subnets
10.10.TENANT#.0/24
External (public) VLAN
101
External (public) subnet
192.168.100.0/24
External default route
192.168.100.1
External IP of controller
192.168.100.2
External IP of first compute node
192.168.100.3
BUILDING AN OPENSTACK CLOUD WITH CUMULUS L INUX
Parameter
Default Setting
OpenStack API VLAN
102
OpenStack API subnet
10.254.192.0/20
OpenStack API IP of controller
10.254.192.1
OpenStack API IP of first compute node
10.254.192.2
Out-of-band management network
192.168.0.0/24
clagd peer VLAN
4094
clagd peer subnet
169.254.255.0/30
clagd system ID (base)
44:38:39:ff:00:01
Build Steps -- Automated

WARNING: This procedure creates a DHCP, ONIE, and PXE server on the switch/server management network, so be sure
that the management network is not bridged to any other network during the installation! Also make sure the version of
ONIE is 2014.02 or later.
The steps for an automated build out of an OpenStack cloud environment with Cumulus Linux switches are:
Step
1.
Tasks
Set up physical network.
Ensure all network switches and servers are properly cabled

and connected as shown in the network topology diagrams.
Verify connectivity.
The switches should boot into ONIE (however, if they
previously had an OS installed, uninstall it first).
The servers must be configured to PXE boot via their 1G
management NIC.
2.
Set up USB drive for bootstrapping.
Unzip the Cumulus Linux OpenStack bootstrap file into the

root of a formatted USB drive with at least 4 GB of free
space.
3.
Copy Cumulus Linux license.
Copy a valid Cumulus Linux license.txt to the root of the

USB drive.
Note that this license file needs to be enabled for
automation. To acquire an automation-enabled license,
consult this KB article.
4.
Optional: Customize your cloud.
Edit config.txt in the root of USB drive to customize your

cloud. The options are documented as comments in the
config file.
15
Step
Tasks
5.
Insert bootstrap USB key.
Insert the Cumulus Linux OpenStack bootstrap USB key into

one of the two spine switches (for the PoC config, one of the
two ToR/access switches).
6.
Power on racks.
Power on the rack(s), starting with the one containing the

spine switches.
7.
(Automated installation and configuration.)
Enjoy a beverage of your choice while the spine switches, leaf

switches, and servers are installed and configured
automatically. This will take between 60 and 90 minutes.
8.
Start VMs using the OpenStack Horizon Web UI.
Attach a laptop to the external network.

Point a Web browser at http://192.168.100.2/horizon and
log in (user: admin, pass: adminpw).
Start a VM in your new OpenStack cloud.
Note that you can also plug the laptop into the management
network, if that is easier.
Behind the Scenes of the Automated Build Out

If you are using the automated install process, you may be curious as to what is actually happening behind the scenes. The
actual install itself is very similar to the manual process detailed below. The differences are discovering the topology so that
each switch can be identified and configured along with setting up a DHCP server, a PXE server, an ONIE installation
environment, and a Puppet master configuration service. The provisioned USB stick includes an example configuration file
to specify the management IP space, bridge support, MLAG, VLAN tags, root password and related options. Finally, the build
process doesnt require Internet access or tools like IPMI. Here are all the steps occurring behind the scenes:
Phase 0 (Install and configure spine01 via USB)
When power is first applied, the switches all boot into ONIE in an OS installation discovery mode. Since spine01 has a
locally connected USB stick, it installs Cumulus Linux first without utilizing the management network. After Cumulus Linux is
installed, it uses zero touch provisioning (ZTP) to look for a configuration script, which it also sources from the USB stick.
This script installs the Cumulus Linux license, restarts switchd, brings up LLDP on all ports, and configures various
daemons to support booting additional switches with ONIE while offering a PXE environment for servers. This process takes
about 15 minutes. spine02 and the leaf switches will install and provision over the eth0 management network.
Phase 1 (Install and configure the other switches via the network)
Now that there is a DHCP server and ONIE is running on the remaining switches, you can find the Cumulus Linux image to
download and install from spine01 using the ZTP script. This script installs the license, restarts switchd, and brings up
LLDP on all ports, thus the topology can be fully determined. The other switches HTTP post their LLDP neighbor table to
spine01, which uses that information to determine what role (leaf or spine) the switch needs to play, and responds with a
set of configuration files customized for that role and switch this designation is based on comparing the received LLDP
data against the local spine01 LLDP data.
Each pair of switches (spine01/spine02, and the leaf pairs) is identified because they are dual-connected (where two swps
are mated to the same unique LLDP chassis ID). This dual connection becomes the MLAG peering bond. Leaf switches are
identified because they are plugged into spine01 and spine02 and take their name from the spine switch port they are
plugged into. For example, leaf01 is the switch plugged into swp1 on spine01, and leaf02 is the switch plugged into swp2
this enumeration pattern continues for all leaf switch hostnames.
16
Once the switches are identified, MLAG is configured, the bridge is created, and the VLAN range is provisioned on every
switch.
Phase 2 (Install and configure the servers using PXE and Puppet)
Now that the network is configured, spine01 offers a PXE and Puppet environment to install an OS on the servers. This
environment uses the same DHCP server, a TFTP server for fetching a network boot loader, and unattended preseed file to
install the OS. When the OS installation process completes, a Puppet agent is configured to receive additional post OS
install steps from the spine01 Puppet master. Puppet handles the OpenStack installation and configuration on the servers.
Build Steps -- Manual

To better understand how the installation and configuration happen, you can build out the cloud manually.
If you are building the PoC/Test/Dev configuration, skip step 5 (configure spine switches), as well as any steps that
reference spine01, spine02, leaf03, and leaf04.
The steps are:
Step
Tasks
Physical Network and Servers

1.
Set up physical network.
Rack and cable all network switches.

Install Cumulus Linux.
Install license.
2.
Basic physical network configuration.
Name switches.
Bring up out of band management ports.
Bring up front panel ports.
3.
Verify connectivity.
Use LLDP to ensure that the topology is as expected,

and that switches can communicate.
4.
Set up physical servers.
Install Ubuntu Server 14.04 on each of the servers.
Network Topology
5.
Configure spine switches.
Configure MLAG peer bond between the pair.
6.
Configure each pair of leaf switches.
Configure MLAG peer bond between the pair.
OpenStack
7.
Configure the OpenStack controller.
8.
Configure each compute node.
9.
Create tenant networks.
Install all components and configure.
17
Step
10.
Tasks
Start VMs using the OpenStack Horizon Web UI.
Attach a laptop to the external network.

Point a Web browser at http://192.168.100.2/horizon,
and log in (user: admin, pass: adminpw).
Start a VM in your new OpenStack cloud.
Note that you can also plug the laptop into the
management network, if that is easier.
1. Set Up Physical Network

Rack all servers and switches, and wire them together according to the wiring plan. Install Cumulus Linux, install your
license, and gain serial console access on each switch, as described in the Quick Start Guide of the Cumulus Linux
Documentation.
2. Basic Physical Network Configuration

Cumulus Linux contains a number of text editors, including nano, vi and zile; this guide uses nano in its examples.
First, edit the hostname file to change the hostname:
cumulus@cumulus$ nano /etc/hostname
Change cumulus to spine01, and save the file.
Make the same change to /etc/hosts:
cumulus@cumulus$ nano /etc/hosts
Change the first occurrence of cumulus on the line that starts with 127.0.1.1, then save the file. For example, for spine01,
you would edit the line to look like:
127.0.1.1
spine01 cumulus
Reboot the switch so the new hostname takes effect:

cumulus@cumulus$ sudo reboot
Configure Interfaces on Each Switch
By default, a switch with Cumulus Linux freshly installed has no switch port interfaces defined. Define the basic
characteristics of swp1 through swpN by creating stanza entries for each switch port (swp) in the
/etc/network/interfaces file. Each stanza should include the following statements:
auto <switch port name>
allow-<alias> <switch port name>
iface <switch port name>
The auto keyword above specifies that the interface is brought up automatically after issuing a reboot or service
networking restart command. The allow- keyword is a way to group interfaces so they can be brought up or down as
a group. For example, allow-hosts host01 adds the device host01 to the alias group hosts. Using ifup -allow=hosts brings up all of the interfaces with allow-hosts in their configuration.
On each switch, define the physical ports to be used according to the network topology as described in figure 8 and the
table that follows the figure.
For the leaf switches, the basic interface configuration is the range of interfaces from swp1 to swp52. On the spine
switches, the range is swp1 to swp32. For example, the configuration on leaf01 would look like:
18
cumulus@leaf01$ nano /etc/network/interfaces

.
.
# physical interface configuration
auto swp1
iface swp1
auto swp2
iface swp2
.
.
auto swp52
iface swp52
Additional attributes such as speed and duplex can be set. Refer to the Settings section of the Understanding Network
Interfaces chapter of the Cumulus Linux Documentation for more information.
Configure all leaf switches identically.
Instead of manually configuring each interface definition, you can programmatically define them using shorthand syntax
that leverages Python Mako templates. For information about configuring interfaces with Mako, read this knowledge base
article.
Once all configurations have been defined in the /etc/network/interfaces file, run the ifquery command to ensure
that all syntax is proper and the interfaces are created as expected:
cumulus@leaf01$ ifquery -a
auto lo
iface lo inet loopback
auto eth0
iface eth0
address 192.168.0.90/24
gateway 192.168.0.254
auto swp1
iface swp1
.
.
.
19
Once all configurations have been defined in the /etc/network/interfaces file, apply the configurations to ensure they
are loaded into the kernel. There are several methods for applying configuration changes depending on when and what
changes you want to apply:
Command
Action
sudo ifreload -a
Parse interfaces labeled with auto that have been added to or modified in the
configuration file, and apply changes accordingly.
Note: This command is disruptive to traffic only on interfaces that have been
modified.
sudo service networking restart
Restart all interfaces labeled with auto as defined in the configuration file,
regardless of what has or has not been recently modified.
Note: This command is disruptive to all traffic on the switch including the eth0
management network.
sudo ifup <swpX>
Parse an individual interface labeled with auto as defined in the configuration

file and apply changes accordingly.
Note: This command is disruptive to traffic only on interface swpX.
For example, on leaf01, to apply the new configuration to all changed interfaces labeled with auto:
cumulus@leaf01:~$ sudo ifreload -a
or individually:
cumulus@leaf01:~$ sudo ifup swp1
.
.
.
The above configuration in the /etc/network/interfaces file is persistent, which means the configuration applies even
after you reboot the switch. Another option to test network connectivity is to run a shell loop to bring up each front-panel
interface temporarily (until the next reboot), so that LLDP traffic can flow. This lets you verify the wiring is done correctly in
the next step:
cumulus@spine01$ for i in `grep '^swp' /var/lib/cumulus/porttab | cut -f1`; do
sudo ip link set dev $i up; done
Repeat the above steps on each of spine02, leaf01, leaf02, leaf03, and leaf04, changing the hostname
appropriately in each command or file.
20
3. Verify Connectivity
Back on spine01, use LLDP to verify that the cabling is correct, according to the cabling diagram:
cumulus@spine01$ sudo lldpctl | less
snip
------------------------------------------------------------------------------Interface:
swp31, via: LLDP, RID: 4, Time: 0 day, 00:12:10
Chassis:
ChassisID:
mac 44:38:39:00:49:0a
SysName:
spine02
SysDescr:
Cumulus Linux
Capability:
Bridge, off
Capability:
Router, on
Port:
PortID:
ifname swp31
PortDescr:
swp31
------------------------------------------------------------------------------Interface:
swp32, via: LLDP, RID: 4, Time: 0 day, 00:12:10
Chassis:
ChassisID:
mac 44:38:39:00:49:0a
SysName:
spine02
SysDescr:
Cumulus Linux
Capability:
Bridge, off
Capability:
Router, on
Port:
PortID:
ifname swp32
PortDescr:
swp32
------------------------------------------------------------------------------The output above shows only the last 2 interfaces, which you can see are correctly connected to the other spine switch,
based on the SysName field being spine02 (shown in green above). Verify that the remote-side interfaces are correct per
the wiring diagram, using the PortID field.
Note: Type q to quit less when you are done verifying.
Repeat the lldpctl command on spine02 to verify the rest of the connectivity.
4. Set Up Physical Servers

Install Ubuntu Server 14.04 LTS release on each server, as described in Ubuntus Installing from CD documentation.
During the install, configure the two drives into a RAID1 mirror, and then configure LVM on the mirror. Create a 1G swap
partition, and a 50G root partition. Leave the rest of the mirrors space free for the creation of VMs.
Make sure that openssh server is installed, and configure the management network such that you have out-of-band SSH
access to the servers. As part of the installation process you will create a user, which will have sudo access. Remember the
username and password you created for later.
Name the controller node (the one attached to swp1 on leaf01/leaf02) controller01 and name the compute nodes
compute01, compute02, and so on.
21
5. Configure Spine Switches

Enable MLAG Peering between Switches
An instance of the clagd daemon runs on each MLAG switch member to keep track of various networking information,
including MAC addresses, which are needed to maintain the peer relationship. clagd communicates with its peer on the
other switch across a Layer 3 interface between the two switches. This Layer 3 network should not be advertised by routing
protocols, nor should the VLAN be trunked anywhere else in the network. This interface is designed to be a keep-alive
reachability test and for synchronizing the switch state across the directly attached peer bond.
Create the VLAN subinterface for clagd communication and assign an IP address for this subinterface. A unique .1q tag is
recommended to avoid mixing data traffic with the clagd control traffic.
To enable MLAG peering between switches, configure clagd on each switch by creating a peerlink subinterface in
/etc/network/interfaces with a unique .1q tag. Set values for the following parameters under the peerlink
subinterface:
address. The local IP address/netmask of this peer switch.
o Cumulus Networks recommends you use a link local address; for example 169.254.1.X/30.
clagd-enable. Set to yes (default).
clagd-peer-ip. Set to the IP address assigned to the peer interface on the peer switch.
clagd-backup-ip Set to an IP address on the peer switch reachable independently of the peerlink. For
example, the management interfaces or a routed interface that does not traverse the peerlink.
clagd-sys-mac. Set to a unique MAC address you assign to both peer switches.
o Cumulus Networks recommends you use addresses within the Cumulus Linux reserved range of
44:38:39:FF:00:00 through 44:38:39:FF:FF:FF.
On both spine switches, edit /etc/network/interfaces and add the following sections at the bottom:
#Bond for the peerlink. MLAG control traffic and data when links are down.
auto peerlink
iface peerlink
bond-slaves swp31 swp32
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
bond-xmit-hash-policy layer3+4
mtu 9000
On spine01, add a VLAN for the MLAG peering communications:
#VLAN for the MLAG control traffic.
auto peerlink.4094
iface peerlink.4094
address 169.254.255.1/30
clagd-enable yes
clagd-peer-ip 169.254.255.2
clagd-backup-ip 192.168.0.95/24
clagd-sys-mac 44:38:39:ff:00:00
22
On spine02, add a VLAN for the MLAG peering communications. Note the change of the last octet in the address and
clagd-peer-ip lines:
auto peerlink.4094
iface peerlink.4094
address 169.254.255.2/30
clagd-enable yes
On both spine switches, bring up the peering interfaces. The --with-depends option tells ifup to bring up the peer first,
since peerlink.4094 depends on it:
cumulus@spine0N:~$ sudo ifup --with-depends peerlink.4094
On spine01, verify that you can ping spine02:
cumulus@spine01$ ping -c 3 169.254.255.2
PING 169.254.255.2 (169.254.255.2) 56(84) bytes of data.
64 bytes from 169.254.255.2: icmp_req=1 ttl=64 time=0.716 ms
--- 169.254.255.2 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.588/0.661/0.716/0.061 ms
Now on both spine switches, verify that the peers are connected:
cumulus@spine01:~$ clagctl
The peer is alive
Peer Priority, ID, and Role:
Our Priority, ID, and Role:
Peer Interface and IP:
Backup IP:
System MAC:
32768 44:38:39:00:49:87 secondary

32768 44:38:39:00:49:06 primary
peerlink.4094 169.254.255.2
192.168.0.95 (active)
44:38:39:ff:00:00
The MAC addresses in the output will be different depending on the MACs issued to your hardware.
Now that the spines are peered, create the bonds for the connections to the leaf switches. On both spine switches, edit
/etc/network/interfaces and add the following at the end:
#Bonds down to the pairs of leafs.
auto downlink1
allow-leafs downlink1
iface downlink1
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
23
mtu 9000
clag-id 1
auto downlink2
iface downlink2
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
mtu 9000
clag-id 2
You can add more stanzas for more pairs of leaf switches as needed, modifying the sections in green above. For example,
to add a third stanza, youd use downlink3; the corresponding swp interfaces would be swp5 and swp6 and clag-id 3.
Bridge together the MLAG peer bond and all the leaf bonds. On both switches, edit /etc/network/interfaces and add
the following at the end:
#Bridge that connects our peer and downlinks to the leafs.
auto bridge
iface bridge
bridge-vlan-aware yes
bridge-ports peerlink downlink1 downlink2
bridge-stp on
bridge-vids 100-2000
mstpctl-treeprio 12288
If you added more downlink# interfaces in the previous step, add them to the bridge-ports line, at the end of the line.
If youre familiar with the traditional Linux bridge mode, you may be surprised that we called the bridge bridge instead of
br0. The reason is that were using the new VLAN-aware Linux bridge mode in this example, which doesnt require multiple
bridge interfaces for common configurations. It trades off some of the flexibility of the traditional mode in return for
supporting very large numbers of VLANs. See the Cumulus Linux Documentation for more information on the two bridging
modes supported in Cumulus Linux.
Finally, on both spine01 and spine02, bring up all the interfaces, bonds and bridges. The --with-depends option tells
ifup to bring up any down interfaces that are needed by the bridge:
cumulus@spine0N:~$ sudo ifup --with-depends bridge
6. Configure Each Pair of Leaf Switches

On each leaf switch, edit /etc/network/interfaces, and add the following sections at the bottom:
#Bond for the peer link. MLAG control traffic and data when links are down.
auto peerlink
iface peerlink
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
24
bond-lacp-rate 1
bond-min-links 1
mtu 9000
On odd numbered leaf switches, add a VLAN for the MLAG peering communications. Note that the last octet of the clagdsys-mac must be the same for each switch in a pair, but incremented for subsequent pairs. For example, leaf03 and
leaf04 should have 03 as the last octet:
auto peerlink.4094
iface peerlink.4094
address 169.254.255.1/30
clagd-enable yes
On even numbered leaf switches, add a VLAN for the MLAG peering communications. Note the change of the last octet in
the address and clagd-sys-peer-ip lines. Also note that for subsequent pairs of switches, the last octet of clagdsys-mac must match as described for the odd-numbered switches:
auto peerlink.4094
iface peerlink.4094
address 169.254.255.2/30
clagd-enable yes
On each leaf switch, bring up the peering interfaces:
cumulus@leaf0N:~$ sudo ifup --with-depends peerlink.4094
On each odd numbered leaf switch, verify that you can ping its corresponding even-numbered leaf switch:
cumulus@leaf0N:~$ ping -c 3 169.254.255.2
PING 169.254.255.2 (169.254.255.2) 56(84) bytes of data.
--- 169.254.255.2 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2001ms
Now, on each leaf switch, verify that the peers are connected:
cumulus@leaf0N:~$ clagctl
The peer is alive
Peer Priority, ID, and Role: 32768 6c:64:1a:00:39:5a primary
Our Priority, ID, and Role: 32768 6c:64:1a:00:39:9b secondary
Peer Interface and IP: peerlink.4094 169.254.255.2
25
Backup IP: 192.168.0.91 (active)

System MAC: 44:38:39:ff:00:02
Now that the leafs are peered, create the uplink bonds connecting the leafs to the spines. On each leaf switch, edit
#Bond up to the spines.
auto uplink
iface uplink
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
mtu 9000
clag-id 1000
On each leaf switch, bring up the bond up to the spine:
cumulus@leaf0N:~$ sudo ifup --with-depends uplink
On each leaf switch, verify that the link to the spine is up:
cumulus@leaf0N:~$ ip link show dev uplink
2: uplink: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP
qlen 1000
link/ether 44:38:39:00:49:06 brd ff:ff:ff:ff:ff:ff
The UP,LOWER_UP (shown in green above) line means that the bond itself is up (UP), and slave interfaces (swp49 and
swp50) are up (LOWER_UP).
On leaf01 and leaf02, and only leaf01 and leaf02, configure the interface going to the core/external routers. These are
part of the external VLAN (101), but are themselves untagged. Edit /etc/network/interfaces and add the following at
the end:
auto swp48
iface swp48
bridge-access 101
mtu 9000
Create the bonds for the connections to the servers. On each leaf switch, edit /etc/network/interfaces and add the
following at the end:
#Bonds down to the host. Only one swp, because the other swp is on the peer
switch.
auto host01
allow-hosts host01
iface host01
bond-slaves swp1
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
26
bond-lacp-bypass-allow 1
mstpctl-portadminedge yes
mstpctl-bpduguard yes
clag-id 1
Repeat the above stanza for each front panel port that has servers attached. Youll need to adjust host01, swp1 and the
value for clag-id everywhere they appear (in green). For example, for swp2, change each host01 to host02 and swp1 to
swp2, and change clag-id from 1 to 2.
Bridge together the MLAG peer bond, the uplink bond, and all the leaf bonds. On each leaf switch, edit
#Bridge that connects our peer, uplink to the spines, and the hosts.
auto bridge
iface bridge
bridge-ports uplink swp48 peerlink host01 host02 host03
bridge-stp on
If you added more host# interfaces in the previous step, add them to the bridge-ports line, at the end of the line. Note
that swp48 (in green above) should only be present on leaf01 and leaf02, not on subsequent leafs.
Finally, on each leaf switch, bring up all the interfaces, bonds and bridges:
cumulus@leaf0N:~$ sudo ifup --with-depends bridge
7. Configure the OpenStack Controller

The server connected to swp1 on leaf01 and leaf02 will be the OpenStack controller. It will manage all the other servers,
which run VMs. ssh into it as the user you configured when installing the OS.
Configure the uplinks. The server has 2 10G interfaces, in this example they are called p1p1 and p2p2. They may be
named differently on other server hardware platforms.
The ifenslave package must be installed for bonding support, and the vlan package must be installed for VLAN support.
To install them, run:
cumulus@controller01$ sudo apt-get install ifenslave vlan
For the bond to come up, the bonding driver needs to be loaded. Similarly, for VLANs, the 802.1q driver must be loaded. So
that they will be loaded automatically at boot time, edit /etc/modules and add the following to the end:
bonding
8021q
Now load the modules:
cumulus@controller01$ sudo modprobe bonding
cumulus@controller01$ sudo modprobe 8021q
Edit /etc/network/interfaces to add the following at the end:
#The bond, one subinterface goes to each leaf.
auto bond0
iface bond0 inet manual
up ip link set dev $IFACE up
27
down ip link set dev $IFACE down

bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-slaves none
#First 10G link.
auto p1p1
iface p1p1 inet manual
bond-master bond0
#Second 10G link.
auto p1p2
bond-master bond0
#OpenStack API VLAN.
auto bond0.102
iface bond0.102 inet static
address 10.254.192.1
netmask 255.255.240.0
#External VLAN.
auto bond0.101
address 192.168.100.2
netmask 255.255.255.0
gateway 192.168.100.1
Note that Ubuntu uses ifupdown, while Cumulus Linux uses ifupdown2. The configuration format is similar, but many
constructs that work on the switch will not work in Ubuntu.
Now bring up the interfaces:
cumulus@controller01$ sudo ifup -a
Verify that the VLAN interface is UP and LOWER_UP:
cumulus@controller01$ sudo ip link show bond0.102
9: bond0.102@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP mode DEFAULT group default
link/ether 90:e2:ba:7c:28:28 brd ff:ff:ff:ff:ff:ff
Add a hostname alias for the controller. Edit /etc/hosts and add the following at the end:
10.254.192.1 controller
In the following sections, read the notes after the links before you follow the OpenStack install guide sections, as they
contain important additional information youll need.
Install the database server (MySQL), using the directions in the OpenStack database install guide. Note that youll have to
use sudo when installing the packages. Make sure to remember the database password you chose for later. Use
10.254.192.1 for the bind-address when configuring MySQL. When running mysql_secure_installation the root
password is the MySQL password you chose. There is no need to reset it during mysql_secure_installation.
28
Install the message broker (RabbitMQ) using the directions in the OpenStack message queue install guide. Note that youll
have to use sudo when installing the packages. Make sure to remember the rabbitmq password you chose, as you will
need it later.
Install the Keystone authentication service using the directions in the OpenStack keystone install guide. Note that youll
have to use sudo with the commands in that guide. Also note that creating the database tables can take a couple of
minutes. Make sure to remember the admin_token you generated using openssl for later.
Create some users, roles, tenants, services and endpoints using the directions in the OpenStack users install guide and
services install guide. Dont use sudo for these commands. For a simple test deployment, we recommend admin/adminpw
and demo/demopw for the usernames and passwords for the admin user and demo user.
Create an OpenStack RC file to set the various environment variables needed to run OpenStack commands. This simplifies
running commands as various OpenStack users; just source the rc file any time you want to change users. The directions
are in the Openstack openrc install guide. Dont use sudo for these commands. If you used admin/adminpw for your admin
user, replace ADMIN_PASS with adminpw.
Verify that Keystone is operating properly using the directions in the OpenStack Keystone verification install guide. Dont
use sudo for these commands. You dont need to recreate the admin-openrc.sh file you created previously in the
openrc install guide.
Install the Glance image storage service using the directions in the OpenStack glance install guide. Note that command
prompts in that guide that end with a # symbol must be run with sudo, while command prompts that end with the $ symbol
do not.
Import a demo Linux image into the Glance inventory, so that you can have an OS to start in VMs to demonstrate
OpenStack. This first command assumes your server has direct access to the Internet.
cumulus@controller01$ wget http://cdn.download.cirros-cloud.net/0.3.2/cirros0.3.2-x86_64-disk.img
If you need an HTTP proxy to access the Internet from your environment, you can modify the previous command:
cumulus@controller01$ http_proxy="http://MY.HTTP.PROXY/" wget
http://cdn.download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img
Install the downloaded Linux image into Glance:
cumulus@controller01$ glance image-create --name="Cirros" --disk-format=qcow2 -container-format=bare --is-public=true --file cirros-0.3.2-x86_64-disk.img
Install the Nova compute controller service using the directions in the OpenStack Nova install guide and the Nova network
install guide. Note that command prompts in that guide that end with a # symbol must be run with sudo, while command
prompts that end with a $ symbol do not. Youll need to replace the example 10.0.0.11 IP addresses in the my_ip,
vncserver_listen and vncserver_proxyclient_address fields with 10.254.192.1.
Install the Horizon Web dashboard packages, then remove the openstack-dashboard-ubuntu-theme package, as it
may cause rendering issues:
cumulus@controller01$ sudo apt-get install apache2 memcached libapache2-mod-wsgi
openstack-dashboard
cumulus@controller01$ sudo apt-get remove --purge openstack-dashboard-ubuntutheme
It is not a good idea to expose the Horizon Web interface to untrusted networks without hardening the configuration.
29
8. Configure Each Compute Node

The remaining servers are all compute nodes. They run VMs, as directed by the controller. ssh into them as the user you
configured when installing the OS. In this example, that user is called cumulus.
Enable IP forwarding. Since youre using multi-host mode, each compute node needs to use NAT for floating IP addresses.
Edit /etc/sysctl.conf and uncomment the following line:
#net. ipv4.ip_forward=1
This enables forwarding on reboots, but not immediately, so enable it right now as well:
cumulus@compute0N:~$ sudo sysctl -w net.ipv4.ip_forward=1
Configure the uplinks. The server has 2 10G interfaces; in this example they are called p1p1 and p2p2. They may be
named differently on other server hardware platforms.
The ifenslave package must be installed for bonding support, and the vlan package must be installed for VLAN support.
To install them, run:
cumulus@compute01$ sudo apt-get install ifenslave vlan
For the bond to come up, the bonding driver needs to be loaded. Similarly, for VLANs, the 802.1q driver must be loaded.
So that they will be loaded automatically at boot time, edit /etc/modules and add the following to the end:
bonding
8021q
Now load the modules:
cumulus@compute0N:~$ sudo modprobe bonding
cumulus@compute0N:~$ sudo modprobe 8021q
Edit /etc/network/interfaces and add the following at the end:
#The bond, one subinterface goes to each leaf.
auto bond0
iface bond0 inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-slaves none
#First 10G link.
auto p1p1
bond-master bond0
#Second 10G link.
auto p1p2
bond-master bond0
#OpenStack API VLAN.
30
auto bond0.102
address 10.254.192.2
netmask 255.255.240.0
#External network access VLAN.
auto bond0.101
address 192.168.100.3
netmask 255.255.240.0
gateway 192.168.100.1
Youll need to increment the API VLANs IP address (show in green above, on bond0.102) for each compute node. Youll
also need to increment the external VLANs IP address (show in green above, on bond0.101). The examples given above
are for compute01. For compute02, you would use 10.254.192.3 and 192.168.100.4.
Note: Ubuntu uses ifupdown, while Cumulus Linux uses ifupdown2. The configuration format is similar, but many
advanced configurations that work on the switch will not work in Ubuntu.
Now bring up the interfaces:
cumulus@compute0N:~$ sudo ifup -a
Verify that the VLAN interface is UP and LOWER_UP:
cumulus@compute0N:~$ sudo ip link show bond0.102
9: bond0.102@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP mode DEFAULT group default
link/ether 90:e2:ba:7c:28:28 brd ff:ff:ff:ff:ff:ff
Add a hostname alias for the controller. Edit /etc/hosts and add the following at the end:
10.254.192.1 controller
Verify that this node can talk to the controller over the API VLAN:
cumulus@compute0N:~$ ping -c 3 controller
PING controller (10.254.192.1) 56(84) bytes of data.
64 bytes from controller (10.254.192.1): icmp_seq=1 ttl=64 time=0.229 ms
--- controller ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 1998ms
Install the Nova compute node service using the directions in the OpenStack Nova compute install guide. Note that
command prompts in that guide that end with a # symbol must be run with sudo, while command prompts that end with a
$ symbol do not. If the step in the directions that checks for hardware acceleration for virtual machines reports that your
server does not have it, you can disable this in the BIOS. Note that youll need to replace the example 10.0.0.31 IP address
in the my_ip and vncserver_proxyclient_address fields with this nodes API IP address. For compute01, that would
be 10.254.192.2.
31
Install the packages required for nova-net:

cumulus@compute0N:~$ sudo apt-get install nova-network nova-api-metadata
Configure nova-net in multi-host mode. Edit /etc/nova/nova.conf and add the following to the [DEFAULT] section:
network_api_class = nova.network.api.API
security_group_api = nova
firewall_driver = nova.virt.libvirt.firewall.IptablesFirewallDriver
network_manager = nova.network.manager.VlanManager
network_size = 254
allow_same_net_traffic = True
multi_host = True
send_arp_for_ha = True
share_dhcp_address = True
force_dhcp_release = True
public_interface = bond0.101
vlan_interface = bond0
Now restart the services:
cumulus@compute0N:~$ sudo service nova-compute restart
cumulus@compute0N:~$ sudo service nova-network restart
cumulus@compute0N:~$ sudo service nova-api-metadata restart
Repeat the all the steps in this section on the rest of the compute nodes, changing the hostnames and IP addresses
appropriately in each command or file.
32
9. Create Tenant Networks

On the controller, create some VLAN networks and associate subnets with them.
cumulus@controller01$ for i in `seq 1 100`; do nova network-create vmnet$i -fixed-range-v4 10.10.$i.0/24 --vlan $((200+i)) --multi-host T; done
This takes quite a while, so you are creating only 100 VLANs in this example.
On the controller, configure the floating (external) IP address range to use the entire external subnet:
cumulus@controller01$ nova floating-ip-bulk-create --interface bond0.101
192.168.100.0/24
On the controller, remove the default route and the external IP address of the controller and each compute node from the
pool, since they are already in use. In this example, the default route is 192.168.100.1, the controller is 192.168.100.2,
and the compute nodes are 192.168.100.3 through 192.168.100.6. Since the range is continuous, you can use a simple
shell loop to remove 192.168.100.1 192.168.100.6 all at once. If you added more compute nodes, youll need to
change the 6 to the highest numbered external IP address assigned to any compute node:
cumulus@controller01$ for i in `seq 1 6`; do nova floating-ip-bulk-delete
192.168.100.$i; done
10. Start VMs Using the OpenStack Horizon Web UI

Point a Web browser at http://192.168.100.2/horizon and log in (user: admin, password: adminpw). Start a VM in your
new OpenStack cloud.
The documentation describing the Horizon Web UI is available here.
Note that because the install is using nova-net, there is no Network tab. A VLAN is dynamically allocated to a tenant when
that tenant starts its first VM.
33
Conclusion
Summary
The fundamental abstraction of hardware from software and providing customers a choice through a hardware agnostic
approach is core to the philosophy of Cumulus Networks and fits very well within the software-centric, commodity hardware
friendly design of OpenStack.
Just as OpenStack users have choice in server compute and storage, they can tap the power of Open Networking and
select from a broad range of switch providers running Cumulus Linux.
Choice and CapEx savings are only the beginning. OpEx savings come from agility through automation. Just as OpenStack
orchestrates the cloud by enabling the automated provisioning of hosts, virtual networks, and VMs through the use of APIs
and interfaces, Cumulus Linux enables network and data center architects to leverage automated provisioning tools and
templates to define and provision physical networks.
References
Article/Document
URL
OpenStack Documentation
http://docs.openstack.org/icehouse/installguide/install/apt/content/index.html
Database Install Guide

Message Queue Install Guide
Keystone Install Guide
Users Install Guide
Services Install Guide
Openrc Install Guide
Keystone Verification Install Guide
Glance Install Guide
Nova Install Guide
Nova Network Install Guide
Nova Compute Install Guide
Cumulus Linux Documentation
34
Quick Start Guide

Understanding Network Interfaces
MLAG
LACP Bypass
Authentication, Authorization, and Accounting
Zero Touch Provisioning
http://docs.cumulusnetworks.com
CONCLUSION
Article/Document
URL
Cumulus Linux KB Articles
https://support.cumulusnetworks.com/hc/enus/articles/202868023
Configuring /etc/network/interfaces with

Mako
Demos and Training, specifically
Implementing the OpenStack Design Guide in
the Cumulus Workbench
Installing collectd and graphite
Manually Putting All Switch Ports into a Single
VLAN
https://support.cumulusnetworks.com/hc/enus/sections/200398866
Cumulus Linux Product Information
Software Pricing
Hardware Compatibility List
http://cumulusnetworks.com/product/pricing/
http://cumulusnetworks.com/support/linux-hardwarecompatibility-list/
Cumulus Linux Downloads
http://cumulusnetworks.com/downloads/
Cumulus Linux Repository
http://repo.cumulusnetworks.com
Cumulus Networks GitHub Repository
https://github.com/CumulusNetworks/
35
Appendix A: Example /etc/network/interfaces

Configurations
leaf01
cumulus@leaf01$ cat /etc/network/interfaces
auto eth0
iface eth0
address 192.168.0.90/24
gateway 192.168.0.254
auto swp1
iface swp1
mtu 9000
auto swp2
iface swp2
mtu 9000
auto swp3
iface swp3
mtu 9000
.
.
auto swp48
iface swp48
bridge-access 101
mtu 9000
.
.
auto swp52
iface swp52
mtu 9000
# peerlink bond for clag
auto peerlink
iface peerlink
bond-mode 802.3ad
bond-miimon 100
36
APPENDIX A: EXAMPLE /ETC/NETWORK/INTERFA CES CONFIGURATIONS
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
mtu 9000
auto peerlink.4094
iface peerlink.4094
address 169.254.255.1/30
auto uplink
iface uplink
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
mtu 9000
clag-id 1000
switch.
auto host01
allow-hosts host01
iface host01
bond-slaves swp1
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
clag-id 1
auto host02
allow-hosts host02
iface host02
bond-slaves swp2
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
37
bond-min-links 1
clag-id 2
auto host03
allow-hosts host03
iface host03
bond-slaves swp3
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
clag-id 3
auto bridge
iface bridge
bridge-ports uplinks swp48 peerlink host01 host02 host03
bridge-stp on
38
leaf02
auto eth0
iface eth0
address 192.168.0.91/24
gateway 192.168.0.254
auto swp1
iface swp1
mtu 9000
auto swp2
iface swp2
mtu 9000
auto swp3
iface swp3
mtu 9000
.
.
auto swp48
iface swp48
bridge-access 101
mtu 9000
.
.
auto swp52
iface swp52
mtu 9000
auto peerlink
iface peerlink
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
mtu 9000
auto peerlink.4094
39
iface peerlink.4094
address 169.254.255.2/30
auto uplink
iface uplink
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
mtu 9000
clag-id 1000
switch.
auto host01
allow-hosts host01
iface host01
bond-slaves swp1
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
clag-id 1
auto host02
allow-hosts host02
iface host02
bond-slaves swp2
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
clag-id 2
40
auto host03
allow-hosts host03
iface host03
bond-slaves swp3
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
clag-id 3
auto bridge
iface bridge
bridge-stp on
41
leaf03
auto eth0
iface eth0
address 192.168.0.92/24
gateway 192.168.0.254
auto swp1
iface swp1
mtu 9000
auto swp2
iface swp2
mtu 9000
auto swp3
iface swp3
mtu 9000
.
.
auto swp52
iface swp52
mtu 9000
auto peerlink
iface peerlink
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
mtu 9000
auto peerlink.4094
iface peerlink.4094
address 169.254.255.1/30
42

auto uplink
iface uplink
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
mtu 9000
clag-id 1000
switch.
auto host01
allow-hosts host01
iface host01
bond-slaves swp1
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
clag-id 1
auto host02
allow-hosts host02
iface host02
bond-slaves swp2
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
clag-id 2
auto host03
allow-hosts host03
iface host03
bond-slaves swp3
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
43
bond-min-links 1
clag-id 3
auto bridge
iface bridge
bridge-stp on
44
leaf04
auto eth0
iface eth0
address 192.168.0.93/24
gateway 192.168.0.254
auto swp1
iface swp1
mtu 9000
auto swp2
iface swp2
mtu 9000
auto swp3
iface swp3
mtu 9000
.
.
auto swp52
iface swp52
mtu 9000
auto peerlink
iface peerlink
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
mtu 9000
auto peerlink.4094
iface peerlink.4094
address 169.254.255.2/30
45

auto uplink
iface uplink
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
mtu 9000
clag-id 1000
switch.
auto host01
allow-hosts host01
iface host01
bond-slaves swp1
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
clag-id 1
auto host02
allow-hosts host02
iface host02
bond-slaves swp2
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
bond-min-links 1
clag-id 2
auto host03
allow-hosts host03
iface host03
bond-slaves swp3
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 1
46
bond-min-links 1
clag-id 3
auto bridge
iface bridge
bridge-stp on
47
spine01
cumulus@spine01$ sudo vi /etc/network/interfaces
auto eth0
iface eth0
address 192.168.0.94/24
gateway 192.168.0.254
auto swp1
iface swp1
mtu 9000
auto swp2
iface swp2
mtu 9000
auto swp3
iface swp3
mtu 9000
.
.
.
auto swp32
iface swp32
mtu 9000
auto peerlink
iface peerlink
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
auto peerlink.4094
iface peerlink.4094
address 169.254.255.1/30
clagd-enable yes
48
# leaf01-leaf02 downlink
auto downlink1
iface downlink1
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
clag-id 1
auto downlink2
iface downlink2
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
clag-id 2
auto bridge
iface bridge
bridge-stp on
49
spine02
cumulus@spine02$ sudo vi /etc/network/interfaces
auto eth0
iface eth0
address 192.168.0.95/24
gateway 192.168.0.254
auto swp1
iface swp1
mtu 9000
auto swp2
iface swp2
mtu 9000
auto swp3
iface swp3
mtu 9000
.
.
.
auto swp32
iface swp32
mtu 9000
auto peerlink
iface peerlink
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1

auto peerlink.4094
iface peerlink.4094
address 169.254.255.2/30
clagd-enable yes
50
auto downlink1
iface downlink1
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
clag-id 1
auto downlink2
iface downlink2
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
clag-id 2
auto bridge
iface bridge
bridge-stp on
51
Appendix B: Network Setup Checklist

Tasks
Considerations
1. Set up physical network.

Select network switches
Refer to the HCL and hardware guides at http://cumulusnetworks.com/support/hcl.
Plan cabling
Refer to KB article, Suggested Transceivers and Cables:

https://support.cumulusnetworks.com/hc/en-us/articles/202983783.
Generally, higher number ports on a switch are reserved for uplink ports, so:
Assign downlinks or host ports to the lower end, like swp1, swp2
Reserve higher number ports for network
Reserve highest ports for MLAG peer links
Connect all console ports.
Install Cumulus Linux
Obtain the latest version of Cumulus Linux.

Obtain license key, which is separate from Cumulus Linux OS distribution.
To minimize variables and aid in troubleshooting, use identical versions across switches
same version X.Y.Z, packages, and patch levels.
See the Quick Start Guide in the Cumulus Linux Documentation.
2. Basic Physical Network Configuration

Reserve management
space
Reserve pool of IP addresses.

Define hostnames and DNS.
RFC 1918 should be used where possible. Note: we used RFC 6598 in our automation
explicitly to avoid the use of any existing RFC 1918 deployments.
Edit configuration files
Apply standards and conventions to promote similar configurations. For example, place
stanzas in the same order in configuration files across switches and specify the child
interfaces before the parent interfaces (so a bond member appears earlier in the file than
the bond itself, for example). This allows for standardization and easier maintenance and
troubleshooting, and ease of automation and the use of templates.
Consider naming conventions for consistency, readability, and manageability. Doing so
helps facilitate automation. For example, call your leaf switches leaf01 and leaf02 rather
than leaf1 and leaf02.
Use all lowercase for names
Avoid characters that are not DNS-compatible.
Define child interfaces before using them in parent interfaces. For example, create the
member interfaces of a bond before defining the bond interface itself.
Define switch ports (swp)

in /etc/network/interfaces
on a switch
52
Instantiate swp interfaces for using the ifup and ifdown commands.
APPENDIX B: NETWORK SETUP CHECKLIST
Tasks
Considerations
Set speed and duplex
These settings are dependent on your network.
3. Verify connectivity.
Use LLDP
(Link Layer Discovery
Protocol)
LLDP is useful to debug or verify cabling between directly attached switches. By default,
Cumulus Linux listens and advertises LLDP packages on all configured Layer 3 routed or
Layer 2 access ports. LLDP is supported on tagged interfaces or those configured as an
802.1q sub interface. The command lldpctl will display a dump of the connected interfaces.
4. Set up physical servers.

Install Ubuntu
5. Configure spine switches.

Create peer link bond
between pair of switches
Assign IP address for clagd peerlink. Consider using a link local address (RFC 3927,
169.254/16) to avoid advertising, or an RFC 1918 private address.
Use a very high number VLAN if possible to separate the peer communication traffic from
typical VLANs handling data traffic. Valid VLAN tags end at 4096.
Enable MLAG
Set up MLAG in switch pairs. Theres no particular order necessary for connecting pairs.
Assign clagd-sys-mac
Assign a unique clagd-sys-mac value per pair. This value is used for spanning tree
calculation, so assigning unique values will prevent overlapping MAC addresses.
Assign priority
Use the range reserved for Cumulus Networks: 44:38:39:FF:00:00 through

44:38:39:FF:FF:FF.
Define primary and secondary switches in an MLAG switch pair, if desired. Otherwise, by
default the switches will elect a primary switch on their own. Set priority if you want to
explicitly control which switches are designated primary switches.
6. Configure each pair of leaf switches.

Repeat steps for
configuring spine switches
Steps for leaf switches are similar.
Connect to core routers
7. Configure the OpenStack controller

Install all components and
configure
53
8. Configure each compute node

Enable IP forwarding
Configure uplinks
Load modules
9. Create tenant networks.
Create VLANs
Configure floating IP
address range
Remove default route and
external IP addresses of
controller and compute
nodes from pool
10. Start VMs using the OpenStack Horizon Web UI
Log into admin web UI
54
There is no Network tab

OpenStack Cumulus Linux Validated Design Guide

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

OpenStack Cumulus Linux Validated Design Guide

Transféré par

Droits d'auteur :

Formats disponibles

OpenStack and Cumulus Linux

Validated Design Guide

Deploying OpenStack with Network Switches Running Cumulus Linux

OPENSTACK AND CUMULU S LINUX: VALIDATED DESIGN GUI DE

About Cumulus Networks

OPENSTACK AND CUMULU S LINUX: VALIDATED DESIGN GUI DE

OpenStack with Cumulus Linux

Enabling Choice of Hardware in the Data Center

Combined Solution Using OpenStack and Cumulus Linux

OPENSTACK WITH CUMUL US LINUX

Figure 1. OpenStack and Cumulus Linux

Driving Towards Operational Efficiencies

Nova, which manages VMs and their virtual network connectivity.

OPENSTACK AND CUMULU S LINUX: VALIDATED DESIGN GUI DE

Intended Audience for Network Design and Build

OpenStack Network Architecture in a PoC or

Figure 2. PoC or Test/Dev OpenStack Environment

OPENSTACK NETWORK AR CHITECTURE IN A POC OR SMALL TEST/DEV EN VIRONMENT

Figure 3. Hypervisor Host Detail

OPENSTACK AND CUMULU S LINUX: VALIDATED DESIGN GUI DE

OpenStack Network Architecture in a Cloud

Figure 4. Enterprise Data Center Network OpenStack Environment

OPENSTACK NETWORK AR CHITECTURE IN A CLOU D DATA CENTER

Figure 5. Adding Additional Switches

Figure 6. Adding Network Pods/OpenStack Clusters

OPENSTACK AND CUMULU S LINUX: VALIDATED DESIGN GUI DE

Figure 7. Out-of-Band Management

BUILDING AN OPENSTAC K CLOUD WITH CUMULUS LINUX

Building an OpenStack Cloud with Cumulus

3x x86 servers, each with 2x 10G NICs + 1x 1G NIC

Note that this design may be scaled up to 47 hypervisor nodes.

5x x86 servers, each with 2x 10G NICs + 1x 1G NIC

OPENSTACK AND CUMULU S LINUX: VALIDATED DESIGN GUI DE

Network Assumptions and Numbering

Figure 8. Cloud Data Center Network Topology

peer bond utilized for MLAG traffic

subinterface used for clagd communication

for MLAG between spine01 and spine02

for accessing the outside network

BUILDING AN OPENSTAC K CLOUD WITH CUMULUS LINUX

swp1 through swp44

bond to controller01 for host-to-switch MLAG

bond to compute01 for host-to-switch MLAG

out-of-band management interface

peer bond utilized for MLAG traffic

subinterface used for clagd

for MLAG between spine01 and spine02

for accessing the outside network

swp1 through swp44

bond to controller01 for host-to-switch

bond to compute01 for host-to-switch

out-of-band management interface

peer bond utilized for MLAG traffic

subinterface used for clagd communication

connect to leaf switch pairs

swp1 through swp30

OPENSTACK AND CUMULU S LINUX: VALIDATED DESIGN GUI DE

bond to another leaf switch pair

bond to another leaf switch pair

out-of-band management interface

peer bond utilized for MLAG traffic

subinterface used for clagd communication

connect to leaf switches

swp1 through swp30