Libvirt, bridging and bonding - a primer
Introduction
Recently, Adam Williamson wrote up a nice long blog post on getting bridging to work with the current state of Fedora. His write up is targeted at desktop users, so he didn't cover two areas - he uses the GUI to configure some of it, and he does not address bonding (which is generally only required for a server, anyhow). So, I dove in to solve this problem for a headless server that is doing some virtualization on the side. We will be using nmcli to configure our bond, bridge, and physical ethernet devices. On the example system, I also have a management interface for out-of-band access to the system.
Procedure
User: root
First, you will need to be root for this - it just makes it that much easier.
$ sudo su -
Example non-working configuration
When I started down this path, my networking was severely mis-configured - nothing really was working properly. It had reached this point by using the intuitive tools provided by Virtual Machine Manager (VMM) and virsh. Once I worked through the "suggested" process with those tools, I ended up with a broken network.
# nmcli con show conf
NAME UUID TYPE TIMESTAMP-REAL
Wired connection 1 56f32835-5d8e-466c-a49f-f3a8acb5cf94 802-3-ethernet Mon 04 Aug 2014 06:04:46 PM CDT
System em1 1dad842d-1912-ef5a-a43a-bc238fb267e7 802-3-ethernet never
System p4p1 5dd47203-fffb-671a-4fd0-4cff98347a3b 802-3-ethernet never
System bond0 ad33d8b0-1f7b-cab9-9447-ba07f855b143 802-3-ethernet never
Wired connection 2 f5984df2-83d8-42e9-bd8c-7830680cb708 802-3-ethernet Mon 04 Aug 2014 05:39:46 PM CDT
Bridge br1 2ee981ca-5ff4-4f9b-03fe-32879aa3dc85 bridge Mon 04 Aug 2014 05:47:56 PM CDT
kvm-br0 6338b494-9e37-4677-b3e4-837938cfd860 bridge Mon 04 Aug 2014 04:17:38 PM CDT
# brctl show
bridge name bridge id STP enabled interfaces
nm-kvm-br0 8000.000000000000 no
virbr0 8000.000000000000 yes
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether 00:04:76:f4:5c:38 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.3/24 brd 10.0.0.255 scope global p2p1
valid_lft forever preferred_lft forever
inet6 fe80::204:76ff:fef4:5c38/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
inet6 fe80::223:aeff:feab:2054/64 scope link
valid_lft forever preferred_lft forever
4: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:1b:21:ab:d6:a1 brd ff:ff:ff:ff:ff:ff
inet6 fe80::21b:21ff:feab:d6a1/64 scope link
valid_lft forever preferred_lft forever
5: nm-kvm-br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 82:2c:4a:76:7f:16 brd ff:ff:ff:ff:ff:ff
inet6 fe80::802c:4aff:fe76:7f16/64 scope link
valid_lft forever preferred_lft forever
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 12:27:18:b3:35:1d brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
# virsh iface-list
Name State MAC Address
--------------------------------------------
lo active 00:00:00:00:00:00
As you can see, the only IP address I had was my management interface, and libvirt didn't see any interfaces at all. So, I decided to start from scratch.
Reset
The first step is to remove all configuration from the system.
# nmcli con show conf NAME UUID TYPE TIMESTAMP-REAL Wired connection 1 56f32835-5d8e-466c-a49f-f3a8acb5cf94 802-3-ethernet Mon 04 Aug 2014 06:19:46 PM CDT System em1 1dad842d-1912-ef5a-a43a-bc238fb267e7 802-3-ethernet never System p4p1 5dd47203-fffb-671a-4fd0-4cff98347a3b 802-3-ethernet never System bond0 ad33d8b0-1f7b-cab9-9447-ba07f855b143 802-3-ethernet never Wired connection 2 f5984df2-83d8-42e9-bd8c-7830680cb708 802-3-ethernet Mon 04 Aug 2014 05:39:46 PM CDT Bridge br1 2ee981ca-5ff4-4f9b-03fe-32879aa3dc85 bridge Mon 04 Aug 2014 05:47:56 PM CDT kvm-br0 6338b494-9e37-4677-b3e4-837938cfd860 bridge Mon 04 Aug 2014 04:17:38 PM CDT # nmcli con delete 6338b494-9e37-4677-b3e4-837938cfd860 # nmcli con delete 2ee981ca-5ff4-4f9b-03fe-32879aa3dc85 # nmcli con delete ad33d8b0-1f7b-cab9-9447-ba07f855b143 # nmcli con delete 5dd47203-fffb-671a-4fd0-4cff98347a3b # nmcli con delete 1dad842d-1912-ef5a-a43a-bc238fb267e7
That deleted all of my current network configuration except for my management interface.
Verify starting point
# nmcli con show conf NAME UUID TYPE TIMESTAMP-REAL Wired connection 1 56f32835-5d8e-466c-a49f-f3a8acb5cf94 802-3-ethernet Mon 04 Aug 2014 06:24:46 PM CDT
So, now I have a clean slate to work with. From here, I want to verify that my management interface is configured properly and won't interfere with the real task at hand:
# nmcli con show conf 56f32835-5d8e-466c-a49f-f3a8acb5cf94
connection.id: Wired connection 1
connection.uuid: 56f32835-5d8e-466c-a49f-f3a8acb5cf94
connection.interface-name: --
connection.type: 802-3-ethernet
connection.autoconnect: yes
connection.timestamp: 1407194686
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: --
connection.slave-type: --
connection.secondaries:
connection.gateway-ping-timeout: 0
802-3-ethernet.port: --
802-3-ethernet.speed: 0
802-3-ethernet.duplex: --
802-3-ethernet.auto-negotiate: yes
802-3-ethernet.mac-address: 00:04:76:F4:5C:38
802-3-ethernet.cloned-mac-address: --
802-3-ethernet.mac-address-blacklist:
802-3-ethernet.mtu: auto
802-3-ethernet.s390-subchannels:
802-3-ethernet.s390-nettype: --
802-3-ethernet.s390-options:
ipv4.method: manual
ipv4.dns:
ipv4.dns-search:
ipv4.addresses: { ip = 10.0.0.3/24, gw = 0.0.0.0 }
ipv4.routes:
ipv4.ignore-auto-routes: no
ipv4.ignore-auto-dns: no
ipv4.dhcp-client-id: --
ipv4.dhcp-send-hostname: yes
ipv4.dhcp-hostname: --
ipv4.never-default: yes
ipv4.may-fail: yes
the key thing we were looking for there were ipv4.never-default: yes and ipv4.may-fail: yes. This ensures that the system does not try to use the management network as it's default route, and that if the connection isn't present it won't affect the system.
Finally, reboot (if possible) to clean every thing up.
Configuring the interfaces
This is where it gets tricky. The key thing to remember here is that the physical interfaces are subordinate to the bond, which is subordinate to the bridge. So, we configure the bridge first.
System -> bridge -> bond -> physical -> network
Bridge interface
The bridge is the main interface to the system. This is what will hold the IP information and be presented to the kernel network stack and libvirt.
I am showing everything on one line here for brevity's sake, but you can also stop after defining the con-name or if-name, then entering the nmcli shell to finish the configuration
# nmcli con add con-name br0 ifname br0 type bridge autoconnect yes stp off ip4 172.16.4.3/24 gw4 172.16.4.1
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) successfully added.
# nmcli con show ed441360-b0b6-49f2-a2c8-599c4d3d4b92
connection.id: br0
connection.uuid: ed441360-b0b6-49f2-a2c8-599c4d3d4b92
connection.interface-name: br0
connection.type: bridge
connection.autoconnect: yes
connection.timestamp: 0
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: --
connection.slave-type: --
connection.secondaries:
connection.gateway-ping-timeout: 0
ipv4.method: manual
ipv4.dns: 172.16.4.2
ipv4.dns-search:
ipv4.addresses: { ip = 172.16.4.3/24, gw = 172.16.4.1 }
ipv4.routes:
ipv4.ignore-auto-routes: no
ipv4.ignore-auto-dns: no
ipv4.dhcp-client-id: --
ipv4.dhcp-send-hostname: yes
ipv4.dhcp-hostname: --
ipv4.never-default: no
ipv4.may-fail: yes
bridge.interface-name: br0
bridge.stp: no
bridge.priority: 128
bridge.forward-delay: 15
bridge.hello-time: 2
bridge.max-age: 20
bridge.ageing-time: 300
Bond Interface
Bonding is used on servers to provide a higher bandwidth, fault tolerance, and/or load balancing. To get the benefit of the faster bandwidth, a switch that supports 802.3AD Link Aggregation is required. To take advantage of the fault tolerance or certain load balancing options, specific hardware is not required. For this example, I am setting up an 802.3AD aggregate link to my switch in order to establish a 2Gbps circuit to the server.
# nmcli con add con-name bond0 ifname bond0 type bond autoconnect yes mode 802.3ad miimon 100 Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) successfully added.
Now the command line options for the bond configuration do not allow me to set it as a bridged interface, so we will need to drop into the nmcli shell to finish up the config.
# nmcli con edit 58fe56dd-a30b-483a-9e35-3727ccc6ab0e
===| nmcli interactive connection editor |===
Editing existing 'bond' connection: '58fe56dd-a30b-483a-9e35-3727ccc6ab0e'
Type 'help' or '?' for available commands.
Type 'describe [<setting>.<prop>]' for detailed property description.
You may edit the following settings: connection, bond, 802-3-ethernet (ethernet), ipv4, ipv6
nmcli> print
===============================================================================
Connection details
===============================================================================
connection.id: bond0
connection.uuid: 58fe56dd-a30b-483a-9e35-3727ccc6ab0e
connection.interface-name: bond0
connection.type: bond
connection.autoconnect: yes
connection.timestamp: 0
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: --
connection.slave-type: --
connection.secondaries:
connection.gateway-ping-timeout: 0
-------------------------------------------------------------------------------
bond.interface-name: bond0
bond.options: miimon=100,mode=balance-rr
-------------------------------------------------------------------------------
nmcli> set connection.master br0
nmcli> set connection.slave-type bridge
nmcli> save
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) sucessfully saved.
nmcli> quit
You now have a bond interface bridged with the bridge interface we set up previously. Now we need to add the physical interfaces to the bond.
Physical Interfaces
Bonded interfaces are pseudo interfaces just like bridges - they need to have physical interfaces slaved to them in order to be operational. Here we will do this all from the command line. You need to know what device name udev gave the interfaces, the easiest way to do this is with the old ifconfig command - it will spit out the udev device names as well as any configuration already applied. Using the udev device name of your physical interfaces, add them as slaves to the bond0 interface we just created.
# nmcli con add con-name eth0 ifname em1 type bond-slave master bond0 Connection 'eth0' (c58e2397-2a9d-409f-b3b9-f6667fd44274) successfully added. # nmcli con add con-name eth1 ifname p4p1 type bond-slave master bond0 Connection 'eth1' (56e6e53f-5039-4e10-bb3f-8b4b66d41d31) successfully added.
Verification and Activation
The configuration should be complete at this point. To verify our setup, we will use nmcli, brctl, ip address, and <virsh>.
# nmcli con show conf
NAME UUID TYPE TIMESTAMP-REAL
mgmt0 56f32835-5d8e-466c-a49f-f3a8acb5cf94 802-3-ethernet Mon 04 Aug 2014 06:35:20 PM CDT
eth1 56e6e53f-5039-4e10-bb3f-8b4b66d41d31 802-3-ethernet never
eth0 c58e2397-2a9d-409f-b3b9-f6667fd44274 802-3-ethernet never
bond0 58fe56dd-a30b-483a-9e35-3727ccc6ab0e bond never
br0 ed441360-b0b6-49f2-a2c8-599c4d3d4b92 bridge never
# nmcli con up br0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)
# nmcli con up bond0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5)
# nmcli con up eth0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/7)
# nmcli con up eth1
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/9)
# nmcli con show act
NAME UUID DEVICES DEFAULT VPN MASTER-PATH
mgmt0 56f32835-5d8e-466c-a49f-f3a8acb5cf94 p2p1 no no --
br0 ed441360-b0b6-49f2-a2c8-599c4d3d4b92 nm-bridge yes no --
bond0 58fe56dd-a30b-483a-9e35-3727ccc6ab0e nm-bond no no /org/freedesktop/NetworkManager/Devices/5
eth0 c58e2397-2a9d-409f-b3b9-f6667fd44274 em1 no no /org/freedesktop/NetworkManager/Devices/6
eth1 56e6e53f-5039-4e10-bb3f-8b4b66d41d31 p4p1 no no /org/freedesktop/NetworkManager/Devices/6
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether 00:04:76:f4:5c:38 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.3/24 brd 10.0.0.255 scope global p2p1
valid_lft forever preferred_lft forever
inet6 fe80::204:76ff:fef4:5c38/64 scope link
valid_lft forever preferred_lft forever
3: em1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
4: p4p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master nm-bridge state UP group default
link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
inet6 fe80::a8e4:e3ff:fe2b:12af/64 scope link
valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
inet 172.16.4.3/24 brd 172.16.4.255 scope global nm-bridge
valid_lft forever preferred_lft forever
inet6 fe80::5c18:76ff:fef5:e166/64 scope link
valid_lft forever preferred_lft forever
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 42:59:71:b6:95:7b brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
# brctl show
bridge name bridge id STP enabled interfaces
br0 0080.0023aeab2054 no bond0
virbr0 8000.000000000000 yes
# virsh iface-list
Name State MAC Address
--------------------------------------------
lo active 00:00:00:00:00:00
br0 active 00:23:ae:ab:20:54
Libvirt is now bridged with the bonded interfaces to the network, 2GB pathway to and from the virtual machines to be built.