From Fedora Project Wiki
Libvirt, bridging and bonding - a primer.
Recently, Adam Williamson wrote up a nice long blog post on getting bridging to work with the current state of Fedora. His write up is deficient in two areas - he uses the GUI to configure some of it, and he does not address bonding (which is generally only required for a server, anyhow). So, I dove in to solve this problem for a headless server that is doing some virtualization on the side. We will be using nmcli to configure our bond, bridge, and physical ethernet devices. On the example system, I also have a management interface for out-of-band access to the system.
First, you will need to be root for this - it just makes it that much easier.
$ sudo su -
When I started down this path, my networking was all fubbed up - nothing really was working properly.
# nmcli con show conf
NAME UUID TYPE TIMESTAMP-REAL
Wired connection 1 56f32835-5d8e-466c-a49f-f3a8acb5cf94 802-3-ethernet Mon 04 Aug 2014 06:04:46 PM CDT
System em1 1dad842d-1912-ef5a-a43a-bc238fb267e7 802-3-ethernet never
System p4p1 5dd47203-fffb-671a-4fd0-4cff98347a3b 802-3-ethernet never
System bond0 ad33d8b0-1f7b-cab9-9447-ba07f855b143 802-3-ethernet never
Wired connection 2 f5984df2-83d8-42e9-bd8c-7830680cb708 802-3-ethernet Mon 04 Aug 2014 05:39:46 PM CDT
Bridge br1 2ee981ca-5ff4-4f9b-03fe-32879aa3dc85 bridge Mon 04 Aug 2014 05:47:56 PM CDT
kvm-br0 6338b494-9e37-4677-b3e4-837938cfd860 bridge Mon 04 Aug 2014 04:17:38 PM CDT
# brctl show
bridge name bridge id STP enabled interfaces
nm-kvm-br0 8000.000000000000 no
virbr0 8000.000000000000 yes
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether 00:04:76:f4:5c:38 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.3/24 brd 10.0.0.255 scope global p2p1
valid_lft forever preferred_lft forever
inet6 fe80::204:76ff:fef4:5c38/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
inet6 fe80::223:aeff:feab:2054/64 scope link
valid_lft forever preferred_lft forever
4: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:1b:21:ab:d6:a1 brd ff:ff:ff:ff:ff:ff
inet6 fe80::21b:21ff:feab:d6a1/64 scope link
valid_lft forever preferred_lft forever
5: nm-kvm-br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 82:2c:4a:76:7f:16 brd ff:ff:ff:ff:ff:ff
inet6 fe80::802c:4aff:fe76:7f16/64 scope link
valid_lft forever preferred_lft forever
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 12:27:18:b3:35:1d brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
# virsh iface-list
Name State MAC Address
--------------------------------------------
lo active 00:00:00:00:00:00
As you can see, the only IP address I had was my management interface, and libvirt didn't see any interfaces at all. So, I decided to start from scratch.
# nmcli con show conf NAME UUID TYPE TIMESTAMP-REAL Wired connection 1 56f32835-5d8e-466c-a49f-f3a8acb5cf94 802-3-ethernet Mon 04 Aug 2014 06:19:46 PM CDT System em1 1dad842d-1912-ef5a-a43a-bc238fb267e7 802-3-ethernet never System p4p1 5dd47203-fffb-671a-4fd0-4cff98347a3b 802-3-ethernet never System bond0 ad33d8b0-1f7b-cab9-9447-ba07f855b143 802-3-ethernet never Wired connection 2 f5984df2-83d8-42e9-bd8c-7830680cb708 802-3-ethernet Mon 04 Aug 2014 05:39:46 PM CDT Bridge br1 2ee981ca-5ff4-4f9b-03fe-32879aa3dc85 bridge Mon 04 Aug 2014 05:47:56 PM CDT kvm-br0 6338b494-9e37-4677-b3e4-837938cfd860 bridge Mon 04 Aug 2014 04:17:38 PM CDT # nmcli con delete 6338b494-9e37-4677-b3e4-837938cfd860 # nmcli con delete 2ee981ca-5ff4-4f9b-03fe-32879aa3dc85 # nmcli con delete ad33d8b0-1f7b-cab9-9447-ba07f855b143 # nmcli con delete 5dd47203-fffb-671a-4fd0-4cff98347a3b # nmcli con delete 1dad842d-1912-ef5a-a43a-bc238fb267e7 # nmcli con show conf NAME UUID TYPE TIMESTAMP-REAL Wired connection 1 56f32835-5d8e-466c-a49f-f3a8acb5cf94 802-3-ethernet Mon 04 Aug 2014 06:24:46 PM CDT
So, now I have a clean slate to work with. From here, I want to verify that my management interface is configured properly and won't interfere with the real task at hand:
# nmcli con show conf 56f32835-5d8e-466c-a49f-f3a8acb5cf94
connection.id: Wired connection 1
connection.uuid: 56f32835-5d8e-466c-a49f-f3a8acb5cf94
connection.interface-name: --
connection.type: 802-3-ethernet
connection.autoconnect: yes
connection.timestamp: 1407194686
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: --
connection.slave-type: --
connection.secondaries:
connection.gateway-ping-timeout: 0
802-3-ethernet.port: --
802-3-ethernet.speed: 0
802-3-ethernet.duplex: --
802-3-ethernet.auto-negotiate: yes
802-3-ethernet.mac-address: 00:04:76:F4:5C:38
802-3-ethernet.cloned-mac-address: --
802-3-ethernet.mac-address-blacklist:
802-3-ethernet.mtu: auto
802-3-ethernet.s390-subchannels:
802-3-ethernet.s390-nettype: --
802-3-ethernet.s390-options:
ipv4.method: manual
ipv4.dns:
ipv4.dns-search:
ipv4.addresses: { ip = 10.0.0.3/24, gw = 0.0.0.0 }
ipv4.routes:
ipv4.ignore-auto-routes: no
ipv4.ignore-auto-dns: no
ipv4.dhcp-client-id: --
ipv4.dhcp-send-hostname: yes
ipv4.dhcp-hostname: --
ipv4.never-default: yes
ipv4.may-fail: yes
the key thing we were looking for there were ipv4.never-default: yes and ipv4.may-fail: yes. This ensures that the system does not try to use the management network as it's default route, and that if the connection isn't present it won't affect the system.
Next, reboot (if possible) to clean every thing up. Here's where it gets tricky. The key thing to remember here is that the Physical interfaces are subordinate to the bond, which is subordinate to the bridge. So, we configure the bridge first:
# nmcli con add con-name br0 type bridge
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) successfully added.
# nmcli con edit ed441360-b0b6-49f2-a2c8-599c4d3d4b92
===| nmcli interactive connection editor |===
Editing existing 'bridge' connection: 'ed441360-b0b6-49f2-a2c8-599c4d3d4b92'
Type 'help' or '?' for available commands.
Type 'describe [<setting>.<prop>]' for detailed property description.
You may edit the following settings: connection, bridge, 802-3-ethernet (ethernet), ipv4, ipv6
nmcli> print
===============================================================================
Connection details
===============================================================================
connection.id: br0
connection.uuid: ed441360-b0b6-49f2-a2c8-599c4d3d4b92
connection.interface-name: nm-bridge
connection.type: bridge
connection.autoconnect: yes
connection.timestamp: 0
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: --
connection.slave-type: --
connection.secondaries:
connection.gateway-ping-timeout: 0
-------------------------------------------------------------------------------
bridge.interface-name: nm-bridge
bridge.stp: yes
bridge.priority: 128
bridge.forward-delay: 15
bridge.hello-time: 2
bridge.max-age: 20
bridge.ageing-time: 300
-------------------------------------------------------------------------------
nmcli> set bridge.stp no
nmcli> save
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) sucessfully saved.
nmcli> quit
I don't have the infrastructure to require STP, and it causes problems with my HP switches, so I turn it off. Also, note the bridge.interface-name - you'll need this for the next step.
# nmcli con add con-name bond0 type bond
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) successfully added.
# nmcli con edit 58fe56dd-a30b-483a-9e35-3727ccc6ab0e
===| nmcli interactive connection editor |===
Editing existing 'bond' connection: '58fe56dd-a30b-483a-9e35-3727ccc6ab0e'
Type 'help' or '?' for available commands.
Type 'describe [<setting>.<prop>]' for detailed property description.
You may edit the following settings: connection, bond, 802-3-ethernet (ethernet), ipv4, ipv6
nmcli> print
===============================================================================
Connection details
===============================================================================
connection.id: bond0
connection.uuid: 58fe56dd-a30b-483a-9e35-3727ccc6ab0e
connection.interface-name: nm-bond
connection.type: bond
connection.autoconnect: yes
connection.timestamp: 0
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: --
connection.slave-type: --
connection.secondaries:
connection.gateway-ping-timeout: 0
-------------------------------------------------------------------------------
bond.interface-name: nm-bond
bond.options: miimon=100,mode=balance-rr
-------------------------------------------------------------------------------
nmcli> set connection.master nm-bridge
nmcli> set connection.slave-type bridge
nmcli> set bond.options miimon=100,mode=802.3ad
nmcli> print
===============================================================================
Connection details
===============================================================================
connection.id: bond0
connection.uuid: 58fe56dd-a30b-483a-9e35-3727ccc6ab0e
connection.interface-name: nm-bond
connection.type: bond
connection.autoconnect: yes
connection.timestamp: 0
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: nm-bridge
connection.slave-type: bridge
connection.secondaries:
connection.gateway-ping-timeout: 0
-------------------------------------------------------------------------------
bond.interface-name: nm-bond
bond.options: miimon=100,mode=802.3ad
-------------------------------------------------------------------------------
nmcli> save
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) sucessfully saved.
nmcli> quit
You now have a bond inteface bridged with your bridge interface. Now we need to add the physical interfaces to the bond.
# nmcli con add con-name eth0 ifname em1 type bond-slave master bond0 Connection 'eth0' (c58e2397-2a9d-409f-b3b9-f6667fd44274) successfully added. # nmcli con add con-name eth1 ifname p4p1 type bond-slave master bond0 Connection 'eth1' (56e6e53f-5039-4e10-bb3f-8b4b66d41d31) successfully added. # nmcli con show conf NAME UUID TYPE TIMESTAMP-REAL mgmt0 56f32835-5d8e-466c-a49f-f3a8acb5cf94 802-3-ethernet Mon 04 Aug 2014 06:35:20 PM CDT eth1 56e6e53f-5039-4e10-bb3f-8b4b66d41d31 802-3-ethernet never eth0 c58e2397-2a9d-409f-b3b9-f6667fd44274 802-3-ethernet never bond0 58fe56dd-a30b-483a-9e35-3727ccc6ab0e bond never br0 ed441360-b0b6-49f2-a2c8-599c4d3d4b92 bridge never
Now we need to add the IP address to the bridge interface.
# nmcli con edit ed441360-b0b6-49f2-a2c8-599c4d3d4b92
===| nmcli interactive connection editor |===
Editing existing 'bridge' connection: 'ed441360-b0b6-49f2-a2c8-599c4d3d4b92'
Type 'help' or '?' for available commands.
Type 'describe [<setting>.<prop>]' for detailed property description.
You may edit the following settings: connection, bridge, 802-3-ethernet (ethernet), ipv4, ipv6
nmcli> goto ipv4
You may edit the following properties: method, dns, dns-search, addresses, routes, ignore-auto-routes, ignore-auto-dns, dhcp-client-id, dhcp-send-hostname, dhcp-hostname, never-default, may-fail
nmcli ipv4> set method manual
nmcli ipv4> set addresses 172.16.4.3/24 172.16.4.1
nmcli ipv4> set dns 172.16.4.2
nmcli ipv4> print
['ipv4' setting values]
ipv4.method: manual
ipv4.dns: 172.16.4.2
ipv4.dns-search:
ipv4.addresses: { ip = 172.16.4.3/24, gw = 172.16.4.1 }
ipv4.routes:
ipv4.ignore-auto-routes: no
ipv4.ignore-auto-dns: no
ipv4.dhcp-client-id: --
ipv4.dhcp-send-hostname: yes
ipv4.dhcp-hostname: --
ipv4.never-default: no
ipv4.may-fail: yes
nmcli ipv4> save
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) sucessfully saved.
nmcli ipv4> quit
Almost there, now we need to turn them on.
# nmcli con up br0 Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4) # nmcli con up bond0 Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5) # nmcli con up eth0 Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/7) # nmcli con up eth1 Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/9) # nmcli con show act NAME UUID DEVICES DEFAULT VPN MASTER-PATH mgmt0 56f32835-5d8e-466c-a49f-f3a8acb5cf94 p2p1 no no -- br0 ed441360-b0b6-49f2-a2c8-599c4d3d4b92 nm-bridge yes no -- bond0 58fe56dd-a30b-483a-9e35-3727ccc6ab0e nm-bond no no /org/freedesktop/NetworkManager/Devices/5 eth0 c58e2397-2a9d-409f-b3b9-f6667fd44274 em1 no no /org/freedesktop/NetworkManager/Devices/6 eth1 56e6e53f-5039-4e10-bb3f-8b4b66d41d31 p4p1 no no /org/freedesktop/NetworkManager/Devices/6 [root@hermes ~]# nmcli con show conf NAME UUID TYPE TIMESTAMP-REAL mgmt0 56f32835-5d8e-466c-a49f-f3a8acb5cf94 802-3-ethernet Mon 04 Aug 2014 06:40:20 PM CDT eth1 56e6e53f-5039-4e10-bb3f-8b4b66d41d31 802-3-ethernet Mon 04 Aug 2014 06:43:44 PM CDT eth0 c58e2397-2a9d-409f-b3b9-f6667fd44274 802-3-ethernet Mon 04 Aug 2014 06:42:46 PM CDT bond0 58fe56dd-a30b-483a-9e35-3727ccc6ab0e bond Mon 04 Aug 2014 06:41:30 PM CDT br0 ed441360-b0b6-49f2-a2c8-599c4d3d4b92 bridge Mon 04 Aug 2014 06:40:20 PM CDT
At this point, I reinstalled the virtualization group and rebooted to make sure everything was up to par.
# yum group install Virtualization
Now, for the verification stage, to make sure everything is as we desired.
# nmcli con show conf
NAME UUID TYPE TIMESTAMP-REAL
bond0 58fe56dd-a30b-483a-9e35-3727ccc6ab0e bond Mon 04 Aug 2014 06:47:01 PM CDT
eth0 c58e2397-2a9d-409f-b3b9-f6667fd44274 802-3-ethernet Mon 04 Aug 2014 06:47:03 PM CDT
mgmt0 56f32835-5d8e-466c-a49f-f3a8acb5cf94 802-3-ethernet Mon 04 Aug 2014 06:47:01 PM CDT
eth1 56e6e53f-5039-4e10-bb3f-8b4b66d41d31 802-3-ethernet Mon 04 Aug 2014 06:47:04 PM CDT
br0 ed441360-b0b6-49f2-a2c8-599c4d3d4b92 bridge Mon 04 Aug 2014 06:47:01 PM CDT
# nmcli con show act
NAME UUID DEVICES DEFAULT VPN MASTER-PATH
br0 ed441360-b0b6-49f2-a2c8-599c4d3d4b92 nm-bridge yes no --
bond0 58fe56dd-a30b-483a-9e35-3727ccc6ab0e nm-bond no no /org/freedesktop/NetworkManager/Devices/2
mgmt0 56f32835-5d8e-466c-a49f-f3a8acb5cf94 p2p1 no no --
eth0 c58e2397-2a9d-409f-b3b9-f6667fd44274 em1 no no /org/freedesktop/NetworkManager/Devices/1
eth1 56e6e53f-5039-4e10-bb3f-8b4b66d41d31 p4p1 no no /org/freedesktop/NetworkManager/Devices/1
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether 00:04:76:f4:5c:38 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.3/24 brd 10.0.0.255 scope global p2p1
valid_lft forever preferred_lft forever
inet6 fe80::204:76ff:fef4:5c38/64 scope link
valid_lft forever preferred_lft forever
3: em1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
4: p4p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
5: nm-bond: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master nm-bridge state UP group default
link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
inet6 fe80::a8e4:e3ff:fe2b:12af/64 scope link
valid_lft forever preferred_lft forever
6: nm-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
inet 172.16.4.3/24 brd 172.16.4.255 scope global nm-bridge
valid_lft forever preferred_lft forever
inet6 fe80::5c18:76ff:fef5:e166/64 scope link
valid_lft forever preferred_lft forever
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 42:59:71:b6:95:7b brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
# virsh iface-list
Name State MAC Address
--------------------------------------------
lo active 00:00:00:00:00:00
nm-bridge active 00:23:ae:ab:20:54
Libvirt is now bridged with the bonded interfaces to the network, 2GB pathway to and from the virtual machines to be built.