From Fedora Project Wiki
(Created page with "Libvirt, bridging and bonding - a primer. Recently, Adam Williamson wrote up a nice long [http://www.happyassassin.net/2014/07/23/bridged-networking-for-libvirt-with-networkm...")
 
No edit summary
Line 237: Line 237:
You now have a bond inteface bridged with your bridge interface. Now we need to add the physical interfaces to the bond.
You now have a bond inteface bridged with your bridge interface. Now we need to add the physical interfaces to the bond.
<pre>
<pre>
# nmcli con add con-name eth0 ifname em1 type bond-slave master bond0
# nmcli con add con-name eth0 ifname em1 type bond-slave master nm-bond
Connection 'eth0' (c58e2397-2a9d-409f-b3b9-f6667fd44274) successfully added.
Connection 'eth0' (c58e2397-2a9d-409f-b3b9-f6667fd44274) successfully added.


# nmcli con add con-name eth1 ifname p4p1 type bond-slave master bond0
# nmcli con add con-name eth1 ifname p4p1 type bond-slave master nm-bond
Connection 'eth1' (56e6e53f-5039-4e10-bb3f-8b4b66d41d31) successfully added.
Connection 'eth1' (56e6e53f-5039-4e10-bb3f-8b4b66d41d31) successfully added.


Revision as of 16:39, 5 August 2014

Libvirt, bridging and bonding - a primer.

Recently, Adam Williamson wrote up a nice long blog post on getting bridging to work with the current state of Fedora. His write up is deficient in two areas - he uses the GUI to configure some of it, and he does not address bonding (which is generally only required for a server, anyhow). So, I dove in to solve this problem for a headless server that is doing some virtualization on the side. We will be using nmcli to configure our bond, bridge, and physical ethernet devices. On the example system, I also have a management interface for out-of-band access to the system.

First, you will need to be root for this - it just makes it that much easier. 

$ sudo su -

When I started down this path, my networking was all fubbed up - nothing really was working properly. 

# nmcli con show conf
NAME                UUID                                  TYPE            TIMESTAMP-REAL                  
Wired connection 1  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:04:46 PM CDT 
System em1          1dad842d-1912-ef5a-a43a-bc238fb267e7  802-3-ethernet  never                           
System p4p1         5dd47203-fffb-671a-4fd0-4cff98347a3b  802-3-ethernet  never                           
System bond0        ad33d8b0-1f7b-cab9-9447-ba07f855b143  802-3-ethernet  never                           
Wired connection 2  f5984df2-83d8-42e9-bd8c-7830680cb708  802-3-ethernet  Mon 04 Aug 2014 05:39:46 PM CDT 
Bridge br1          2ee981ca-5ff4-4f9b-03fe-32879aa3dc85  bridge          Mon 04 Aug 2014 05:47:56 PM CDT 
kvm-br0             6338b494-9e37-4677-b3e4-837938cfd860  bridge          Mon 04 Aug 2014 04:17:38 PM CDT 

# brctl show
bridge name     bridge id               STP enabled     interfaces
nm-kvm-br0              8000.000000000000       no
virbr0          8000.000000000000       yes

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 00:04:76:f4:5c:38 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global p2p1
       valid_lft forever preferred_lft forever
    inet6 fe80::204:76ff:fef4:5c38/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::223:aeff:feab:2054/64 scope link 
       valid_lft forever preferred_lft forever
4: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1b:21:ab:d6:a1 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::21b:21ff:feab:d6a1/64 scope link 
       valid_lft forever preferred_lft forever
5: nm-kvm-br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether 82:2c:4a:76:7f:16 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::802c:4aff:fe76:7f16/64 scope link 
       valid_lft forever preferred_lft forever
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 12:27:18:b3:35:1d brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

# virsh iface-list 
Name                 State      MAC Address
--------------------------------------------
lo                   active     00:00:00:00:00:00

As you can see, the only IP address I had was my management interface, and libvirt didn't see any interfaces at all. So, I decided to start from scratch. 

# nmcli con show conf
NAME                UUID                                  TYPE            TIMESTAMP-REAL                  
Wired connection 1  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:19:46 PM CDT 
System em1          1dad842d-1912-ef5a-a43a-bc238fb267e7  802-3-ethernet  never                           
System p4p1         5dd47203-fffb-671a-4fd0-4cff98347a3b  802-3-ethernet  never                           
System bond0        ad33d8b0-1f7b-cab9-9447-ba07f855b143  802-3-ethernet  never                           
Wired connection 2  f5984df2-83d8-42e9-bd8c-7830680cb708  802-3-ethernet  Mon 04 Aug 2014 05:39:46 PM CDT 
Bridge br1          2ee981ca-5ff4-4f9b-03fe-32879aa3dc85  bridge          Mon 04 Aug 2014 05:47:56 PM CDT 
kvm-br0             6338b494-9e37-4677-b3e4-837938cfd860  bridge          Mon 04 Aug 2014 04:17:38 PM CDT 

# nmcli con delete 6338b494-9e37-4677-b3e4-837938cfd860
# nmcli con delete 2ee981ca-5ff4-4f9b-03fe-32879aa3dc85
# nmcli con delete ad33d8b0-1f7b-cab9-9447-ba07f855b143
# nmcli con delete 5dd47203-fffb-671a-4fd0-4cff98347a3b
# nmcli con delete 1dad842d-1912-ef5a-a43a-bc238fb267e7
# nmcli con show conf 
NAME                UUID                                  TYPE            TIMESTAMP-REAL                  
Wired connection 1  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:24:46 PM CDT

So, now I have a clean slate to work with. From here, I want to verify that my management interface is configured properly and won't interfere with the real task at hand: 

# nmcli con show conf 56f32835-5d8e-466c-a49f-f3a8acb5cf94
connection.id:                          Wired connection 1
connection.uuid:                        56f32835-5d8e-466c-a49f-f3a8acb5cf94
connection.interface-name:              --
connection.type:                        802-3-ethernet
connection.autoconnect:                 yes
connection.timestamp:                   1407194686
connection.read-only:                   no
connection.permissions:                 
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.secondaries:                 
connection.gateway-ping-timeout:        0
802-3-ethernet.port:                    --
802-3-ethernet.speed:                   0
802-3-ethernet.duplex:                  --
802-3-ethernet.auto-negotiate:          yes
802-3-ethernet.mac-address:             00:04:76:F4:5C:38
802-3-ethernet.cloned-mac-address:      --
802-3-ethernet.mac-address-blacklist:   
802-3-ethernet.mtu:                     auto
802-3-ethernet.s390-subchannels:        
802-3-ethernet.s390-nettype:            --
802-3-ethernet.s390-options:            
ipv4.method:                            manual
ipv4.dns:                               
ipv4.dns-search:                        
ipv4.addresses:                         { ip = 10.0.0.3/24, gw = 0.0.0.0 }
ipv4.routes:                            
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.never-default:                     yes
ipv4.may-fail:                          yes

the key thing we were looking for there were ipv4.never-default: yes and ipv4.may-fail: yes. This ensures that the system does not try to use the management network as it's default route, and that if the connection isn't present it won't affect the system.

Next, reboot (if possible) to clean every thing up. Here's where it gets tricky. The key thing to remember here is that the Physical interfaces are subordinate to the bond, which is subordinate to the bridge. So, we configure the bridge first: 

# nmcli con add con-name br0 type bridge
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) successfully added.
# nmcli con edit ed441360-b0b6-49f2-a2c8-599c4d3d4b92

===| nmcli interactive connection editor |===

Editing existing 'bridge' connection: 'ed441360-b0b6-49f2-a2c8-599c4d3d4b92'

Type 'help' or '?' for available commands.
Type 'describe [<setting>.<prop>]' for detailed property description.

You may edit the following settings: connection, bridge, 802-3-ethernet (ethernet), ipv4, ipv6
nmcli> print
===============================================================================
                              Connection details
===============================================================================
connection.id:                          br0
connection.uuid:                        ed441360-b0b6-49f2-a2c8-599c4d3d4b92
connection.interface-name:              nm-bridge
connection.type:                        bridge
connection.autoconnect:                 yes
connection.timestamp:                   0
connection.read-only:                   no
connection.permissions:                 
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.secondaries:                 
connection.gateway-ping-timeout:        0
-------------------------------------------------------------------------------
bridge.interface-name:                  nm-bridge
bridge.stp:                             yes
bridge.priority:                        128
bridge.forward-delay:                   15
bridge.hello-time:                      2
bridge.max-age:                         20
bridge.ageing-time:                     300
-------------------------------------------------------------------------------
nmcli> set bridge.stp no
nmcli> save
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) sucessfully saved.
nmcli> quit

I don't have the infrastructure to require STP, and it causes problems with my HP switches, so I turn it off. Also, note the bridge.interface-name - you'll need this for the next step. 

# nmcli con add con-name bond0 type bond
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) successfully added.
# nmcli con edit 58fe56dd-a30b-483a-9e35-3727ccc6ab0e

===| nmcli interactive connection editor |===

Editing existing 'bond' connection: '58fe56dd-a30b-483a-9e35-3727ccc6ab0e'

Type 'help' or '?' for available commands.
Type 'describe [<setting>.<prop>]' for detailed property description.

You may edit the following settings: connection, bond, 802-3-ethernet (ethernet), ipv4, ipv6
nmcli> print
===============================================================================
                              Connection details
===============================================================================
connection.id:                          bond0
connection.uuid:                        58fe56dd-a30b-483a-9e35-3727ccc6ab0e
connection.interface-name:              nm-bond
connection.type:                        bond
connection.autoconnect:                 yes
connection.timestamp:                   0
connection.read-only:                   no
connection.permissions:                 
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.secondaries:                 
connection.gateway-ping-timeout:        0
-------------------------------------------------------------------------------
bond.interface-name:                    nm-bond
bond.options:                           miimon=100,mode=balance-rr
-------------------------------------------------------------------------------
nmcli> set connection.master nm-bridge
nmcli> set connection.slave-type bridge
nmcli> set bond.options miimon=100,mode=802.3ad
nmcli> print
===============================================================================
                              Connection details
===============================================================================
connection.id:                          bond0
connection.uuid:                        58fe56dd-a30b-483a-9e35-3727ccc6ab0e
connection.interface-name:              nm-bond
connection.type:                        bond
connection.autoconnect:                 yes
connection.timestamp:                   0
connection.read-only:                   no
connection.permissions:                 
connection.zone:                        --
connection.master:                      nm-bridge
connection.slave-type:                  bridge
connection.secondaries:                 
connection.gateway-ping-timeout:        0
-------------------------------------------------------------------------------
bond.interface-name:                    nm-bond
bond.options:                           miimon=100,mode=802.3ad
-------------------------------------------------------------------------------
nmcli> save
Connection 'bond0' (58fe56dd-a30b-483a-9e35-3727ccc6ab0e) sucessfully saved.
nmcli> quit

You now have a bond inteface bridged with your bridge interface. Now we need to add the physical interfaces to the bond. 

# nmcli con add con-name eth0 ifname em1 type bond-slave master nm-bond
Connection 'eth0' (c58e2397-2a9d-409f-b3b9-f6667fd44274) successfully added.

# nmcli con add con-name eth1 ifname p4p1 type bond-slave master nm-bond
Connection 'eth1' (56e6e53f-5039-4e10-bb3f-8b4b66d41d31) successfully added.

# nmcli con show conf
NAME   UUID                                  TYPE            TIMESTAMP-REAL                  
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:35:20 PM CDT 
eth1   56e6e53f-5039-4e10-bb3f-8b4b66d41d31  802-3-ethernet  never                           
eth0   c58e2397-2a9d-409f-b3b9-f6667fd44274  802-3-ethernet  never                           
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  bond            never                           
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  bridge          never

Now we need to add the IP address to the bridge interface. 

# nmcli con edit ed441360-b0b6-49f2-a2c8-599c4d3d4b92

===| nmcli interactive connection editor |===

Editing existing 'bridge' connection: 'ed441360-b0b6-49f2-a2c8-599c4d3d4b92'

Type 'help' or '?' for available commands.
Type 'describe [<setting>.<prop>]' for detailed property description.

You may edit the following settings: connection, bridge, 802-3-ethernet (ethernet), ipv4, ipv6
nmcli> goto ipv4
You may edit the following properties: method, dns, dns-search, addresses, routes, ignore-auto-routes, ignore-auto-dns, dhcp-client-id, dhcp-send-hostname, dhcp-hostname, never-default, may-fail
nmcli ipv4> set method manual
nmcli ipv4> set addresses 172.16.4.3/24 172.16.4.1
nmcli ipv4> set dns 172.16.4.2
nmcli ipv4> print
['ipv4' setting values]
ipv4.method:                            manual
ipv4.dns:                               172.16.4.2
ipv4.dns-search:                        
ipv4.addresses:                         { ip = 172.16.4.3/24, gw = 172.16.4.1 }
ipv4.routes:                            
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.never-default:                     no
ipv4.may-fail:                          yes
nmcli ipv4> save
Connection 'br0' (ed441360-b0b6-49f2-a2c8-599c4d3d4b92) sucessfully saved.
nmcli ipv4> quit

Almost there, now we need to turn them on. 

# nmcli con up br0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/4)

# nmcli con up bond0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5)

# nmcli con up eth0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/7)

# nmcli con up eth1
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/9)

# nmcli con show act
NAME   UUID                                  DEVICES    DEFAULT  VPN  MASTER-PATH                               
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  p2p1       no       no   --                                        
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  nm-bridge  yes      no   --                                        
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  nm-bond    no       no   /org/freedesktop/NetworkManager/Devices/5 
eth0   c58e2397-2a9d-409f-b3b9-f6667fd44274  em1        no       no   /org/freedesktop/NetworkManager/Devices/6 
eth1   56e6e53f-5039-4e10-bb3f-8b4b66d41d31  p4p1       no       no   /org/freedesktop/NetworkManager/Devices/6 
[root@hermes ~]# nmcli con show conf
NAME   UUID                                  TYPE            TIMESTAMP-REAL                  
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:40:20 PM CDT 
eth1   56e6e53f-5039-4e10-bb3f-8b4b66d41d31  802-3-ethernet  Mon 04 Aug 2014 06:43:44 PM CDT 
eth0   c58e2397-2a9d-409f-b3b9-f6667fd44274  802-3-ethernet  Mon 04 Aug 2014 06:42:46 PM CDT 
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  bond            Mon 04 Aug 2014 06:41:30 PM CDT 
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  bridge          Mon 04 Aug 2014 06:40:20 PM CDT

At this point, I reinstalled the virtualization group and rebooted to make sure everything was up to par. 

# yum group install Virtualization

Now, for the verification stage, to make sure everything is as we desired. 

# nmcli con show conf
NAME   UUID                                  TYPE            TIMESTAMP-REAL                  
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  bond            Mon 04 Aug 2014 06:47:01 PM CDT 
eth0   c58e2397-2a9d-409f-b3b9-f6667fd44274  802-3-ethernet  Mon 04 Aug 2014 06:47:03 PM CDT 
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  802-3-ethernet  Mon 04 Aug 2014 06:47:01 PM CDT 
eth1   56e6e53f-5039-4e10-bb3f-8b4b66d41d31  802-3-ethernet  Mon 04 Aug 2014 06:47:04 PM CDT 
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  bridge          Mon 04 Aug 2014 06:47:01 PM CDT 

# nmcli con show act
NAME   UUID                                  DEVICES    DEFAULT  VPN  MASTER-PATH                               
br0    ed441360-b0b6-49f2-a2c8-599c4d3d4b92  nm-bridge  yes      no   --                                        
bond0  58fe56dd-a30b-483a-9e35-3727ccc6ab0e  nm-bond    no       no   /org/freedesktop/NetworkManager/Devices/2 
mgmt0  56f32835-5d8e-466c-a49f-f3a8acb5cf94  p2p1       no       no   --                                        
eth0   c58e2397-2a9d-409f-b3b9-f6667fd44274  em1        no       no   /org/freedesktop/NetworkManager/Devices/1 
eth1   56e6e53f-5039-4e10-bb3f-8b4b66d41d31  p4p1       no       no   /org/freedesktop/NetworkManager/Devices/1 

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 00:04:76:f4:5c:38 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global p2p1
       valid_lft forever preferred_lft forever
    inet6 fe80::204:76ff:fef4:5c38/64 scope link 
       valid_lft forever preferred_lft forever
3: em1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
4: p4p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-bond state UP group default qlen 1000
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
5: nm-bond: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master nm-bridge state UP group default 
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a8e4:e3ff:fe2b:12af/64 scope link 
       valid_lft forever preferred_lft forever
6: nm-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 00:23:ae:ab:20:54 brd ff:ff:ff:ff:ff:ff
    inet 172.16.4.3/24 brd 172.16.4.255 scope global nm-bridge
       valid_lft forever preferred_lft forever
    inet6 fe80::5c18:76ff:fef5:e166/64 scope link 
       valid_lft forever preferred_lft forever
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 42:59:71:b6:95:7b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

# virsh iface-list
Name                 State      MAC Address
--------------------------------------------
lo                   active     00:00:00:00:00:00
nm-bridge            active     00:23:ae:ab:20:54

Libvirt is now bridged with the bonded interfaces to the network, 2GB pathway to and from the virtual machines to be built.

Retrieved from "https://fedoraproject.org/w/index.php?title=User:Dmossor/libvirt-bridge&oldid=383505"