This is a fun lab which involves Linux bridges, Linux containers and spanning tree protocol. I will create L2 switch network using three Linux bridges and three containers connected to bridges. It is going to be a ring network so we can test some basic basic spanning tree function.
Prerequisite:
Install Linux containers using procedure in Lab-36.
In this lab I am using Centos 7. We will be using brctl command to create bridge so Install bridge-utils if not already installed
#yum install bridge-utils
Procedure:
Create three Linux bridges, br0,br1 & br2 using brctl command.
sudo brctl addbr <bridge name>
[root@localhost]# sudo brctl addbr br0 [root@localhost]# sudo brctl addbr br1 [root@localhost]# sudo brctl addbr br2 //bring bridge up [root@localhost]# ifconfig br0 up [root@localhost]# ifconfig br0 br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::c0f6:91ff:fed2:38b0 prefixlen 64 scopeid 0x20 ether c2:f6:91:d2:38:b0 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 6 bytes 508 (508.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@localhost]# ifconfig br2 up [root@localhost]# ifconfig br1 up
Create virtual ethernet (veth) links and bring them up. veth created in pair, these will serve as trunk ports between bridges
//create veth with pair i.e. veth0 paired with veth1 [root@localhost ~]# ip link add veth0 type veth peer name veth1 [root@localhost ~]# ifconfig veth0 up [root@localhost ~]# ifconfig veth1 up //add veth to respective bridges (see topology diagram) [root@localhost ~]# sudo brctl addif br0 veth0 [root@localhost ~]# sudo brctl addif br1 veth1 [root@localhost ~]# sudo brctl show bridge name bridge id STP enabled interfaces br0 8000.e274c68ffca6 no veth0 br1 8000.d28623e04c88 no veth1 //create veth2 paired with veth3, veth4 paired with veth5 [root@localhost ~]# ip link add veth2 type veth peer name veth3 [root@localhost ~]# ip link add veth4 type veth peer name veth5 //bring veth up [root@localhost ~]# ifconfig veth2 up [root@localhost ~]# ifconfig veth3 up [root@localhost ~]# ifconfig veth4 up [root@localhost ~]# ifconfig veth5 up //add veth to respective bridges [root@localhost ~]# sudo brctl addif br1 veth2 [root@localhost ~]# sudo brctl addif br2 veth3 [root@localhost ~]# sudo brctl addif br2 veth4 [root@localhost ~]# sudo brctl addif br0 veth5 //as can be seen veth interfaces are attached to bridges [root@localhost ~]# sudo brctl show bridge name bridge id STP enabled interfaces br0 8000.c2f691d238b0 no veth0 veth5 br1 8000.d28623e04c88 no veth1 veth2 br2 8000.9a127ad7bf76 no veth3 veth4 //check bridge and veth interfaces [root@localhost ~]# ip addr 52: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether c2:f6:91:d2:38:b0 brd ff:ff:ff:ff:ff:ff 53: br1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether d2:86:23:e0:4c:88 brd ff:ff:ff:ff:ff:ff 54: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UP qlen 1000 link/ether d2:86:23:e0:4c:88 brd ff:ff:ff:ff:ff:ff inet6 fe80::d086:23ff:fee0:4c88/64 scope link valid_lft forever preferred_lft forever 55: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP qlen 1000 link/ether e2:74:c6:8f:fc:a6 brd ff:ff:ff:ff:ff:ff inet6 fe80::e074:c6ff:fe8f:fca6/64 scope link valid_lft forever preferred_lft forever 56: veth3@veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br2 state UP qlen 1000 link/ether 9a:12:7a:d7:bf:76 brd ff:ff:ff:ff:ff:ff inet6 fe80::9812:7aff:fed7:bf76/64 scope link valid_lft forever preferred_lft forever 57: veth2@veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UP qlen 1000 link/ether ee:bf:b0:92:54:a0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ecbf:b0ff:fe92:54a0/64 scope link valid_lft forever preferred_lft forever 58: veth5@veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP qlen 1000 link/ether c2:f6:91:d2:38:b0 brd ff:ff:ff:ff:ff:ff inet6 fe80::c0f6:91ff:fed2:38b0/64 scope link valid_lft forever preferred_lft forever 59: veth4@veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br2 state UP qlen 1000 link/ether fe:fe:da:e0:3a:09 brd ff:ff:ff:ff:ff:ff inet6 fe80::fcfe:daff:fee0:3a09/64 scope link valid_lft forever preferred_lft forever 60: br2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 9a:12:7a:d7:bf:76 brd ff:ff:ff:ff:ff:ff [root@localhost ~]#
Create three Linux containers, c1,c2 & c3
#lxc-create -t ubuntu -n c1
#lxc-create -t ubuntu -n c2
#lxc-create -t ubuntu -n c3
[root@localhost ~]# lxc-create -t ubuntu -n c1 Checking cache download in /var/cache/lxc/precise/rootfs-amd64 ... Copy /var/cache/lxc/precise/rootfs-amd64 to /var/lib/lxc/c1/rootfs ... Copying rootfs to /var/lib/lxc/c1/rootfs ... Generating locales... en_US.UTF-8... up-to-date Generation complete. Creating SSH2 RSA key; this may take some time ... Creating SSH2 DSA key; this may take some time ... Creating SSH2 ECDSA key; this may take some time ... Timezone in container is not configured. Adjust it manually. ## # The default user is 'ubuntu' with password 'ubuntu'! # Use the 'sudo' command to run tasks as root in the container. ## [root@localhost ~]# lxc-create -t ubuntu -n c2 Checking cache download in /var/cache/lxc/precise/rootfs-amd64 ... Copy /var/cache/lxc/precise/rootfs-amd64 to /var/lib/lxc/c2/rootfs ... Copying rootfs to /var/lib/lxc/c2/rootfs ... Generating locales... en_US.UTF-8... up-to-date Generation complete. Creating SSH2 RSA key; this may take some time ... Creating SSH2 DSA key; this may take some time ... Creating SSH2 ECDSA key; this may take some time ... Timezone in container is not configured. Adjust it manually. ## # The default user is 'ubuntu' with password 'ubuntu'! # Use the 'sudo' command to run tasks as root in the container. ## [root@localhost ~]# lxc-create -t ubuntu -n c3 Checking cache download in /var/cache/lxc/precise/rootfs-amd64 ... Copy /var/cache/lxc/precise/rootfs-amd64 to /var/lib/lxc/c3/rootfs ... Copying rootfs to /var/lib/lxc/c3/rootfs ... Generating locales... en_US.UTF-8... up-to-date Generation complete. Creating SSH2 RSA key; this may take some time ... Creating SSH2 DSA key; this may take some time ... Creating SSH2 ECDSA key; this may take some time ... Timezone in container is not configured. Adjust it manually. ## # The default user is 'ubuntu' with password 'ubuntu'! # Use the 'sudo' command to run tasks as root in the container. ##
Edit container config file to create veth network types. Container config files for my system is located at /var/lib/lxc/<container name>/config
Container c1 attached to bridge br0, c2 to br1 and c3 to br2
Below sample container c1 config file /var/lib/lxc/c1/config
# Network configuration lxc.network.type = veth lxc.network.hwaddr = 00:16:3e:80:81:03 lxc.network.flags = up lxc.network.link = br0 lxc.network.ipv4 = 192.168.2.1/24
below sample container c2 config file /var/lib/lxc/c2/config
# Network configuration lxc.network.type = veth lxc.network.hwaddr = 00:16:3e:6e:70:a7 lxc.network.flags = up lxc.network.link = br1 lxc.network.ipv4 = 192.168.2.2/24
Below sample container c3 config file /var/lib/lxc/c3/config
# Network configuration lxc.network.type = veth lxc.network.hwaddr = 00:16:3e:18:56:d5 lxc.network.flags = up lxc.network.link = br2 lxc.network.ipv4 = 192.168.2.3/24
Check container network interface. Use lxc-attach command to list ip interfaces
#lxc-attach -n <container name> /sbin/ip addr
//container c1 assigned with ip address 192.168.2.1 [root@localhost]# lxc-attach -n c1 /sbin/ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 67: eth0@if68: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 00:16:3e:80:81:03 brd ff:ff:ff:ff:ff:ff inet 192.168.2.1/24 brd 192.168.2.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::216:3eff:fe80:8103/64 scope link valid_lft forever preferred_lft forever //container c2 assigned ip address 192.168.2.2 [root@localhost]# lxc-attach -n c2 /sbin/ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 69: eth0@if70: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 00:16:3e:6e:70:a7 brd ff:ff:ff:ff:ff:ff inet 192.168.2.2/24 brd 192.168.2.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::216:3eff:fe6e:70a7/64 scope link valid_lft forever preferred_lft forever //container c3 assigned ip address 192.168.2.3 [root@localhost]# lxc-attach -n c3 /sbin/ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 71: eth0@if72: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 00:16:3e:18:56:d5 brd ff:ff:ff:ff:ff:ff inet 192.168.2.3/24 brd 192.168.2.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::216:3eff:fe18:56d5/64 scope link valid_lft forever preferred_lft forever
Check bridge status. As can be seen three virtual ethernet interfaces are attached to each bridge
[root@localhost]# brctl show bridge name bridge id STP enabled interfaces br0 8000.c2f691d238b0 no veth0 veth5 vethOC0NPG br1 8000.d28623e04c88 no veth1 veth2 vethSQUCH3 br2 8000.9a127ad7bf76 no veth3 veth4 veth577WF3
At this point we have created this topology. Containers c1,c2 & c3 attached to respective bridges and all bridges connected in a ring. Our L2 network is ready for testing
Check spanning tree (stp) status on bridges. As can be seen stp is disabled on all bridges as every bridge is a designated root bridge
[root@localhost]# brctl showstp br0 br0 bridge id 8000.c2f691d238b0 designated root 8000.c2f691d238b0 root port 0 path cost 0 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.00 hello timer 1.66 tcn timer 0.00 topology change timer 0.00 gc timer 53.31 flags veth0 (1) port id 8001 state forwarding designated root 8000.c2f691d238b0 path cost 2 designated bridge 8000.c2f691d238b0 message age timer 0.00 designated port 8001 forward delay timer 0.00 designated cost 0 hold timer 0.66 flags veth5 (2) port id 8002 state forwarding designated root 8000.c2f691d238b0 path cost 2 designated bridge 8000.c2f691d238b0 message age timer 0.00 designated port 8002 forward delay timer 0.00 designated cost 0 hold timer 0.66 flags vethOC0NPG (3) port id 8003 state forwarding designated root 8000.c2f691d238b0 path cost 2 designated bridge 8000.c2f691d238b0 message age timer 0.00 designated port 8003 forward delay timer 0.00 designated cost 0 hold timer 0.66 flags [root@localhost]# brctl showstp br1 br1 bridge id 8000.d28623e04c88 designated root 8000.d28623e04c88 root port 0 path cost 0 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.00 hello timer 1.04 tcn timer 0.00 topology change timer 0.00 gc timer 45.06 flags veth1 (1) port id 8001 state forwarding designated root 8000.d28623e04c88 path cost 2 designated bridge 8000.d28623e04c88 message age timer 0.00 designated port 8001 forward delay timer 0.00 designated cost 0 hold timer 0.04 flags veth2 (2) port id 8002 state forwarding designated root 8000.d28623e04c88 path cost 2 designated bridge 8000.d28623e04c88 message age timer 0.00 designated port 8002 forward delay timer 0.00 designated cost 0 hold timer 0.04 flags vethSQUCH3 (3) port id 8003 state forwarding designated root 8000.d28623e04c88 path cost 2 designated bridge 8000.d28623e04c88 message age timer 0.00 designated port 8003 forward delay timer 0.00 designated cost 0 hold timer 0.04 flags [root@localhost]# brctl showstp br2 br2 bridge id 8000.9a127ad7bf76 designated root 8000.9a127ad7bf76 root port 0 path cost 0 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.00 hello timer 1.07 tcn timer 0.00 topology change timer 0.00 gc timer 16.98 flags veth3 (1) port id 8001 state forwarding designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 0.00 designated port 8001 forward delay timer 0.00 designated cost 0 hold timer 0.07 flags veth4 (2) port id 8002 state forwarding designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 0.00 designated port 8002 forward delay timer 0.00 designated cost 0 hold timer 0.07 flags veth577WF3 (3) port id 8003 state forwarding designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 0.00 designated port 8003 forward delay timer 0.00 designated cost 0 hold timer 0.07 flags
Login to container c2 and ping to container c1 (192.168.2.1). Because of loop in the topology network will experience broadcast storm and create congestion. Ping will intermittently pass.
You can confirm the broadcast storm by checking packet counts on bridges. You will see packet counts increment at very high rate
ubuntu@c1:~$ ifconfig eth0 Link encap:Ethernet HWaddr 00:16:3e:80:81:03 inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::216:3eff:fe80:8103/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:87 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:648 (648.0 B) TX bytes:18666 (18.6 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:56 errors:0 dropped:0 overruns:0 frame:0 TX packets:56 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1 RX bytes:5440 (5.4 KB) TX bytes:5440 (5.4 KB) ubuntu@c1:~$ ip route 192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.1 ubuntu@c1:~$ ping 192.168.2.2 PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data. 64 bytes from 192.168.2.2: icmp_req=13 ttl=64 time=4009 ms 64 bytes from 192.168.2.2: icmp_req=19 ttl=64 time=3007 ms 64 bytes from 192.168.2.2: icmp_req=20 ttl=64 time=2005 ms [root@localhost ~]# ifconfig br0 br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::c0f6:91ff:fed2:38b0 prefixlen 64 scopeid 0x20<link> ether c2:f6:91:d2:38:b0 txqueuelen 1000 (Ethernet) RX packets 233826705 bytes 49096357638 (45.7 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@localhost ~]# ifconfig br0 br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::c0f6:91ff:fed2:38b0 prefixlen 64 scopeid 0x20<link> ether c2:f6:91:d2:38:b0 txqueuelen 1000 (Ethernet) RX packets 233921147 bytes 49127334614 (45.7 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Stop ping and check bridge packet count again. As can be seen packets counts are are still incrementing at a very high rate. Packets are circulating in the network due to topology loop
[root@localhost]# ifconfig br0 br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::c0f6:91ff:fed2:38b0 prefixlen 64 scopeid 0x20 ether c2:f6:91:d2:38:b0 txqueuelen 1000 (Ethernet) RX packets 158174506 bytes 28354814180 (26.4 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@localhost]# ifconfig br0 br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::c0f6:91ff:fed2:38b0 prefixlen 64 scopeid 0x20 ether c2:f6:91:d2:38:b0 txqueuelen 1000 (Ethernet) RX packets 159069415 bytes 28530733852 (26.5 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@localhost]# ifconfig br1 br1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::d086:23ff:fee0:4c88 prefixlen 64 scopeid 0x20 ether d2:86:23:e0:4c:88 txqueuelen 1000 (Ethernet) RX packets 160435925 bytes 28800879432 (26.8 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 3 bytes 258 (258.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@localhost]# ifconfig br1 br1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::d086:23ff:fee0:4c88 prefixlen 64 scopeid 0x20 ether d2:86:23:e0:4c:88 txqueuelen 1000 (Ethernet) RX packets 161046481 bytes 28923084308 (26.9 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 3 bytes 258 (258.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@localhost]# ifconfig br2 br2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::9812:7aff:fed7:bf76 prefixlen 64 scopeid 0x20 ether 9a:12:7a:d7:bf:76 txqueuelen 1000 (Ethernet) RX packets 162073378 bytes 29129182172 (27.1 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@localhost]# ifconfig br2 br2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::9812:7aff:fed7:bf76 prefixlen 64 scopeid 0x20 ether 9a:12:7a:d7:bf:76 txqueuelen 1000 (Ethernet) RX packets 162616331 bytes 29238047188 (27.2 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Enable spanning tree on all bridges, br0,br1 & br2.
As can be seen bridge br2 is elected as designated root bridge by spanning tree protocol. This is because the mac address of br2 is lower then br0 & br1
Bridge br1 port veth1(1) blocked by stp to break the loop and now there is no broadcast storm. The new topology after stp enabled should look like this
You ask why veth1 blocked this is because spanning tree packets received on this port has higher path cost than packet received on veth2 which is directly connected to root bridge
Another logical view
[root@localhost]# brctl stp br0 on [root@localhost]# brctl stp br1 on [root@localhost]# brctl stp br2 on [root@localhost]# brctl show bridge name bridge id STP enabled interfaces br0 8000.c2f691d238b0 yes veth0 veth5 vethOC0NPG br1 8000.d28623e04c88 yes veth1 veth2 vethSQUCH3 br2 8000.9a127ad7bf76 yes veth3 veth4 veth577WF3 //all ports in bridge br0 are in forwarding state [root@localhost]# brctl showstp br0 br0 bridge id 8000.c2f691d238b0 designated root 8000.9a127ad7bf76 root port 2 path cost 2 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.00 hello timer 0.00 tcn timer 0.00 topology change timer 0.00 gc timer 62.44 flags TOPOLOGY_CHANGE veth0 (1) port id 8001 state forwarding designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.c2f691d238b0 message age timer 0.00 designated port 8001 forward delay timer 0.00 designated cost 2 hold timer 0.00 flags veth5 (2) port id 8002 state forwarding designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 18.79 designated port 8002 forward delay timer 0.00 designated cost 0 hold timer 0.00 flags vethOC0NPG (3) port id 8003 state forwarding designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.c2f691d238b0 message age timer 0.00 designated port 8003 forward delay timer 0.00 designated cost 2 hold timer 0.00 flags //port veth1(1) is in blocking state [root@localhost]# brctl showstp br1 br1 bridge id 8000.d28623e04c88 designated root 8000.9a127ad7bf76 root port 2 path cost 2 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.00 hello timer 0.00 tcn timer 0.00 topology change timer 0.00 gc timer 73.66 flags veth1 (1) port id 8001 state blocking designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.c2f691d238b0 message age timer 19.64 designated port 8001 forward delay timer 0.00 designated cost 2 hold timer 0.00 flags veth2 (2) port id 8002 state forwarding designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 19.64 designated port 8001 forward delay timer 0.00 designated cost 0 hold timer 0.00 flags vethSQUCH3 (3) port id 8003 state forwarding designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.d28623e04c88 message age timer 0.00 designated port 8003 forward delay timer 0.00 designated cost 2 hold timer 0.57 flags //all ports in bridge br2 are in forwarding state [root@localhost]# brctl showstp br2 br2 bridge id 8000.9a127ad7bf76 designated root 8000.9a127ad7bf76 root port 0 path cost 0 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.00 hello timer 1.68 tcn timer 0.00 topology change timer 0.00 gc timer 66.65 flags veth3 (1) port id 8001 state forwarding designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 0.00 designated port 8001 forward delay timer 0.00 designated cost 0 hold timer 0.68 flags veth4 (2) port id 8002 state forwarding designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 0.00 designated port 8002 forward delay timer 0.00 designated cost 0 hold timer 0.68 flags veth577WF3 (3) port id 8003 state forwarding designated root 8000.9a127ad7bf76 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 0.00 designated port 8003 forward delay timer 0.00 designated cost 0 hold timer 0.68 flags
Let’s try to change the root bridge selection. The easiest way to do is by changing bridge priority
Change bridge br2 priority to make it root bridge. Bridge with lower priority become root. In this case I am reducing bridge priority just by one 7999.
As can be seen port veth5(2) on bridge br0 transition to block state to break loop. Bridge br1 elected as root bridge
[root@localhost ~]# sudo brctl setbridgeprio br1 7999 [root@localhost]# brctl showstp br1 br1 bridge id 1f3f.d28623e04c88 designated root 1f3f.d28623e04c88 root port 0 path cost 0 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.00 hello timer 1.30 tcn timer 0.00 topology change timer 19.86 gc timer 243.09 flags TOPOLOGY_CHANGE TOPOLOGY_CHANGE_DETECTED veth1 (1) port id 8001 state forwarding designated root 1f3f.d28623e04c88 path cost 2 designated bridge 1f3f.d28623e04c88 message age timer 0.00 designated port 8001 forward delay timer 0.00 designated cost 0 hold timer 0.30 flags veth2 (2) port id 8002 state forwarding designated root 1f3f.d28623e04c88 path cost 2 designated bridge 1f3f.d28623e04c88 message age timer 0.00 designated port 8002 forward delay timer 0.00 designated cost 0 hold timer 0.30 flags vethSQUCH3 (3) port id 8003 state forwarding designated root 1f3f.d28623e04c88 path cost 2 designated bridge 1f3f.d28623e04c88 message age timer 0.00 designated port 8003 forward delay timer 0.00 designated cost 0 hold timer 0.30 flags [root@localhost ns]# brctl showstp br0 br0 bridge id 8000.c2f691d238b0 designated root 1f3f.d28623e04c88 root port 1 path cost 2 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.00 hello timer 0.00 tcn timer 0.00 topology change timer 0.00 gc timer 223.34 flags TOPOLOGY_CHANGE veth0 (1) port id 8001 state forwarding designated root 1f3f.d28623e04c88 path cost 2 designated bridge 1f3f.d28623e04c88 message age timer 19.57 designated port 8001 forward delay timer 0.00 designated cost 0 hold timer 0.00 flags veth5 (2) port id 8002 state blocking designated root 1f3f.d28623e04c88 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 19.57 designated port 8002 forward delay timer 0.00 designated cost 2 hold timer 0.00 flags vethOC0NPG (3) port id 8003 state forwarding designated root 1f3f.d28623e04c88 path cost 2 designated bridge 8000.c2f691d238b0 message age timer 0.00 designated port 8003 forward delay timer 0.00 designated cost 2 hold timer 0.55 flags [root@localhost]# brctl showstp br2 br2 bridge id 8000.9a127ad7bf76 designated root 1f3f.d28623e04c88 root port 1 path cost 2 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.00 hello timer 0.00 tcn timer 0.00 topology change timer 0.00 gc timer 0.32 flags TOPOLOGY_CHANGE veth3 (1) port id 8001 state forwarding designated root 1f3f.d28623e04c88 path cost 2 designated bridge 1f3f.d28623e04c88 message age timer 18.34 designated port 8002 forward delay timer 0.00 designated cost 0 hold timer 0.00 flags veth4 (2) port id 8002 state forwarding designated root 1f3f.d28623e04c88 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 0.00 designated port 8002 forward delay timer 0.00 designated cost 2 hold timer 0.00 flags veth577WF3 (3) port id 8003 state forwarding designated root 1f3f.d28623e04c88 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 0.00 designated port 8003 forward delay timer 0.00 designated cost 2 hold timer 0.00 flags
The new topology should look like this
Let’s change the root port on br0 from veth0 to veth5. The easiest way to do that is by changing the port cost on veth0. Edit port cost on veth0 to 10. This will cause veth5 to move to forwarding state and veth0 to blocking.
As can be seen port veth0 transitioned to blocking state
[root@localhost ~]# brctl setpathcost br0 veth0 10 [root@localhost ~]# brctl showstp br0 br0 bridge id 8000.c2f691d238b0 designated root 1f3f.d28623e04c88 root port 2 path cost 4 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.00 hello timer 0.00 tcn timer 0.00 topology change timer 0.00 gc timer 226.87 flags TOPOLOGY_CHANGE veth0 (1) port id 8001 state blocking designated root 1f3f.d28623e04c88 path cost 10 designated bridge 1f3f.d28623e04c88 message age timer 18.81 designated port 8001 forward delay timer 0.00 designated cost 0 hold timer 0.00 flags veth5 (2) port id 8002 state listening designated root 1f3f.d28623e04c88 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 18.81 designated port 8002 forward delay timer 12.41 designated cost 2 hold timer 0.00 flags vethOC0NPG (3) port id 8003 state forwarding designated root 1f3f.d28623e04c88 path cost 2 designated bridge 8000.c2f691d238b0 message age timer 0.00 designated port 8003 forward delay timer 0.00 designated cost 4 hold timer 0.78 flags [root@localhost ~]# brctl showstp br0 br0 bridge id 8000.c2f691d238b0 designated root 1f3f.d28623e04c88 root port 2 path cost 4 max age 20.00 bridge max age 20.00 hello time 2.00 bridge hello time 2.00 forward delay 15.00 bridge forward delay 15.00 ageing time 300.00 hello timer 0.00 tcn timer 0.00 topology change timer 0.00 gc timer 32.26 flags veth0 (1) port id 8001 state blocking designated root 1f3f.d28623e04c88 path cost 10 designated bridge 1f3f.d28623e04c88 message age timer 19.07 designated port 8001 forward delay timer 0.00 designated cost 0 hold timer 0.00 flags veth5 (2) port id 8002 state forwarding designated root 1f3f.d28623e04c88 path cost 2 designated bridge 8000.9a127ad7bf76 message age timer 19.01 designated port 8002 forward delay timer 0.00 designated cost 2 hold timer 0.00 flags vethOC0NPG (3) port id 8003 state forwarding designated root 1f3f.d28623e04c88 path cost 2 designated bridge 8000.c2f691d238b0 message age timer 0.00 designated port 8003 forward delay timer 0.00 designated cost 4 hold timer 0.00 flags
Other useful command is to check mac table
[root@localhost ~]# brctl showmacs br0 port no mac addr is local? ageing timer 2 00:16:3e:18:56:d5 no 10.04 2 00:16:3e:6e:70:a7 no 12.00 3 00:16:3e:80:81:03 no 17.73 2 c2:f6:91:d2:38:b0 yes 0.00 1 e2:74:c6:8f:fc:a6 yes 0.00 1 e2:74:c6:8f:fc:a6 yes 0.00 3 fe:0e:18:bc:7e:e3 yes 0.00 3 fe:0e:18:bc:7e:e3 yes 0.00 2 fe:fe:da:e0:3a:09 no 0.81
You can dump packets on virtual interfaces and bridge
[root@localhost ~]# tcpdump -i veth5 -XX tcpdump: WARNING: veth5: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on veth5, link-type EN10MB (Ethernet), capture size 65535 bytes 22:03:48.738230 STP 802.1d, Config, Flags [none], bridge-id 8000.9a:12:7a:d7:bf:76.8002, length 35 0x0000: 0180 c200 0000 fefe dae0 3a09 0026 4242 ..........:..&BB 0x0010: 0300 0000 0000 1f3f d286 23e0 4c88 0000 .......?..#.L... 0x0020: 0002 8000 9a12 7ad7 bf76 8002 0001 1400 ......z..v...... 0x0030: 0200 0f00 .... 22:03:50.738236 STP 802.1d, Config, Flags [none], bridge-id 8000.9a:12:7a:d7:bf:76.8002, length 35 0x0000: 0180 c200 0000 fefe dae0 3a09 0026 4242 ..........:..&BB 0x0010: 0300 0000 0000 1f3f d286 23e0 4c88 0000 .......?..#.L... 0x0020: 0002 8000 9a12 7ad7 bf76 8002 0001 1400 ......z..v...... 0x0030: 0200 0f00
Welcome to cloudNet 