Linux networking sucks. XEN Networking sucks.

Yes, it does. For a while I started to move stuff away from physical machines onto XEN-virtualized servers. Using CentOS, this works reasonably well, except…
… well, when you use the regular xen network-bridge and vif-bridge scripts to set up and tear down your virtual interfaces, suddenly other services start to scream:

Jan 24 19:45:33 shirley kernel: xenbr0: port 1(vif0.0) entering forwarding state
Jan 24 19:45:33 shirley ntpd[4806]: sendto(192.53.103.104) (fd=18): Network is unreachable
Jan 24 19:45:33 shirley ntpd[4806]: sendto(131.188.3.221) (fd=18): Network is unreachable

Great. The bridge setup changed the network topology and the ntpd (which is bound to eth0 for multicast) doesn’t get it. But wait, there is more.

Jan 28 22:47:04 plucky kernel: xenbr0: port 2(peth0) entering forwarding state
Jan 28 22:47:04 plucky snmpd[2278]: error on subcontainer ‘’ insert (-1)
Jan 28 22:47:04 plucky last message repeated 12 times

So the snmpd chokes on topology changes?!? But hey! Who would run stuff like time sync and network monitoring on a production system? Well, just about, everyone???

The whole xen network setup and changing magic is pretty much useful for your little at-home-boot-every day box. Not for serious production needs (Well, I’m sure that lots of people will scream now in anger, that their production boxes work perfectly with the scripts. If you a) run snmpd and ntpd, b) check your log files on a regular base and c) don’t find the bugs described above, please let me know. Else, don’t. Because without time sync and monitoring, you are not running in production mode.).

So what to do? Don’t let xen setup the networking. Let the OS do it!

Because it has a marginally better understanding of what it is doing. And the scripts are written by people who are paid to think about more complex setups than the one-person, one-interface PC at home.

Setup a bridge for dom0: /etc/sysconfig/network-scripts/ifcfg-br0:

DEVICE=br0
ONBOOT=yes
BOOTPROTO=none
IPADDR=1.2.3.4
NETMASK=255.255.255.0
GATEWAY=1.2.3.1
NO_ALIASROUTING=yes
TYPE=Bridge

Now connect your physical interface to the bridge: /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
BRIDGE=br0
HWADDR=22:44:66:88:aa:cc

The last line should be the HW address of your physical interface. (If you don’t know what this is, please use the XEN supplied networking scripts and don’t bother…)

You must enable ip forwarding on your box. Add the following line to /etc/sysctl.conf:

net.ipv4.ip_forward = 1

Poof! Instant, stable, bridged networking when booting up.

However, starting xend produces a set of ugly veth / vif0. pairs. Because it unconditionally loads the netloop module which in turn creates four (four? Why is it always four? Why not a nice round number like, e.g. zero) pairs by default unless you explicitly disable this. Save yourself the googling. Just add

options netloop nloopbacks=0

to your /etc/modprobe.conf file. Do it. As long as the module is loaded, XEN is able to create its needed netloops on the fly. You really don’t need to have four more dangling around.

Almost there. Now edit your virtual host definitions in /etc/xen/auto and change the bridge name in the vif= lines from xenbr0 to br0. E.g. like this:

vif = [ ‘mac=ee:cc:aa:88:66:44, bridge=br0′, ]

And finally, open the /etc/xen/xend-config.sxp file and comment out all lines beginning with (network-script …). Really. You don’t need these any longer. Because the networking is already set up correctly. Keep the (vif-script vif-bridge) line.

==> Working, stable and reliable networking with XEN. And even snmpd and ntpd are happy.

linux networking xen

2 February 2008 | Netstuff | Comments