I apologize for the quickie here; it’s late, I’m tired, and I just cobbled together a solution to a problem that’s been bothering me since this weekend. First, some backstory since I know you guys appreciate it when I go off on tangents. (And if you don’t, you can just scroll passed this nonsense.)

I’ve been migrating most of my services on my home file server into their own containers. I already did this on my VPS, and I plan on doing it to others eventually, because the service isolation is somewhat helpful and prevents stupid mistakes from doing equally stupid things to the host. Containers aren’t a panacea, of course, and the man pages specifically warn against using them as a security apparatus. It’s not effective, so they say, but I’m also a believer in defense-in-depth.

Anyway, the intent has been mostly a matter of isolation. I have a Minecraft server running, among other things, and I’ve been wanting to isolate it from the rest of the system. Not only does this make configuration somewhat easier since I can have service-specific user accounts without polluting the host passwd entries, but it means I can provide some package isolation as well (no Java on the host–yay!). This isn’t without its shortcomings, mind you, and it’s been one Hell of an interesting battle, particularly when IPv6 is thrown into the mix. But I digress, and the server-specific configuration is a welcome post for another day.

In the coming weeks, I’ll see about making a post on using systemd-nspawn for container isolation, why I chose it over Docker and LXC, how to get it working with user namespaces, and how to configure it with IPv6 addresses routed to each container (rather than through auto-config). Hint: you need routable prefixes like a /48 or /56–trying to (ab)use the neighbor discovery proxy and the respective sysctl settings to segment parts of your /64 non-routable prefix won’t work. Don’t even bother, because it’s not worth the misery.

Meanwhile, back to business: Since I do quite a bit of development of various sorts on my desktop, I also wanted to muck about with containers (which I’ve done for isolating builds), but I wanted to bridge the container with my network-facing interface in a manner that it would happily pull down IPv4 and IPv6 addresses from DHCP. My network, on both IP protocols, is serviced by DHCPv4 and DHCPv6 with more-or-less static assignments tossed out to the appropriate systems with a few exceptions. Specifically, connected devices either get their automatically configured IPv6 address, a static address plus an automatically configured address, or some mix (or absence) thereof. Ideally, containers should receive these addresses as well and behave just as an ordinary OS instance or virtual machine would.

Unfortunately, if you follow the systemd-networkd guides for Arch, you’ll quickly wind up in a circumstance where your network is only going to be half-configured. You might be able to get the IPv4 addresses you expect, if you configure the bridge for DHCP, but your IPv6 range isn’t necessarily going to work even if it does accumulate a set of addresses. Bummer.

Oh, and pay attention to casing: If you enter any of the values entirely in lowercase even by accident, nothing will work. I made this mistake, and my eyes never saw anything wrong because I was too focused on reading “bridge” to notice that it should have been entered as “Bridge.” The obsession, in this case, with the word “bridge” apparently served as an override for sensible parsing that included letter case, but this is a fairly common problem when you’ve been steeped in a similar class of issues for days on multiple systems; it’s a similar phenomenon to semantic satiation wherein words begin to look “strange” if you stare at them too long.

For the purposes of this discussion, we’ll assume that we’ve created files in a similar spirit to the Arch guide for systemd-networkd bridges: vbr0.netdev, vbr0.network, and whatever the name is of your hardware interface (mine is enp6s0, so I named its configuration rather creatively: enp6s0.network).

As with the guide, our files are as follows:

# vbr0.netdev
[NetDev]
Name=vbr0
Kind=bridge

# vbr0.network
[Match]
Name=vbr0
 
[Network]
DHCP=yes

(Notice DHCP=yes above. If you need static assignments, change this accordingly.)

And finally:

# enp6s0.network
[Match]
Name=enp6s0
 
[Network]
IPv6AcceptRouterAdvertisements=no
Bridge=vbr0

Oh! What’s this IPv6AcceptRouterAdvertisements=no? More importantly why do we need it? We want to accept IPv6 RAs on our network, don’t we?

The answer might surprise you: Yes and no. Yes, we need router advertisements, but no, we don’t need them on the bridge’s slave device. (Bonus: You won’t find this in the current documentation.) If you fail to add this option, your physical ethernet device will collect an autoconfiguration address, probably accumulate the appropriate route information, configure itself for IPv6, all while your bridge device does the same thing. You’ll be left with an interface that works for IPv4, is configured for both IPv4 and IPv6, but refuses to do anything with its IPv6 assignments. Obviously, if your network isn’t running radvd or similar, you won’t need this, and you certainly won’t need this if you’re not using IPv6. However, if you’re not running IPv6, you probably aren’t reading this article, are you?

If you’ve made it this far, I assume you’re interested in the final moment of truth and the culmination of our journey together. I therefore present to you a systemd-nspawn container with DHCP assignments for IPv4 and IPv6 auto configuration (addresses redacted for privacy):

[root@buildbot network]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: host0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1e:21:2b:fc:51:b9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.5.24/23 brd 192.168.5.255 scope global dynamic host0
       valid_lft 938sec preferred_lft 938sec
    inet6 [prefix redacted]:664e:9f3d/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 [prefix redacted]:fefc:51b9/64 scope global mngtmpaddr noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::1c21:2bff:fefc:51b9/64 scope link 
       valid_lft forever preferred_lft forever

If you haven’t bothered planning your IPv6 migration, you really ought to. The inevitable is coming sooner or later. You do want to be prepared, don’t you?

I recently updated the VPS this blog is sitting on. Coincidentally, this also updated nginx to the latest version and broke everything. I didn’t think much of it at the time, but when I linked a friend to this post over on my fun blog, he was delivered to the default nginx page. Puzzled, I poked around for a while, mostly examining DNS records and server configurations. I couldn’t find anything wrong.

Then I had a eureka moment.

I’m on IPv6 at home. I have this site (and others) configured to use IPv6. It hadn’t occurred to me until then that it might be protocol related. Using curl (curl -4 and curl -6), I confirmed my suspicions. Although the server was listening on TCP and TCP6, it was only serving up the vhosts on IPv6 and not IPv4. IPv4 was receiving the standard welcome page.

I knew that I had configured the server appropriately for both stacks. I’ve read through the docs. I combed through dozens of blog posts documenting the process. I was convinced the server was correctly configured. I must’ve fiddled with it for a good hour or so, reviewing documentation and the likes to no avail.

Infuriating.

Since nginx 1.2 or 1.3 (I can’t remember precisely), it’s been necessary to add ipv6only=off to the listen directives in order to support a dual stack environment. It’s my understanding this trick doesn’t work on some BSDs, but I know for a fact it worked fine under Linux. Or so I thought. I tried it successfully under Arch and Ubuntu with identical results with the exception that I neglected to recall one minor detail: My Arch install updated to nginx 1.4-something well after I had configured my desktop for developing on a dual IPv4/IPv6 stack. I suspect it’s probably broken in the same manner. But, I use it strictly for development, so I’m not particularly concerned whether or not it works on IPv4. I don’t use the protocol much within my network, so why worry, right?

To continue: I decided to take another stab at it and discovered something curious. Previously, all that was required to enable dual-stack support in nginx was to add the following to whatever was configured as the default host

    listen [::]:80 ipv6only=off default_server

And then all subsequent vhosts simply required

    listen[::]:80;

That’s all. It used to work–like magic. But, sadly, magic eventually runs out. This is why electronics stop working once you let all the “magic smoke” escape. Sorry, it’s an old electrical engineering joke my father has oft repeated. I guess it’s brushed off onto me.

Anyway, here’s the solution. You might find it contrary to some of the antiquated information out there lurking on various blogs dating back from 2011 through the middle of 2013. It works for nginx 1.4.5 (and possibly earlier versions), but the trick is to add this to the default vhost configuration

    listen[::]:80 ipv6only=on default_server;
    listen 80 default_server;

And for all subsequent vhosts

    listen[::]:80;
    listen 80;

I should note it works fine without adding the ipv6only=on directive, just like the generic vhost config (above). I believe I’ve read that this is because the default behavior enables ipv6only automatically. However, if you’re running a slightly older version, you might need to keep it. Hence why I’m not going to remove it from my examples. Better safe than sorry, right?

default_server is (hopefully) obvious, but only required if you want to provide a default site (or page) for users hitting your web server’s IP. Or for ancient browsers that haven’t been taught how to use the Host header. Are there any of those left?

So, the trick is that you need two listen directives. Period. Yes, even for TLS/SSL. If you skip these directives on any vhost, the missing protocol binding will be skipped for that vhost. I suspect this is probably documented somewhere. The problem though is that there are literally dozens of blogs pointing the old instructions that used to work. These are now deprecated. Following them will only lead to sadness.

Initial frustration aside, I find meshes well with my preferences. It’s more explicit and there’s no question which protocols nginx will use when binding to the configure port or ports. However, it will cause headaches for IPv6-enabled sites migrating from nginx 1.2. So, if you’re running Ubuntu and have decided to update in order to gain access to newer features (websocket support, SPDY, et al), expect breakage. More importantly, be absolutely certain you’ve independently tested all of your deployed sites using IPv4 and IPv6. Make liberal use of the -4 and -6 switches for curl. It’ll save you from unpleasant surprises.

» I hate walls of text. Take me to the downloads! «

Updated October 11th, 2010: Added support for Twisted applications and epoll (and possibly kqueue).

Updated December 3rd, 2010: Changed a few things with the patch distribution. See comments for details. Be aware that this information is or will soon be deprecated. Twisted 10.2 is now available along with a number of improvements, and this patch will likely be ported to plugin status. I am leaving this post mostly intact for historical purposes, although corrections have been made to switch the namespace to tx from twistedpatch to clarify further that this patch has absolutely nothing to do with the wonderful folks who write Twisted.

Introduction

IPv6 support doesn’t exist in the base distribution of Twisted Python (the current version as of this writing is v10.1). Over the years, there have been several inquiries related to IPv6 support including a proposed patch, but support for the protocol is still forthcoming. Personally, I don’t like patching a base distribution. After all, it’ll be overwritten the next time I update, and I really don’t want to go through the effort of patching a second time. And who’s to whether or not the patch will apply cleanly in a few months? If you’re not certain this is an important subject, consider that some statistics estimate IPv4 exhaustion will occur in about 230+ days. Food for thought.

My solution is a little different than just simply modifying a handful of files somewhere in $PYTHONPATH and borrows from some of the methods Zope has used for quite some time with their hot fixes. Instead of patching Twisted directly, I have elected to monkey patch Twisted at runtime. This holds several advantages:

• Whenever IPv6 support is finally added to Twisted, it should be fairly simple to remove this patch. Simply change two lines: Replace the appropriate import statement for the Twisted reactor and replace the listenTCP6 method call with whatever Twisted eventually settles on.
• The patch consists of logically separate Python code and is therefore easier to maintain, update, and make other interesting changes to.
• Zope uses monkey patching to issue hot fixes; if it’s good enough for them, it’s good enough for us.
• There’s no need to modify your base Twisted distribution. That’s what this patch does at runtime. This means less hassle every time your distribution pushes an update to Twisted.
• Less code, less duplication. All IPv6 implementations in this patch make use of existing Twisted classes. Nearly everything is provided by subclassing IPv4 Twisted classes and overriding the appropriate methods. This means that IPv6 support should be functionally equivalent to IPv4–when it’s finished.

A word of warning: This will not currently work with .tac Twisted apps and you it has not been tested with a reactor other than the select reactor. This patch only works if you are launching the reactor yourself. If someone wants to take this and modify it to work with twistd, feel free. It shouldn’t be too hard. Also, be aware that IPv6 support appears to be going forward in the Twisted base distribution, so this patch is probably superfluous. I have posted a version of the Twisted patch that should work with twistd. This means you can now use the patch in your .tac Twisted application code! The same caveats apply, of course, and there are some additional usage instructions. See below. You should only consider this as an interim solution. This patch is not a perfect solution, it isn’t the best solution, but it is a solution that precludes you from having to patch Twisted directly.

It’s worth mentioning that this patch uses listenTCP6 instead of outright replacing listenTCP; the latter appears to be the preferred solution. I don’t necessarily agree. Instead, I recommend using the listenTCP6 nomenclature, because it provides the opportunity to listen separately on IPv4 and IPv6 interfaces simply by changing the method call. However, if you wish to use a single method (like listenTCP) to listen on both IPv4 and IPv6 interfaces, simply comment out line 86 in tcp6.py:

            s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1)

But before you do, you should probably consider the implications of using IPv4-mapped addresses which is what the socket may fall back to using when an IPv4 endpoint is connected. You were warned.

I also argue that adding another method like listenTCP6 is in developers’ best interests because this reason is no longer valid. DNS AAAA records extend IPv6 support to the same hostname an IPv4 address can reference; after all, how does one expect KAME.net implemented the dancing turtle?

Usage Instrutions – Using the Reactor Directly

Usage is exceedingly basic. Extract the twistedpatch into your source directory and replace this import:

from twisted.internet import reactor

With this import:

from tx.ipv6.internet import reactor

This does not import a separate reactor. Instead, what you receive is the exact same reactor import as you would from importing twisted.internet (in fact, that’s what the twistedpatch does) with the listenTCP6 method patched into it. You may then change your application code to read as follows:

if __name__ == "__main__":
    app = MyApp()
    reactor.listenTCP(8080, app)
    reactor.listenTCP6(8080, app) # Add IPv6 support.
    reactor.run()

Usage Instrutions – Making a Twisted Application

As with the listenTCP call, I have committed further blasphemy and added a new class to the Twisted application framework. This allows you to isolate and control IPv4 and IPv6 instances independently without having to worry about IPv4-mapped addresses and other interesting side effects. Remember: This solution isn’t optimal; the Twisted developers would much rather have a single TCP instance to keep the API clean. (I disagree, because IPv6 is a different protocol entirely compared to IPv4… but no matter.) Here’s how you would construct a Twisted application with this patch:

# Import the Twisted service. DO NOT import the internet features.
from twisted.application import service
 
# Import the internet features separately from the twisted patch.
from tx.ipv6.application import internet
 
application = service.Application("myapp")
internet.TCPServer(8000, Application()).setServiceParent(application)
internet.TCP6Server(8000, Application()).setServiceParent(application) # Note the use of TCP6 here

This will launch two separate factory application instances, both on port 8000, using IPv4 and IPv6. You can control each instance independently.

Obligatory Warnings

I have neither presented this patch to Twisted Matrix Laboratories nor have I addressed IPv6 with them in any way. Thus, you should infer that this patch is entirely unauthorized and unsupported by Twisted Matrix Labs. This patch is mostly unfinished and I have yet to run any of Twisted’s unit tests on it. I suspect they will probably fail. It does, however, appear to work rather well with Cyclone:

2010-10-06 14:54:05-0600 [HTTPConnection,0,2001:470:d:407:2d62:6624:eac6:1b96] 200 GET / (2001:470:d:407:2d62:6624:eac6:1b96) 3.29ms

Effectively, here’s what you should expect:

• There is no UDP support. Yet. It should be trivial to add. Simply subclass the appropriate items in twisted.internet.udp and copy the results into tx/ipv6/internet/udp6.py.
• Address resolution as supplied by Twisted will probably break. Frankly, anything related to addresses internally in Twisted will probably break when faced with IPv6. This is expected.
• listenTCP6 will listen only on IPv6 addresses. This is by design; if there is a compelling reason to outright replace listenTCP, I may implement that in the future. As it stands right now, I suspect that IPv4-mapped addresses have a potential to cause more issues than allowing them would otherwise solve. I also don’t believe that this encumbers the Twisted API in any way. After all, Twisted does have listenUNIX for domain sockets–why not add something unique for IPv6?
• Importing the reactor from tx.ipv6.internet instead of twisted.internet should not break existing code. If it does, there’s a problem. This patch does not introduce a new reactor, it simply patches the existing one.
• This patch is fully unsupported by Twisted Matrix Laboratories. Do not submit tickets to them if you have this patch installed; instead, revert the two changes (mentioned above), and test your code. If the problem persists, then it is probably a bug in your code–or a bug in Twisted (unlikely). If the problem disappears, then it’s a problem in my code, and you may complain to me about it. Do not complain to Twisted Matrix under any circumstance unless you are absolutely convinced that it is a problem with Twisted.
• I am not a particularly good Python programmer, and this patch was written hastily in about a half hour. My expertise currently lies in the unfortunate realm of PHP and Java with about 3 or 4 other languages (Python included) added somewhere into that mix. I don’t know enough about Twisted internals or Python to know whether or not I am committing a horrible blasphemy with this patch. Thus, if you don’t like this patch, please submit corrections or write your own. I really don’t mind. I’m just one guy who happens to want IPv6 support distributed as widely as possible, and if this patch can help meet or inspire others to meet that goal, I’ll be happy.

Downloads

If I haven’t scared you off with the notion that this patch might just kick your puppy late one night or bring a swarm of locusts barreling through your neighborhood, perhaps you wish to give it a try. Download it here:

twistedpatch-ipv6.tar.gz
MD5(42e3e8047fbbff165f5a598dbeff7129)
SHA1(a297c882ea2267155e948bcc7d7f2a28a869d953)

twistedpatch-ipv6.zip
MD5(14ecb5ad3dfd9780d1baa8204d907865)
SHA1(ea46ea482aed38e06f1a49dbd672beba45a26dba)