» I hate walls of text. Take me to the downloads! «

Updated October 11th, 2010: Added support for Twisted applications and epoll (and possibly kqueue).

Updated December 3rd, 2010: Changed a few things with the patch distribution. See comments for details. Be aware that this information is or will soon be deprecated. Twisted 10.2 is now available along with a number of improvements, and this patch will likely be ported to plugin status. I am leaving this post mostly intact for historical purposes, although corrections have been made to switch the namespace to tx from twistedpatch to clarify further that this patch has absolutely nothing to do with the wonderful folks who write Twisted.

Introduction

IPv6 support doesn’t exist in the base distribution of Twisted Python (the current version as of this writing is v10.1). Over the years, there have been several inquiries related to IPv6 support including a proposed patch, but support for the protocol is still forthcoming. Personally, I don’t like patching a base distribution. After all, it’ll be overwritten the next time I update, and I really don’t want to go through the effort of patching a second time. And who’s to whether or not the patch will apply cleanly in a few months? If you’re not certain this is an important subject, consider that some statistics estimate IPv4 exhaustion will occur in about 230+ days. Food for thought.

My solution is a little different than just simply modifying a handful of files somewhere in $PYTHONPATH and borrows from some of the methods Zope has used for quite some time with their hot fixes. Instead of patching Twisted directly, I have elected to monkey patch Twisted at runtime. This holds several advantages:

• Whenever IPv6 support is finally added to Twisted, it should be fairly simple to remove this patch. Simply change two lines: Replace the appropriate import statement for the Twisted reactor and replace the listenTCP6 method call with whatever Twisted eventually settles on.
• The patch consists of logically separate Python code and is therefore easier to maintain, update, and make other interesting changes to.
• Zope uses monkey patching to issue hot fixes; if it’s good enough for them, it’s good enough for us.
• There’s no need to modify your base Twisted distribution. That’s what this patch does at runtime. This means less hassle every time your distribution pushes an update to Twisted.
• Less code, less duplication. All IPv6 implementations in this patch make use of existing Twisted classes. Nearly everything is provided by subclassing IPv4 Twisted classes and overriding the appropriate methods. This means that IPv6 support should be functionally equivalent to IPv4–when it’s finished.

A word of warning: This will not currently work with .tac Twisted apps and you it has not been tested with a reactor other than the select reactor. This patch only works if you are launching the reactor yourself. If someone wants to take this and modify it to work with twistd, feel free. It shouldn’t be too hard. Also, be aware that IPv6 support appears to be going forward in the Twisted base distribution, so this patch is probably superfluous. I have posted a version of the Twisted patch that should work with twistd. This means you can now use the patch in your .tac Twisted application code! The same caveats apply, of course, and there are some additional usage instructions. See below. You should only consider this as an interim solution. This patch is not a perfect solution, it isn’t the best solution, but it is a solution that precludes you from having to patch Twisted directly.

It’s worth mentioning that this patch uses listenTCP6 instead of outright replacing listenTCP; the latter appears to be the preferred solution. I don’t necessarily agree. Instead, I recommend using the listenTCP6 nomenclature, because it provides the opportunity to listen separately on IPv4 and IPv6 interfaces simply by changing the method call. However, if you wish to use a single method (like listenTCP) to listen on both IPv4 and IPv6 interfaces, simply comment out line 86 in tcp6.py:

            s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1)

But before you do, you should probably consider the implications of using IPv4-mapped addresses which is what the socket may fall back to using when an IPv4 endpoint is connected. You were warned.

I also argue that adding another method like listenTCP6 is in developers’ best interests because this reason is no longer valid. DNS AAAA records extend IPv6 support to the same hostname an IPv4 address can reference; after all, how does one expect KAME.net implemented the dancing turtle?

Usage Instrutions – Using the Reactor Directly

Usage is exceedingly basic. Extract the twistedpatch into your source directory and replace this import:

from twisted.internet import reactor

With this import:

from tx.ipv6.internet import reactor

This does not import a separate reactor. Instead, what you receive is the exact same reactor import as you would from importing twisted.internet (in fact, that’s what the twistedpatch does) with the listenTCP6 method patched into it. You may then change your application code to read as follows:

if __name__ == "__main__":
    app = MyApp()
    reactor.listenTCP(8080, app)
    reactor.listenTCP6(8080, app) # Add IPv6 support.
    reactor.run()

Usage Instrutions – Making a Twisted Application

As with the listenTCP call, I have committed further blasphemy and added a new class to the Twisted application framework. This allows you to isolate and control IPv4 and IPv6 instances independently without having to worry about IPv4-mapped addresses and other interesting side effects. Remember: This solution isn’t optimal; the Twisted developers would much rather have a single TCP instance to keep the API clean. (I disagree, because IPv6 is a different protocol entirely compared to IPv4… but no matter.) Here’s how you would construct a Twisted application with this patch:

# Import the Twisted service. DO NOT import the internet features.
from twisted.application import service
 
# Import the internet features separately from the twisted patch.
from tx.ipv6.application import internet
 
application = service.Application("myapp")
internet.TCPServer(8000, Application()).setServiceParent(application)
internet.TCP6Server(8000, Application()).setServiceParent(application) # Note the use of TCP6 here

This will launch two separate factory application instances, both on port 8000, using IPv4 and IPv6. You can control each instance independently.

Obligatory Warnings

I have neither presented this patch to Twisted Matrix Laboratories nor have I addressed IPv6 with them in any way. Thus, you should infer that this patch is entirely unauthorized and unsupported by Twisted Matrix Labs. This patch is mostly unfinished and I have yet to run any of Twisted’s unit tests on it. I suspect they will probably fail. It does, however, appear to work rather well with Cyclone:

2010-10-06 14:54:05-0600 [HTTPConnection,0,2001:470:d:407:2d62:6624:eac6:1b96] 200 GET / (2001:470:d:407:2d62:6624:eac6:1b96) 3.29ms

Effectively, here’s what you should expect:

• There is no UDP support. Yet. It should be trivial to add. Simply subclass the appropriate items in twisted.internet.udp and copy the results into tx/ipv6/internet/udp6.py.
• Address resolution as supplied by Twisted will probably break. Frankly, anything related to addresses internally in Twisted will probably break when faced with IPv6. This is expected.
• listenTCP6 will listen only on IPv6 addresses. This is by design; if there is a compelling reason to outright replace listenTCP, I may implement that in the future. As it stands right now, I suspect that IPv4-mapped addresses have a potential to cause more issues than allowing them would otherwise solve. I also don’t believe that this encumbers the Twisted API in any way. After all, Twisted does have listenUNIX for domain sockets–why not add something unique for IPv6?
• Importing the reactor from tx.ipv6.internet instead of twisted.internet should not break existing code. If it does, there’s a problem. This patch does not introduce a new reactor, it simply patches the existing one.
• This patch is fully unsupported by Twisted Matrix Laboratories. Do not submit tickets to them if you have this patch installed; instead, revert the two changes (mentioned above), and test your code. If the problem persists, then it is probably a bug in your code–or a bug in Twisted (unlikely). If the problem disappears, then it’s a problem in my code, and you may complain to me about it. Do not complain to Twisted Matrix under any circumstance unless you are absolutely convinced that it is a problem with Twisted.
• I am not a particularly good Python programmer, and this patch was written hastily in about a half hour. My expertise currently lies in the unfortunate realm of PHP and Java with about 3 or 4 other languages (Python included) added somewhere into that mix. I don’t know enough about Twisted internals or Python to know whether or not I am committing a horrible blasphemy with this patch. Thus, if you don’t like this patch, please submit corrections or write your own. I really don’t mind. I’m just one guy who happens to want IPv6 support distributed as widely as possible, and if this patch can help meet or inspire others to meet that goal, I’ll be happy.

Downloads

If I haven’t scared you off with the notion that this patch might just kick your puppy late one night or bring a swarm of locusts barreling through your neighborhood, perhaps you wish to give it a try. Download it here:

twistedpatch-ipv6.tar.gz
MD5(42e3e8047fbbff165f5a598dbeff7129)
SHA1(a297c882ea2267155e948bcc7d7f2a28a869d953)

twistedpatch-ipv6.zip
MD5(14ecb5ad3dfd9780d1baa8204d907865)
SHA1(ea46ea482aed38e06f1a49dbd672beba45a26dba)

If you’ve ever written web applications in Python using Pylons or TurboGears (>=1.1), you’re probably familiar with an incredibly handy tool known as Routes. It’s modeled after the Ruby on Rails routes system and does quite a marvelous job at mapping friendly, SEO URLs to useful code. If you’re not aware of it, you really should check it out.

However, I was curious to see if it were possible to create a Routes-inspired PHP library that would perform essentially the same task. I’m sure there’s an already-written PHP routes system, and I’ve allowed myself to fall into the not invented here trap. But, that’s part of the fun in coding: If you don’t like what someone else wrote or you want to give it a hand yourself, no one is stopping you. It’s great.

Thus, I’ve written Routing, the PHP routes system. It’s similar in syntax and intent to Python Routes with the exception that it’s PHPized (in other words: bastardized). If you read the sources, you might be able to see a little Python influence here and there (along with my frustrations toward PHP as a language vented in silence). I wouldn’t recommend it as anything short of entertainment as yet; I haven’t cleaned up the sources, and I expect they’re going to be really messy for a little while. The route search mechanism is awful, and I plan on refactoring it in a future revision. For now, the system works, and that’s good enough. (That’s my INTJ side talking.)

So, if you’re curious enough to take a look, head over to the Routing Trac page. I have some downloads posted there in a variety of formats (.zip, .tar.gz, and .tar.bz2). There’s no API documentation yet. I’m going to add it soon. Since Routing was born on Wednesday, August 5th, 2009 it’s quite young. However, I have some preliminary unit tests written (also incomplete) that you can read to get a general understanding of how Routing works. If you’re more interested in the principle behind Routing, you might want to read the documentation for Python Routes. Routes and Routing work roughly the same with the exception that Routing, unlike Python Routes, is extraordinarily buggy. I hope I can evolve this into something of an acceptable sibling to Routes but that will take time and a whole lot of effort. I have a few additional ideas, too.

Links of the Week will be delayed until tomorrow or Friday.

Today, I’d like to share with you a real treat. It happened well into the evening last night when my coherent thought processes were lost in a semi-drunken stupor induced by lack of sleep (I don’t drink, but I do have slight insomnia sometimes). What was most ironic is that in the five or so years of writing in Python (off and on) I have never encountered this particular problem.

Click that fancy link (below) to read more. This is a fairly lengthy article, so make sure you have some free time! Read more…