» I hate walls of text. Take me to the downloads! «
Updated October 11th, 2010: Added support for Twisted applications and epoll (and possibly kqueue).
Updated December 3rd, 2010: Changed a few things with the patch distribution. See comments for details. Be aware that this information is or will soon be deprecated. Twisted 10.2 is now available along with a number of improvements, and this patch will likely be ported to plugin status. I am leaving this post mostly intact for historical purposes, although corrections have been made to switch the namespace to tx from twistedpatch to clarify further that this patch has absolutely nothing to do with the wonderful folks who write Twisted.
Introduction
IPv6 support doesn’t exist in the base distribution of Twisted Python (the current version as of this writing is v10.1). Over the years, there have been several inquiries related to IPv6 support including a proposed patch, but support for the protocol is still forthcoming. Personally, I don’t like patching a base distribution. After all, it’ll be overwritten the next time I update, and I really don’t want to go through the effort of patching a second time. And who’s to whether or not the patch will apply cleanly in a few months? If you’re not certain this is an important subject, consider that some statistics estimate IPv4 exhaustion will occur in about 230+ days. Food for thought.
My solution is a little different than just simply modifying a handful of files somewhere in $PYTHONPATH and borrows from some of the methods Zope has used for quite some time with their hot fixes. Instead of patching Twisted directly, I have elected to monkey patch Twisted at runtime. This holds several advantages:
- Whenever IPv6 support is finally added to Twisted, it should be fairly simple to remove this patch. Simply change two lines: Replace the appropriate import statement for the Twisted reactor and replace the
listenTCP6method call with whatever Twisted eventually settles on. - The patch consists of logically separate Python code and is therefore easier to maintain, update, and make other interesting changes to.
- Zope uses monkey patching to issue hot fixes; if it’s good enough for them, it’s good enough for us.
- There’s no need to modify your base Twisted distribution. That’s what this patch does at runtime. This means less hassle every time your distribution pushes an update to Twisted.
- Less code, less duplication. All IPv6 implementations in this patch make use of existing Twisted classes. Nearly everything is provided by subclassing IPv4 Twisted classes and overriding the appropriate methods. This means that IPv6 support should be functionally equivalent to IPv4–when it’s finished.
A word of warning: This will not currently work with I have posted a version of the Twisted patch that should work with .tac Twisted apps and you it has not been tested with a reactor other than the select reactor. This patch only works if you are launching the reactor yourself. If someone wants to take this and modify it to work with twistd, feel free. It shouldn’t be too hard. Also, be aware that IPv6 support appears to be going forward in the Twisted base distribution, so this patch is probably superfluous.twistd. This means you can now use the patch in your .tac Twisted application code! The same caveats apply, of course, and there are some additional usage instructions. See below. You should only consider this as an interim solution. This patch is not a perfect solution, it isn’t the best solution, but it is a solution that precludes you from having to patch Twisted directly.
It’s worth mentioning that this patch uses listenTCP6 instead of outright replacing listenTCP; the latter appears to be the preferred solution. I don’t necessarily agree. Instead, I recommend using the listenTCP6 nomenclature, because it provides the opportunity to listen separately on IPv4 and IPv6 interfaces simply by changing the method call. However, if you wish to use a single method (like listenTCP) to listen on both IPv4 and IPv6 interfaces, simply comment out line 86 in tcp6.py:
s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1) |
But before you do, you should probably consider the implications of using IPv4-mapped addresses which is what the socket may fall back to using when an IPv4 endpoint is connected. You were warned.
I also argue that adding another method like listenTCP6 is in developers’ best interests because this reason is no longer valid. DNS AAAA records extend IPv6 support to the same hostname an IPv4 address can reference; after all, how does one expect KAME.net implemented the dancing turtle?
Usage Instrutions – Using the Reactor Directly
Usage is exceedingly basic. Extract the twistedpatch into your source directory and replace this import:
from twisted.internet import reactor |
With this import:
from tx.ipv6.internet import reactor |
This does not import a separate reactor. Instead, what you receive is the exact same reactor import as you would from importing twisted.internet (in fact, that’s what the twistedpatch does) with the listenTCP6 method patched into it. You may then change your application code to read as follows:
if __name__ == "__main__": app = MyApp() reactor.listenTCP(8080, app) reactor.listenTCP6(8080, app) # Add IPv6 support. reactor.run() |
Usage Instrutions – Making a Twisted Application
As with the listenTCP call, I have committed further blasphemy and added a new class to the Twisted application framework. This allows you to isolate and control IPv4 and IPv6 instances independently without having to worry about IPv4-mapped addresses and other interesting side effects. Remember: This solution isn’t optimal; the Twisted developers would much rather have a single TCP instance to keep the API clean. (I disagree, because IPv6 is a different protocol entirely compared to IPv4… but no matter.) Here’s how you would construct a Twisted application with this patch:
# Import the Twisted service. DO NOT import the internet features. from twisted.application import service # Import the internet features separately from the twisted patch. from tx.ipv6.application import internet application = service.Application("myapp") internet.TCPServer(8000, Application()).setServiceParent(application) internet.TCP6Server(8000, Application()).setServiceParent(application) # Note the use of TCP6 here |
This will launch two separate factory application instances, both on port 8000, using IPv4 and IPv6. You can control each instance independently.
Obligatory Warnings
I have neither presented this patch to Twisted Matrix Laboratories nor have I addressed IPv6 with them in any way. Thus, you should infer that this patch is entirely unauthorized and unsupported by Twisted Matrix Labs. This patch is mostly unfinished and I have yet to run any of Twisted’s unit tests on it. I suspect they will probably fail. It does, however, appear to work rather well with Cyclone:
2010-10-06 14:54:05-0600 [HTTPConnection,0,2001:470:d:407:2d62:6624:eac6:1b96] 200 GET / (2001:470:d:407:2d62:6624:eac6:1b96) 3.29ms
Effectively, here’s what you should expect:
- There is no UDP support. Yet. It should be trivial to add. Simply subclass the appropriate items in twisted.internet.udp and copy the results into
tx/ipv6/internet/udp6.py. - Address resolution as supplied by Twisted will probably break. Frankly, anything related to addresses internally in Twisted will probably break when faced with IPv6. This is expected.
listenTCP6will listen only on IPv6 addresses. This is by design; if there is a compelling reason to outright replacelistenTCP, I may implement that in the future. As it stands right now, I suspect that IPv4-mapped addresses have a potential to cause more issues than allowing them would otherwise solve. I also don’t believe that this encumbers the Twisted API in any way. After all, Twisted does havelistenUNIXfor domain sockets–why not add something unique for IPv6?- Importing the reactor from
tx.ipv6.internetinstead oftwisted.internetshould not break existing code. If it does, there’s a problem. This patch does not introduce a new reactor, it simply patches the existing one. - This patch is fully unsupported by Twisted Matrix Laboratories. Do not submit tickets to them if you have this patch installed; instead, revert the two changes (mentioned above), and test your code. If the problem persists, then it is probably a bug in your code–or a bug in Twisted (unlikely). If the problem disappears, then it’s a problem in my code, and you may complain to me about it. Do not complain to Twisted Matrix under any circumstance unless you are absolutely convinced that it is a problem with Twisted.
- I am not a particularly good Python programmer, and this patch was written hastily in about a half hour. My expertise currently lies in the unfortunate realm of PHP and Java with about 3 or 4 other languages (Python included) added somewhere into that mix. I don’t know enough about Twisted internals or Python to know whether or not I am committing a horrible blasphemy with this patch. Thus, if you don’t like this patch, please submit corrections or write your own. I really don’t mind. I’m just one guy who happens to want IPv6 support distributed as widely as possible, and if this patch can help meet or inspire others to meet that goal, I’ll be happy.
Downloads
If I haven’t scared you off with the notion that this patch might just kick your puppy late one night or bring a swarm of locusts barreling through your neighborhood, perhaps you wish to give it a try. Download it here:
twistedpatch-ipv6.tar.gz MD5(42e3e8047fbbff165f5a598dbeff7129) SHA1(a297c882ea2267155e948bcc7d7f2a28a869d953) twistedpatch-ipv6.zip MD5(14ecb5ad3dfd9780d1baa8204d907865) SHA1(ea46ea482aed38e06f1a49dbd672beba45a26dba)
4 Responses to “Twisted Python and IPv6”
4 things:
1. Our agents are everywhere. We know what you’re up to.
2. Please participate in the effort to fix this For Real™. The ticket you reference – http://tm.tl/3014 – is making progress very slowly because it’s exhausting to work out all the fiddly little details and get the fixes in the right procedural order for compatibility purposes. Nothing breaks up that type of exhaustion better than someone with a real, pressing interest in getting a fix committed.
3. Thanks for the disclaimers. It’s nice that you don’t dump responsibility for this on us :).
4. As of Twisted 10.2 (released just yesterday!), there’s a new plugin API you could use to distribute this support in a clean way without monkey-patching anything – http://tm.tl/4695 has some details. Since e.g. ‘twistd web’ uses endpoint descriptions (previously known as “strports”), you could do ‘twistd web –port my-tcp6-plugin:4321’ and immediately get IPv6 support.
Thanks for your interest, and for keeping the Twisted ecosystem thriving :).
I’m impressed and somewhat shocked. I think I need a moment to myself…
In reference to point 1: I apologize if WP sprayed various URIs with a slew of pingbacks. It may be annoying, but I suppose it’s a feature that has its use.
2: I’d absolutely love to help. My Python skills aren’t the best (I blame my day job and the shambling horror that is PHP). Regardless, I think IPv6 is important enough to warrant some work.
3: IPv6 supoprt isn’t something that can be pieced together overnight. While I may disagree on some points, at the end of the day it doesn’t matter; I did technical support prior to (and partway during) college, and I dislike it when people assign blame incorrectly to the wrong parties. I doubt that’s a concern, but people can be quite surprising at times…
4: Excellent! I haven’t had a chance to play with my projects that use the best networking library in existence due to the holidays, but I intend to follow each release when/if I have time.
I’m insanely impressed, shocked, and otherwise at a loss for words that Glyph, the founder of the Twisted project, would take time out of an undoubtedly busy schedule to post on some random Joe’s one-off blog! Neither my site nor I matter much in the grand scheme of things, but this does prove a great deal to me about the leadership (and by extension the community) behind Twisted. Thank you! I am greatly humbled.
(Apologies in advance for the monster comment here. I do tend to go on at length.)
If it did, I didn’t notice, so don’t worry about it. I was just saying that Twisted Matrix Labs’s insidious globe-spanning reach is inescapable. Nowhere is safe.
I hear this concern from potential contributors more and more these days. I guess that as Twisted’s codebase has gotten better and better over time, it has started projecting this aura of mystery, like you have to be some kind of super-genius to contribute. Please let me assure you that this isn’t the case. I should probably do a blog post of my own on this, soon. Of course, when writing code, it always helps if you are a super-genius, but the reason that Twisted’s code is generally good (and getting better) is not the raw genius of an individual contributor; it’s the process by which things get accepted.
Every change gets reviewed and improved when submitted. This means that pretty much nobody, myself included, ever just tosses off a patch and then immediately gets it included. There’s some feedback, it gets addressed, and if you’re really good, maybe it gets accepted on the second submission.
This doesn’t mean that your contributions are subjected to a round of mockery or derision! You can ask pretty much anyone who has been through this process. The feedback is direct, helpful, and almost always educational in one way or another. Contributions are so valuable that we are not going to be rude to a contributor :).
So you can be an absolutely terrible programmer, and you don’t have to worry about it. You’ll get lots of help on improving your code and the end product will be as good as we can make it.
The down-side of this process is that sometimes the original author runs out of steam, or gets busy with other things, and can’t finish responding to the feedback. That’s basically where the IPv6 ticket is right now; the design has pretty much been hammered out, and it’s pretty straightforward, there are already some patches, but the I’s need to be dotted and the T’s need to be crossed. If you catch up with the discussion on that ticket, you could probably help out much more easily than you think.
Ah, yes. I saw the best minds of my generation destroyed by PHP, etc.
Whatever the reason for your care in placing responsibility, it’s appreciated. Given the layer that Twisted occupies in the application’s stack, we get blamed for a lot of things that aren’t our fault, from the way that TCP works (no, you may NOT assume that one send() equals one recv()!) to operating system bugs (PTYs are hard to get right, apparently), to GUI library design mistakes (modal dialogs != blocking functions, people, let’s get this right).
One minor nit though: we like to encourage external/community projects to use ‘tx’ as a prefix, as opposed to the built-in twisted projects which use ‘twisted’. If you’re going to continue maintaining this patch, might I suggest that you refer to it as ‘tx’? (And perhaps also register it on Launchpad, so that you can add it to the Twisted Community Code super-project?)
That’s awful nice of you to say – can we quote you on it? :)
Thanks for saying so! I think you’ll quite like the plugin API in question, when you get a chance to play with it.
Finally:
No problem. It’s a pleasure to find someone posting such thoughtful description of issues with Twisted, especially when what you’re describing is basically a design problem with Twisted (adding your own IPv6 sockets shouldn’t be nearly this hard). Usually when people hit an issue like this with a library – “people” which occasionally includes myself, I’m sad to say – their initial reaction is to say “bah, this sucks, forget it”.
The only better things than such thoughtful comments, of course, are fully-documented, fully-tested patches submitted to our tracker, ready to apply :). It is well worth my time to stop by and add some commentary to your blog, if that would push you further in the direction of interacting with us directly, and getting your help maintaining this beast in the longer term. Everybody who is a Twisted core team member now was, at one point, just some random person with a problem with Twisted :).
Please feel free to drop by #twisted on Freenode or the twisted-python mailing list to discuss this or any other issues you have, or just to say hi.
That’s pretty hilarious. World domination one network stack at a time.
I think most F/OSS projects are in this same boat. Fortunately, I’m also well aware of the personality types (predominantly INTJ/INTP with a mix of others) that tend to dominate these fields. Consequently, I also understand that what might strike others as abruptness is typically nothing more than brutal honesty.
My primary concern is that I hope this original whining rant of mine didn’t completely come off as a whining rant. Okay, maybe a little whining, but I think we all need to vent from time to time.
Side note: I don’t necessarily view Twisted’s internals as some kind of “black magic.” There really isn’t such a thing (except maybe in the Windows’ universe), because with sufficient time, it’s possible to understand the whys and hows with regard to the internal workings of something. I just confess that I’m not a particularly fantastic Python programmer by any stretch of the imagination. Though, that isn’t really a huge concern with regards to a F/OSS project anyway. The premise of “many eyes” comes to mind.
Sigh. Don’t get me started on their idea of namespacing. Maintaining code others wrote in PHP (seriously, would it kill anyone to use prepared statements at least once in their life or learn something about SQL injection?) is bad enough, but I’m almost dreading the day significant code migration to 5.3 occurs.
Ugh.
I think that anyone who complains about Twisted internals should probably be forced to write a network stack on their own. Preferably on a lossy network so that they understand some of the issues described elsewhere. I’m not particularly good at networking code, but I do confess that I’ve written some myself both for purposes of self-education and, well, because I had to (unrelated, but this is fundamentally what lead me to discover Twisted a couple of years ago).
My primary motivation is that I have a couple of one-off projects as a pure hobby, written in Python. I like Twisted better than anything else because the interfaces are well defined, it’s insanely easy to do simple stuff with, and the coding standards used by you guys are actually sane and make for easy reading. Deferreds still make me want to occasionally kick puppies, but I think I’m coming around to understanding them a little better. The idea is fantastic, though.
No problem, that’s my own fault for not researching the Twisted community a little more and selecting something inappropriate. I’ll have that fixed tonight.
Before I post this anywhere else, though, I’d much rather rewrite it as a Twisted plugin (as you’ve suggested), then follow the various IPv6 discussions a little more closely. I also have some ideas that I’d like to hash out, if at all possible, but that’s mailing list discussion material. (Which I have subscribed to and will undoubtedly lurk for a little while to get a feel for the community.)
Certainly. It’s only the truth.
I don’t blame Twisted, to be honest. Yes, some things could be “easier” (for whatever value of “easy” you prefer), but being as it is written in Python, even things of moderate difficulty are easy to shoehorn into place. Design problem or otherwise, think of how many vendors are being paid to sell hardware that doesn’t yet support IPv6 and are refusing to honor any discussions thereof! Ahh, the benefits of F/OSS. (Though, it helps to have a greatly energetic project founder who’s willing to invest so much time into discussion alone–that’s you, by the way, Glyph!)
As I mentioned, I’ve subscribed to the mailing list and will be looking to do something with regards to this. I don’t have an ETA, but I will follow through with your advice and likely post it to the Twisted list.
Thanks again for the discussion!
Leave a comment