Earlier this month, I stumbled across this article on Slashdot and my comments inflamed a few posters with an uncomfortable truth:

Insecurity is your fault.

No, really, it is. Now that the storm has mostly blown over and much of this has been forgotten (plus I’ve gotten some time to sit down and write!), I’d like to explain why.

First, though, a brief recap for those of you who may have missed out on all the excitement.

Memcached: Security Hole or Convenient Misconfiguration?

I won’t explain what Memcached is. If you’re curious, the Wikipedia article details its usage model, behavior, and other interesting tidbits for the curious. However, the important part is that Memcached provides absolutely no method or mechanism by which to authenticate attached clients. Neither does Memcached provide an interface to protect data integrity–that’s your job. The whole Slashdot meltdown occurred when some security researchers discovered that many well-known sites failed to protect their Memcached instances, exposing sensitive data to the outside world including passwords, password hashes, e-mail addresses, and other assorted data typically stored in a Memcached instance. The surprising discovery here isn’t so much that the problem existed but that it afflicted the sites of several well known names including PBS! Really, the system administrators of the sites showcased in this study should be ashamed of themselves. (Maybe; I’m sure many of them were too busy to notice the err of their ways. I’m also sure that a sizable chunk of them felt a little more uncomfortable puckering of anatomical places better left to the imagination than is otherwise healthy.)

I also hate to point out the obvious here, but a simple firewall would have circumvented the issue entirely. The authors of the article in question emphasized this point by appending in their conclusion several strings containing “FW”–and then some. It’s not like setting up a firewall is difficult either, so I’m not sure if this is equal parts ignorance, oversight, laziness, economics, and harsh schedules for the IT staff involved. I don’t think any of these would make for a particularly good excuse. After all, it doesn’t take long to set up ipfw, iptables, and kin. There’s also plenty of documentation and quick-start guides available just a Google search away.

Unfortunately, the Slashdot article was slightly sensationalist and came close to leading readers into believing that running Memcached was a security hole. If you were to take a few minutes (do so now) to browse through the comments, you’ll find that they fall into roughly two categories:

• Security is the responsibility of the system administrator, network administrator, or whomever happens to be in charge of handling the servers, their configuration, and/or network topography.
• Security is the responsibility of the developer to ensure that their application doesn’t ship in a potentially hazardous state such that the exposure of critical data could be made accidentally.

Which statement do you agree with?

Take your time. I’ll be here all day.

It’s Your Fault

If you think it’s the developer’s responsibility to ensure their software doesn’t ship in a potentially hazardous state, you’re wrong, and Memcached is a perfect example why the responsibility of service configuration rests squarely on your shoulders. Of course, there are many reasons why the developers are absolved of responsibility, and I’ll attempt to outline some of them here. Essentially, though, if you’re blaming Memcached for this particular deficit, you’re demonstrating great ignorance of the F/OSS ecosystem.

How dare I point out the obvious!

Package Maintainers Know Everything

Most open source projects, particularly those centered around popular operating system distributions like FreeBSD, Gentoo, or Debian often have specific individuals who, largely on a volunteer basis, maintain a specific software package or packages. In many projects, these maintainers operate rather loosely as a committee whose primary objectives are to: 1) make sure the software installs and runs correctly on their target platform(s) and 2) configure the package to ensure that it suits the specific goals of their distribution (or not, sometimes inaction is as much of a goal as action). This is why package maintainers know everything, and I certainly don’t mean it in a derogatory sense: These individuals, typically picked by the community or by merit, understand both the software they’re maintaining as well as the environment they’re maintaining it for. Package maintainers are an integral part of a particular distribution’s microculture, and they understand what defaults are likely to work best and what won’t. There’s also a certain degree of personal preferences involved but package inclusions and behaviors are almost exclusively influenced by the community as a whole.

When it comes to Memcached, Debian-based distributions tend to configure it to listen exclusively on localhost. FreeBSD and Gentoo both leave Memcached as it ships–not configured and not turned on by default. (Ubuntu enables Memcached as soon as you install it unless you’re running Ubuntu server, in which case it remains disabled.) Since the default–not configured–instructs Memcached to listen on all available interfaces, those distributions that happen to ship Memcached in its default state are therefore the same ones that will expose it to the Big Bad Interwebs the very instant it happens to be enabled. Sort of.

When Configurations… are Not

This is a minor point of contention, but I feel it’s worthwhile to mention because it bears some relevance to the topic of Memcached security. Plus, since several comments on Slashdot made reference to some magical Memcached configuration floating around somewhere in the sky, I confess that I feel this overwhelming desire to point out the obvious. It’s a lot like honking at the motorist in front of you whose blissful ignorance of the surrounding world is responsible for backing up traffic for miles while the unyielding stoplight changes from green to yellow to red and back again–you just can’t help but lay it on thick. I’m also convinced that some of you share this blissful dream-like trance with our hypothetical motorist, because I otherwise cannot fathom why you’d think we software developers are out to get you. (Maybe we are, but that’s another story.)

Okay, here’s the earth-shattering news: Memcached has no configuration file. That’s it. That’s the anticlimactic reality my last paragraph was building up to. I don’t believe I can hear you honking yet, so let me say that again.

Memcached has no configuration file. If it has one, it’s a lie. It’s a lot like that cake. The configuration file is a lie.

Several OS vendors (FreeBSD isn’t one of them–same for Gentoo) happen to ship with their particular Memcached package a file conveniently named “memcached.conf.” To most of us servermonkeys, it would appear that this file has something to do with Memcached–and we’d be right. It’s amazing how that works. Ah, but such assumptions only go so far. You see, “memcached.conf” is just a convenience wrapper. Memcached itself has no integral facilities to read external configuration files and relies instead on command line parameters. In fact, I would argue that vendors who provide a separate configuration file of the sort may be misleading impressionable individuals into believing that Memcached ships with its own configuration file, and that those someones who got caught with their firewall down happened to get the short end of the stick by starting up the service without actually checking that file.

Admittedly, it’s easy to be mislead. Here’s a sample of the Ubuntu (server) Memcached configuration file:

# memcached default config file
# 2003 - Jay Bonci


But, if you were to continue reading…

# This configuration file is read by the start-memcached script provided as
# part of the Debian GNU/Linux distribution.

Oh! Memcached doesn’t read it after all! Good thing I read passed the top two lines, otherwise I would’ve been convinced it shipped with Memcached!

Fortunately, both FreeBSD and Gentoo configure Memcached through their RC system and make it fairly clear that any configuration is done directly to the Memcached daemon. FreeBSD configures Memcached via /etc/rc.conf. Gentoo configures Memcached via /etc/conf.d/memcached. Both of these configurations therefore operate in Memcached’s default mode of listening on all available interfaces.

The question, then, becomes a matter of whether Memcached is insecure by default.

If a Cache Server Listens on all Interfaces in the Forest… is it really There?

Memcached gained its reputation as a light weight, fast, minimalistic memory-based caching server partially by not integrating high-latency features like authentication, data integrity checking, and encryption. If the end user’s environment requires any one of these features, it’s up to the end user to implement them on top of Memcached–and most likely, other solutions. (You can run Memcached over an encrypted connection using a variety of techniques, for example, but they require some effort on your behalf to implement.) Failure to secure a network-accessible service isn’t the fault of the developers of that package. After all, if the server were properly firewalled, it wouldn’t be network-accessible outside the network for which it was intended.

(This, of course, assumes that the attackers don’t have access to your internal network. Granted, if this were the case, I suspect Memcached would be the least of your concerns.)

The obvious point of this entire debate–specifically, the one on Slashdot–is that Memcached shouldn’t be shipped to listen on all available interfaces. Certainly there are great points on either side of this issue, but the fundamental truth is that Memcached’s default behavior isn’t up to us to decide. The developers decide what goes, and I’m sure they have a good reason for attaching to every interface under the sun. In this case, the obvious solution isn’t to argue with me about why it wasn’t your fault you just exposed 15,000 passwords to every 16 year old pimply hacker with an Internet connection. Nay, you should argue with the developers of Memcached and explain to them why their software should listen on localhost by default. Maybe they’ll even agree with you and release a new version that does exactly that. On the other hand, they may just give a hearty laugh in your direction and explain that such a drastic change in default behavior would very likely impact the bazillion or so installations of Memcached that exist out there on the Interwebs the next time some system administrator goes to upgrade.

Generally speaking, it’s better to piss off the people who haven’t done enough research to know how to actually use the software than it is to squirt a healthy dose of Tabasco sauce into the shorts of the very people who probably have several dozen large-scale installations each and enough disposable income to donate generously to your project.

Not surprisingly, that brings me to my next point.

If You can’t Configure that Package Securely… It’s Your Fault

I really hate to harp on this point, but since it seems that there’s so many individuals out there bellyaching over the simple fact that Memcached doesn’t listen exclusively on localhost by default, I really have to hammer it until I drive my point home (or break my hammer).

It’s your fault.

Don’t like it? Tough. Write that sentence 300 times on some college ruled notebook paper and get back to me in the morning. No cheating! I’ll be counting every single one of ’em, too.

The implication of my attitude–or attitude “problem,” depending on how you take this–is that configuring any software you install is your business. If you configure it incorrectly and lose your company a million dollars, it’s your fault. If you configure it incorrectly and drive traffic to your site, earning your company a million dollars, it’s your fault. Conversely, if I write a software package, hypothetically, that advertises itself as some must-have network-connected service that’ll make your servers hum, your teeth whiter, and maybe cure world hunger and you install it without bothering to look at the manual to find out that, holy bat nuggets it really is network-connected, you won’t find a lot of sympathy from me when the entire Internet discovers a few dozen things better left secret just because you couldn’t read a two paragraph instruction manual or secure your server. It’s not my fault that my hypothetical software might be used in an inadequately protected environment anymore than it is the Memcached developers’ fault that their server can be accessed remotely without authentication.

Now that We’ve Established Blame, What Next?

If you happened to read the article(s) the Slashdot summary linked to and the first thing that came to mind was “firewall!”, consider yourself dismissed. You’re free to go home, and you’re getting an A for the course. You needn’t trouble yourself with reading through my inane ramblings, because you probably know a lot more than I do. However, if you’re still fuming out the ears blaming the Memcached developers for defaults that clearly should have been better chosen, you’re staying after class.

I torqued off a number of posters on Slashdot when I uttered these words, so I’ll utter them again: It is your responsibility as a systems administrator to understand the security implications of installing any given software package. If it’s installed on your server–doubly so if you’re the one installing it–you damn well better know what it’s going to do when it starts running. If you don’t have a clue what might happen or how to configure it, then shame on you when someone sniffs critical data from a service you didn’t configure correctly. System administration isn’t intended to be easy. System administration isn’t opening a couple of menu-driven wizards, clicking “next,” and hoping for the best. System administration is about best, secure practices; vigilance; researching and reading; understanding, mitigating, and managing risks; understanding the software you’re using; understanding the operating system you’re using; understanding your network environment and topography; awareness; and always, always, always keeping in mind that anything connected to the Internet is at risk. Sounds like a lot of work, doesn’t it?

“But,” I hear you protest, “system administration is about taking care of the users, resetting their passwords when they forget, and making sure everything is hunky-dory when they pull up their TPS reports! You’re describing a consultant’s job.”

Nope, son, that’s the help desk. They’re the colleged-aged kids up front who daydream of murderous intentions every time someone calls up thinking capslock is cruise control for cool and just can’t seem to get their password right. Consultants are the guys you hire to design (or build) the system. System administrators (probably you) are the guys who are hired to understand the system, fix it when it breaks, and be on call when Godzilla comes gnawing through the company’s upstream. If you think system administration should be easy, there are places you can go to for help.

Why Does it Matter?

Part of the reason I like the minimalistic approach FreeBSD and Gentoo both take to configuring (and I use that term loosely) Memcached is because they make no assumptions–other than the defaults, of course. It doesn’t matter whether you’re running a Memcached instance for a farm of 50 webservers and need to set it up to listen on all interfaces–uh oh! defaults!–since the machine is only attached to a private network. It doesn’t matter whether you’re going to be setting Memcached to use 64 megs (the default!) of RAM or 16 gigs. What does matter is that you’ve looked at the configuration your package maintainer–the guy who knows everything–shipped with the software you installed to figure out what it’s going to do. Better yet, it helps to read the manual page. The world isn’t full of pop-out coloring books, and if you’re going to maintain systems for a living, you’re going to have to do some reading. Manpages are a fantastic place to start.

I’ve read the excuse on Slashdot more often than I care to admit that the reason Memcached should listen by default on localhost is because there are far more instances installed by mom and pop web sites that just expect to install it and have it working than there are large scale installations that listen on multiple network interfaces (well, 0.0.0.0:11211 or :::11211). Is that true? I don’t know. Maybe there’s a significant number of webserver admins out there who have heard about this Memcached business and know only enough to know they need to install it. Maybe there’s a lot of people who just run apt-get install memcached, emerge memcached, or whatever install script happens to float their collective fleet of rubber rafts because they like to watch a bunch of text fly across their screen. I have no idea.

I do know that there’s a huge problem with the assumption that most administrators installing Memcached aren’t aware of the security implications. Why? Simple:

• Memcached is a semi-niche market solution for a large scale problem. By the time you start looking around for a solution to offloading some of your server load elsewhere, you’ve probably done enough research to know what Memcached is and why you need it.
• If you’re administering a large site that needs Memcached (see previous point), you’re probably well aware of network security. You wouldn’t let the world connect to every port on your home computer just for fun, so why would you let them do that to your web server?
• Most system administrators are competent, careful, methodical individuals who consider their server farm something like their own children. If we lesser beings so much as tried to touch their machines with our grubby mitts, they’d chase us out of the data center with a shotgun. This whole problem, and the article linked to by Slashdot, doesn’t mention these guys. Those same guys who spend sleepless nights making sure db-140.cluster.lax.internal-net.megacorp.com is running smoothly are the same guys who firewalled their Memcached servers and were not noticed in the report. They couldn’t be, because the report was about incorrectly configured servers. The world is full of surprises, and it may surprise you to learn that there are competent individuals out there.

The other excuse that drove me to the verge of insanity was the proposal that Memcached is a great object store that can increase the performance of a mom and pop webserver, so it’s just as likely to be installed on bobs-fancy-widgets.com as it is on a massive load-balanced site. Thus, since this target market likely doesn’t know the first thing about installing a webserver, Memcached should be set up safely by default. Fine–maybe this is true–but it also assumes that bobs-fancy-widgets.com isn’t firewalled. It should be, and if it’s not, then I’m afraid there’s not much any of us can do for him. Maybe Bob doesn’t know anything about firewalls. Maybe Bob is also receiving a gazillion SSH scans from China, too. Just wait ’til they find out his password is “password”.

I don’t have the data available that these security researchers do. Therefore, I have to base my assumptions on the sample data they released. The companies affected by this security oversight (it’s not a hole) weren’t small fly-by-night organizations. These were just about everything from trendy start-ups to major television network webservers! My guess? It was probably an honest mistake more often than not. Someone forgot to enable the firewall or configure Memcached. They got burned. It sucks, but it happens often enough to serve as a reminder to the rest of us.

Lesson Learned

There’s a valuable lesson to take away from this, and while I realize this essay is reaching almost epic proportions, it’s important enough to warrant just a few more sentences. Never make assumptions about your software’s configuration. If you’re running a business or being paid by a business to run their machines, you should always double-check (or triple-check) the configurations of software you’ve installed to ensure they’re going to run as you expect them to. Make certain to read the appropriate manpages for your software, too, because they often have valuable insight into what the software does or how to configure it. Search the Internet. Oh, and never forget–if you’re using a *nix-based system, there are tools to help you understand what’s running and what’s not. Use utilities like ps and netstat until you can read the output of these programs by heart. netstat is of particular importance; with it, you can understand at a glance what your system is reporting as listening on network interfaces. (-p helps, too, as does understanding similar non-Linux tools like FreeBSD’s sockstat.)

Remember: If some 16 year old hacker discovers the CEO’s password and sends out “Your Fired!” e-mails to the company, wipes the database server, and turns your web server into an archive that’d make the Pirate Bay blush, you’re the one who’s going to get that uncomfortable call at 3:00AM. The Memcached devs? Well, they’ll be sound asleep while you’re sweating bullets. After all, that’s what you’re paid for, right?