A close friend of mine recently decided it was time to update his Arch Linux installation. It had been over a year since the core OS was completely updated, and I did everything I could to discourage him from trying the usual pacman -Syu. The problem, for all intents and purposes, is that I knew it wouldn’t work. Much of the file system structure has changed dramatically over the course of the past year (the /lib move and the more recent merging of /bin and /sbin into their /usr counterparts) and a straightforward update was now difficult (but not impossible–more on this in a minute). Specifically, the glibc and filesystem packages have gone through several iterations even last July and would now be permanently blocking each other thanks to these two updates with no immediately obvious path forward.

I have observed some comments dismissing the update process from a pre-systemd installation as virtually impossible. While I suspect this is because the individual(s) asking for help are seen by others to be insufficiently experienced to do such an update, I’m not a particularly huge fan of the term “impossible” as it pertains to difficult feats–even for the inexperienced. After all, few things are truly impossible; it’s simply a matter of how much time and energy one is willing to invest in working toward a specific goal. Besides, if the recommended solution is to reinstall, why not figure out an optimal way to upgrade an “impossible” system? If it’s going to break either way, we may as well have some fun with it. (You have backups, right?) Even if you don’t have the necessary experience or knowledge relating to the underlying system, there’s no time like the present to begin the learning process.

So, Thursday evening, I set out looking for a solution. Fortuitously, I had an old Arch Linux virtual machine installed on my desktop (which, conveniently, is also Arch Linux) that dated back to approximately the same kernel revision as my friend’s (v3.3.6 for mine; v3.3.4 for his) and roughly analogous software. Like his system, my VM was also a pre-systemd installation, was still running glibc 2.15, and a very early version of filesystem-2012. In many regards, my VM was more or less in a state identical to that of the machine we needed to fix. However, the problem was compounded by an additional requirement. Console access was somewhat tedious for my friend to obtain because of the systems he had to go through, and he was exceptionally busy the afternoon we planned the updates. So, I had to to keep network access (particularly SSH) up and running as best as I could.

Disclaimer

This narrative guide is intended as general reference for anyone who might be in a similar situation: Pre-systemd, pre-filesystem moves, and operating with a requirement that the machine be (mostly) network-accessible throughout the duration of the update. Be aware that nothing in this guide should be considered authoritative. I am a user of Arch Linux. There’s still much about system internals (and Linux in general) that I don’t know or don’t fully understand. Consequently, I’m always learning, and there may be better alternatives to specific problems encountered during this update process. However, I do know enough about Arch to recommend that you never use –force unless you’re specifically instructed to do so. At no point during this update process should you use it. Doing so will assuredly destroy your system, and you’ll be forced to reinstall.

Secondly, this update process is not supported by the Arch Linux developers. If you have failed to maintain your system by keeping it up-to-date (and updating your Arch Linux installation is one of the core tenants of being an Arch user), you’re unlikely to receive much help since you’ve already established that your OS has been inadequately maintained. Furthermore, this update process relies on tricks using partial updates which are also unsupported. The developers and users of Arch who frequently the forums are nice folks who have lives outside of Arch, and due to the tremendous task presented to the community of supporting a relatively large user base, it’s impractical for them to spend a significant chunk of their time helping you with issues that–to avoid mincing words–are the fault of no one but yourself.

Thus, I hope this guide will be useful to those of you who may have a neglected Arch Linux box under your care. Be aware that I have targeted this guide to a system that was last updated somewhere in mid-2012. Although it should work for earlier systems, I’d highly recommend reading through the Arch news archives if you’re updating such a beast. You can usually determine where you need to begin by examining the version of your filesystem package, e.g. pacman -Qs filesystem. This guide should be general enough such that even if it doesn’t help you determine the exact update path for your system, you’ll be able to figure out where to get started.

Also, be sure to read through the guide in its entirety first. Check your installed packages, and make sure you’re not downgrading them. filesystem and glibc should never be downgraded at any point during this update otherwise you may exacerbate system breakage. This guide illustrates the update process with the x86_64 architecture. If you’re using a 32-bit system, you’ll need to replace all references to x86_64 with i686.

Read more…

So, KDE 4.10 just came out last week, the Arch repos sometime thereafter, and I updated today. I really love new versions of KDE–honestly–but this time something really bizarre happened. I updated, rebooted to the new kernel (I was running 3.7.4, the update was to 3.7.6), fired up KDE, and started loading my usual applications.

Then nothing worked quite right. I’d open one window, and the others I had open would disappear. Opening the launcher would also exhibit this problem: Open terminal windows would disappear the instant the launcher was clicked or any other application (including Dolphin) was opened. Infuriating. It seemed that I couldn’t have all my usual windows open at once–everything would disappear!

Fortunately, after some trial and error, I found the culprit. This latest version of KDE has made significant changes to ~/.kde4/share/config/kconf_updaterc. So, there’s two solutions:

Easy Mode

Log out, switch to a virtual terminal (ctrl+alt+F1), log in from there, and delete the file ~/.kde4/share/config/kconf_updaterc. Then switch back to X (usually ctrl+alt+F7 or F8) and log back in to KDE so it regenerates the file. Enjoy!

Hard Mode

Follow the previous steps but retain a backup of your .kde4 directory. Then run a diff on the two versions of kconf_updaterc and figure out what you want to keep, what line caused the problem, and retain as many of your previous settings as possible.

Otherwise, I hope this saves you some difficulty if you run into the same thing I have!

February 13th edit:

It seems that this post by “George” on the Arch Linux forums indicates a potentially easier solution, which is to change the window transparency settings. I changed the title transparency settings, but I’m not entirely sure if this will resolve the disappearing windows.

As I mentioned early on in part one (you may read it here), comments like this are somewhat frequent on the “newbie” forum over at the Arch Linux forums, although the one I linked is probably the mildest of the lot. While they follow the same motif, they range from individuals voicing veiled frustration over the fact that Arch doesn’t seem to “work” like their favorite distro, that it requires too much effort, or that Debian/Ubuntu/<insert distro here> is so much easier (for some value of easy). Fortunately, the majority of these posts are filtered out quickly by the moderators and dealt with accordingly (see the “Topics Going Nowhere” forum for samples). It’s one of the reasons I love the Arch community–the wheat is separated from the chaff early on, and consequently, most of the threads that live beyond a day or two tend to be useful, interesting, or informative.

Now, fair warning: I’ll be illustrating my point with a fair amount of allegory. There are also a few rather broad generalizations ahead, and I’m aware of this. I also don’t address related issues like self-sustenance or the benefits of having broad, practical knowledge (I’ve always wanted to make my own cheese, for instance). This is intentional.

So, let’s begin.

The problem, fundamentally, is not one that’s endemic to any one Linux distro in particular. It’s actually a consequence of society coddling the fools and bringing up the helpless. Before you feel offended, let me clarify that I don’t mean to sound brash. It isn’t the fault of those who are helpless per se. An efficient, productive society creates specialized niches, and sooner or later there’s a threshold that’s crossed where it’s no longer possible for any one individual to know everything they need to know in order to function without outsourcing some of their daily activities.

I realize that the word “outsourcing” has developed exceedingly negative connotations recently, particularly in the US, but it’s the only term that fits. Outsourcing isn’t inherently wrong, nor is it evil. In fact, you might be surprised to learn that you do it all the time. When you go to the grocery store, you’re buying products that have essentially been outsourced for you: Rather than baking your bread, you outsource the baking to the store and exchange some currency for the finished product. Rather than farming your own wheat or milling your own flour, you purchased bagged flour at the store. An efficient society is one where the individual can focus on other tasks (like programming) instead of being overly concerned about day to day needs (such as growing food). An efficient society therefore approaches a point where individual specialization is such that we’re all “helpless” to a point. For example, I’m not particularly mechanically inclined and wouldn’t know where to begin to look to fix a car, but I have family and friends who often look to me for help with technical matters (or building their own systems). We outsource to each other. Put another way, you and I outsource our issues in areas where we lack the expertise or the domain specific knowledge to those who have it.

As I alluded to earlier, this sort of helplessness isn’t a bad thing. It’s the hallmark of a productive, efficient society. However, it’s also important to know and understand these individual limitations before embarking into areas where one lacks the required knowledge. Ignorance is a refusal to learn; arrogance is a refusal to listen. Unfortunately, ignorance and arrogance often play a part in some of the discussions I’ve seen, and it’s all too common that the two traipse around together causing mischief where there otherwise should be none.

The problem then is that the efficiency of society–of outsourcing–has created a clime where a certain subset of the population accumulates the mistaken belief that their problems are the fault of others who are unwilling to help them. They fail to understand the difference between outsourcing a product, such as bread, and voluntary efforts like Arch Linux which require the consumer to be his or her own navigator in seas uncharted. Perhaps the most sinister thing about this lack of understanding is that it’s not primarily cognitive–the consumer in this scenario fully understands that open source projects are largely staffed by volunteers–but it is a form of devious dissonance that leads them to behave as though they’re still outsourcing. It therefore expresses itself as a deep seated inability to make the correlation between F/OSS (Free/Open Source Software) and volunteer labor versus outsourcing and paid-for products. I won’t cover it further here, but if you’re interested in a much more lengthy discussion along these lines, it would behoove you to read the excellent article Linux is not Windows.

I should point out that it’s one thing to say “Help! I don’t have a clue what I’m doing! Can someone point me in the right direction?” but it’s another thing entirely to say “Help! I can’t get my favorite feature to work. It worked fine on my previous platform. Can yours not do this too?” The former expresses ignorance combined with a willingness to learn–this is a good thing, because it’s easily correctable. The latter combines both ignorance and arrogance, culminating in a distinct unwillingness to do research while placing the blame on those who would otherwise offer their help.

But I digress.

To continue: Because of the efficiency and specialization that our society encourages–another good thing–there are some individuals who therefore think that their problem is not their own. The “problem ownership” is then shifted, in their minds, to the people trying to help. This is exacerbated by those who refuse to do any legwork on their own, and it is this personality type that often leaves with unresolved problems and a certain level of anger.

What this means for Arch but not limited to Arch is that there’s always going to be some small number of people who aren’t necessarily willing to help themselves. The clueful ones may eventually acknowledge the error of their ways, apologize, and possibly redeem themselves after some embarrassment. Others are beyond redemption, and when they find their repeated inquiries for help are no longer wanted on the Arch forums, they’ll eventually become a bother to someone else.

However, I think that at least a small part of the problem stems from the evangelism we Arch users periodically exhibit. (Yes, I’m guilty of this.) Because we tout the benefits of rolling-release distros and the simplicity of Arch often without the appropriate warnings attached, newbie users hear these and immediately develop an unrealistic ideal of what Arch is (and can never be) and then project that into their pleas for help. On at least two occasions I’ve seen this develop, and the easiest way to spot these types are from remarks such as “Arch is supposed to be the best rolling-release distro, but I’ve never had these problems with ${other_distro} before!”

The best solution, of course, is to amend our evangelism with warnings like “Arch isn’t for everybody,” but I doubt that would work. We’re all aware that most people gloss over warnings (myself included) whenever a specific threshold of positive remarks is reached. If one’s mind is made up that the road is paved in gold, little attention will be paid to the sign that reads “there be dragons!”

The next best solution is to educate in addition to evangelizing. If you know someone who’s interested in Arch, point them to the forums and the wiki, but be sure to emphasize that Arch is a do-it-yourself distro. Educate interested souls on the merits of problem solving and figuring things out. (Bonus points if you link them to the late Richard Feynman’s book The Pleasure of Finding Things Out as a tongue-in-cheek gesture.) If your friend is someone who isn’t particularly interested in or adept at solving problems, Arch is very likely not the best match. That’s not to say they may not benefit from learning Arch, but you’ll likely save them (and yourself) from some frustration in the event of mismatched expectations.

I really like Arch Linux, and I’d love it if everyone I knew used it. I also think that’s an unrealistic expectation, because I know that not everyone finds enjoyment from the same things I do. If you’re considering using Arch, you need to be aware that you’re going to run into problems. The difficulty of the problems you’re likely to encounter will be determined by your relative skill; the more skilled you are, the easier a potential problem will be to solve. There will also come a time when an update may break the system, and you must be ready to spend an hour or more without access to a graphical environment (shell only). It also helps to familiarize yourself with chrooting Arch in the event you have to rescue your system from a live CD. You must also read the front page news articles prior to an update process, because important information about potentially breaking changes is posted there (if applicable). If this sounds like too much work, you may have to concede defeat. Arch may not be for you.