A close friend of mine recently decided it was time to update his Arch Linux installation. It had been over a year since the core OS was completely updated, and I did everything I could to discourage him from trying the usual pacman -Syu. The problem, for all intents and purposes, is that I knew it wouldn’t work. Much of the file system structure has changed dramatically over the course of the past year (the /lib move and the more recent merging of /bin and /sbin into their /usr counterparts) and a straightforward update was now difficult (but not impossible–more on this in a minute). Specifically, the glibc and filesystem packages have gone through several iterations even last July and would now be permanently blocking each other thanks to these two updates with no immediately obvious path forward.

I have observed some comments dismissing the update process from a pre-systemd installation as virtually impossible. While I suspect this is because the individual(s) asking for help are seen by others to be insufficiently experienced to do such an update, I’m not a particularly huge fan of the term “impossible” as it pertains to difficult feats–even for the inexperienced. After all, few things are truly impossible; it’s simply a matter of how much time and energy one is willing to invest in working toward a specific goal. Besides, if the recommended solution is to reinstall, why not figure out an optimal way to upgrade an “impossible” system? If it’s going to break either way, we may as well have some fun with it. (You have backups, right?) Even if you don’t have the necessary experience or knowledge relating to the underlying system, there’s no time like the present to begin the learning process.

So, Thursday evening, I set out looking for a solution. Fortuitously, I had an old Arch Linux virtual machine installed on my desktop (which, conveniently, is also Arch Linux) that dated back to approximately the same kernel revision as my friend’s (v3.3.6 for mine; v3.3.4 for his) and roughly analogous software. Like his system, my VM was also a pre-systemd installation, was still running glibc 2.15, and a very early version of filesystem-2012. In many regards, my VM was more or less in a state identical to that of the machine we needed to fix. However, the problem was compounded by an additional requirement. Console access was somewhat tedious for my friend to obtain because of the systems he had to go through, and he was exceptionally busy the afternoon we planned the updates. So, I had to to keep network access (particularly SSH) up and running as best as I could.

Disclaimer

This narrative guide is intended as general reference for anyone who might be in a similar situation: Pre-systemd, pre-filesystem moves, and operating with a requirement that the machine be (mostly) network-accessible throughout the duration of the update. Be aware that nothing in this guide should be considered authoritative. I am a user of Arch Linux. There’s still much about system internals (and Linux in general) that I don’t know or don’t fully understand. Consequently, I’m always learning, and there may be better alternatives to specific problems encountered during this update process. However, I do know enough about Arch to recommend that you never use –force unless you’re specifically instructed to do so. At no point during this update process should you use it. Doing so will assuredly destroy your system, and you’ll be forced to reinstall.

Secondly, this update process is not supported by the Arch Linux developers. If you have failed to maintain your system by keeping it up-to-date (and updating your Arch Linux installation is one of the core tenants of being an Arch user), you’re unlikely to receive much help since you’ve already established that your OS has been inadequately maintained. Furthermore, this update process relies on tricks using partial updates which are also unsupported. The developers and users of Arch who frequently the forums are nice folks who have lives outside of Arch, and due to the tremendous task presented to the community of supporting a relatively large user base, it’s impractical for them to spend a significant chunk of their time helping you with issues that–to avoid mincing words–are the fault of no one but yourself.

Thus, I hope this guide will be useful to those of you who may have a neglected Arch Linux box under your care. Be aware that I have targeted this guide to a system that was last updated somewhere in mid-2012. Although it should work for earlier systems, I’d highly recommend reading through the Arch news archives if you’re updating such a beast. You can usually determine where you need to begin by examining the version of your filesystem package, e.g. pacman -Qs filesystem. This guide should be general enough such that even if it doesn’t help you determine the exact update path for your system, you’ll be able to figure out where to get started.

Also, be sure to read through the guide in its entirety first. Check your installed packages, and make sure you’re not downgrading them. filesystem and glibc should never be downgraded at any point during this update otherwise you may exacerbate system breakage. This guide illustrates the update process with the x86_64 architecture. If you’re using a 32-bit system, you’ll need to replace all references to x86_64 with i686.

Read more…

It’s not often that we run into unusual conflicts rooted in hard to find places, but those few times that they are encountered–no matter how rare–can be infuriating. I was recently bitten by this one, demonstrated by a particularly bizarre error when attempting to load an image (any type) into the php-imagick ImageMagick extension:

$ php -a 
Interactive shell
 
php > $c = new Imagick('kmix.png');
PHP Warning:  Uncaught exception 'ImagickException' with message 'no decode delegate for this image format `kmix.png' @ error/constitute.c/ReadImage/552' in php shell code:1
Stack trace:
#0 php shell code(1): Imagick->__construct('kmix.png')
#1 {main}
  thrown in php shell code on line 1

Yet, paradoxically, convert -list delegate, convert -list configure, and convert -list format all hinted that PNG and JPEG support were both installed. Worse, searching for “php imagick no decode delegate” resulted in few useful hits, and of those hits most were indicative of configuration problems, custom builds missing the appropriate libraries, and other dilemmas punctuated by the usual PEBKAC retort (though such accusations are sometimes accurate).

Nothing worked, of course. Swapping out to older versions of ImageMagick failed, and I had no intention of trying an earlier version of PHP or php-imagick. At this point, I had no expectations to find the solution online, and the only thing I surmised might work would be to attempt duplicating the problem on another machine. So, I set about installing ImageMagick (v6.8.5-10) and php-imagick (v3.0.1 plus PHP 5.4 patches and v3.1.0RC2) with the current-latest version of PHP available in the Arch repositories (v5.4.16) and quickly discovered that it worked quite well. This was a positive sign: Nothing about the latest builds indicated there was a problem, so it was plausible that the problem lurked elsewhere.

And it did.

I soon noted that the virtual machine I booted was lacking php-gmagick and GraphicsMagick both, though ImageMagick and GraphicsMagick worked quite well on my machine together. Thus, I speculated that perhaps php-gmagick was conflicting somehow with php-imagick. I commented out the gmagick extension, re-ran some unit tests, and presto! php-imagick was now working perfectly fine. Uncommenting the gmagick extension resulted in a return of the problem, and so I was (and am) convinced that the problem is a conflict between both extensions. Possibly, this is due to the definition of symbols or other such nonsense in gmagick that imagick requires (and is very upset when it discovers they already exist).

Is it possible to get them to play nicely together? Maybe.

On my machine, PHP extensions and their options are loaded from /etc/php/conf.d. These files are, apparently, loaded alphabetically in order, and since “G” comes before “I,” you might imagine (correctly) then that gmagick gets loaded before imagick which is where the magic stops (my apologies; that pun was awful). By renaming them to something creative, such as “10-imagick.ini” and “20-gmagick.ini”, it is then possible to force imagick to load before gmagick, and they’ll both work cooperatively near as I can tell. Perhaps gmagick defines–but doesn’t use–some interpreter symbols that imagick actually needs, but if they’re already defined gmagick doesn’t care? I don’t know. I can only speculate, because I know next to nothing about PHP internals. (Perhaps now might be a good time to start.)

Although renaming or otherwise moving the extension configs into an order that would suggest imagick.so gets loaded first seems like an adequate solution, the problem is that these two extensions most assuredly do conflict. Between PHP versions, you can’t be guaranteed that the loader mechanism for extensions will remain static and thus the possibility exists that renaming the *.ini files is only an ephemeral solution. Moreover, it’s also possibly an accident that it works on my machine; perhaps the inode order is such that imagick.so now gets loaded before gmagick.so simply due to luck. Or maybe I’ve now reduced it to a race condition? The point is that you shouldn’t rely on this in production. Instead, you should select between GraphicsMagick or ImageMagick, picking the one that best fits your use case. If you’re using a wrapper library to select the best backend available, force it to load a specific one in production mode. Obviously such concerns aren’t much of a bother on a development box, but I’d nevertheless recommend a healthy dose of caution.

If you’ve encountered this issue yourself but never found a reasonable solution, let me know. If this article helps in any measurable way, I’d be happy to hear from you. It’s been a frustrating few hours for me, and I hope that this will at least alleviate you of wasting further time on something that is poorly documented but easy to fix.

So, KDE 4.10 just came out last week, the Arch repos sometime thereafter, and I updated today. I really love new versions of KDE–honestly–but this time something really bizarre happened. I updated, rebooted to the new kernel (I was running 3.7.4, the update was to 3.7.6), fired up KDE, and started loading my usual applications.

Then nothing worked quite right. I’d open one window, and the others I had open would disappear. Opening the launcher would also exhibit this problem: Open terminal windows would disappear the instant the launcher was clicked or any other application (including Dolphin) was opened. Infuriating. It seemed that I couldn’t have all my usual windows open at once–everything would disappear!

Fortunately, after some trial and error, I found the culprit. This latest version of KDE has made significant changes to ~/.kde4/share/config/kconf_updaterc. So, there’s two solutions:

Easy Mode

Log out, switch to a virtual terminal (ctrl+alt+F1), log in from there, and delete the file ~/.kde4/share/config/kconf_updaterc. Then switch back to X (usually ctrl+alt+F7 or F8) and log back in to KDE so it regenerates the file. Enjoy!

Hard Mode

Follow the previous steps but retain a backup of your .kde4 directory. Then run a diff on the two versions of kconf_updaterc and figure out what you want to keep, what line caused the problem, and retain as many of your previous settings as possible.

Otherwise, I hope this saves you some difficulty if you run into the same thing I have!

February 13th edit:

It seems that this post by “George” on the Arch Linux forums indicates a potentially easier solution, which is to change the window transparency settings. I changed the title transparency settings, but I’m not entirely sure if this will resolve the disappearing windows.