A close friend of mine recently decided it was time to update his Arch Linux installation. It had been over a year since the core OS was completely updated, and I did everything I could to discourage him from trying the usual pacman -Syu. The problem, for all intents and purposes, is that I knew it wouldn’t work. Much of the file system structure has changed dramatically over the course of the past year (the /lib move and the more recent merging of /bin and /sbin into their /usr counterparts) and a straightforward update was now difficult (but not impossible–more on this in a minute). Specifically, the glibc and filesystem packages have gone through several iterations even last July and would now be permanently blocking each other thanks to these two updates with no immediately obvious path forward.

I have observed some comments dismissing the update process from a pre-systemd installation as virtually impossible. While I suspect this is because the individual(s) asking for help are seen by others to be insufficiently experienced to do such an update, I’m not a particularly huge fan of the term “impossible” as it pertains to difficult feats–even for the inexperienced. After all, few things are truly impossible; it’s simply a matter of how much time and energy one is willing to invest in working toward a specific goal. Besides, if the recommended solution is to reinstall, why not figure out an optimal way to upgrade an “impossible” system? If it’s going to break either way, we may as well have some fun with it. (You have backups, right?) Even if you don’t have the necessary experience or knowledge relating to the underlying system, there’s no time like the present to begin the learning process.

So, Thursday evening, I set out looking for a solution. Fortuitously, I had an old Arch Linux virtual machine installed on my desktop (which, conveniently, is also Arch Linux) that dated back to approximately the same kernel revision as my friend’s (v3.3.6 for mine; v3.3.4 for his) and roughly analogous software. Like his system, my VM was also a pre-systemd installation, was still running glibc 2.15, and a very early version of filesystem-2012. In many regards, my VM was more or less in a state identical to that of the machine we needed to fix. However, the problem was compounded by an additional requirement. Console access was somewhat tedious for my friend to obtain because of the systems he had to go through, and he was exceptionally busy the afternoon we planned the updates. So, I had to to keep network access (particularly SSH) up and running as best as I could.

Disclaimer

This narrative guide is intended as general reference for anyone who might be in a similar situation: Pre-systemd, pre-filesystem moves, and operating with a requirement that the machine be (mostly) network-accessible throughout the duration of the update. Be aware that nothing in this guide should be considered authoritative. I am a user of Arch Linux. There’s still much about system internals (and Linux in general) that I don’t know or don’t fully understand. Consequently, I’m always learning, and there may be better alternatives to specific problems encountered during this update process. However, I do know enough about Arch to recommend that you never use –force unless you’re specifically instructed to do so. At no point during this update process should you use it. Doing so will assuredly destroy your system, and you’ll be forced to reinstall.

Secondly, this update process is not supported by the Arch Linux developers. If you have failed to maintain your system by keeping it up-to-date (and updating your Arch Linux installation is one of the core tenants of being an Arch user), you’re unlikely to receive much help since you’ve already established that your OS has been inadequately maintained. Furthermore, this update process relies on tricks using partial updates which are also unsupported. The developers and users of Arch who frequently the forums are nice folks who have lives outside of Arch, and due to the tremendous task presented to the community of supporting a relatively large user base, it’s impractical for them to spend a significant chunk of their time helping you with issues that–to avoid mincing words–are the fault of no one but yourself.

Thus, I hope this guide will be useful to those of you who may have a neglected Arch Linux box under your care. Be aware that I have targeted this guide to a system that was last updated somewhere in mid-2012. Although it should work for earlier systems, I’d highly recommend reading through the Arch news archives if you’re updating such a beast. You can usually determine where you need to begin by examining the version of your filesystem package, e.g. pacman -Qs filesystem. This guide should be general enough such that even if it doesn’t help you determine the exact update path for your system, you’ll be able to figure out where to get started.

Also, be sure to read through the guide in its entirety first. Check your installed packages, and make sure you’re not downgrading them. filesystem and glibc should never be downgraded at any point during this update otherwise you may exacerbate system breakage. This guide illustrates the update process with the x86_64 architecture. If you’re using a 32-bit system, you’ll need to replace all references to x86_64 with i686.

The Update: Preparation

The good news is that updating a relatively ancient Arch Linux install is not especially difficult, if you’re willing to spend a half hour (or longer), take your time, cautiously follow each update, and pay careful attention to what the system is telling you after each step. More generally, use of tools like this one can be useful (as of August 17th, 2013 ARM has been taken offline; you’ll need to obtain the packages listed below from another source) in gracefully moving the system through several stages of updates. Make no mistake about it: This update process most assuredly flows through several stages, and there are specific periods of time where a power failure or accidental reboot might render your system unbootable. Before you begin, you should have backups of your data, a bootable ISO or USB stick for recovering an unbootable system, and copious amounts of free time just in case something goes wrong. For VPSes or other remote systems, you should also have ready access to the system’s console.

If you’re updating from approximately May 2012, I’d recommend finding the following packages. We’ll be using them later. You may be able to use the Arch Linux SVN repository to snag the appropriate versions.

dbus-1.6.2-1-x86_64.pkg.tar.xz (extra, circa July 2012)
dbus-core-1.6.2-2-x86_64.pkg.tar.xz (core, circa July 2012)
dbus-glib-0.100-1-x86_64.pkg.tar.xz (extra, circa July 2012)
device-mapper-2.02.96-3-x86_64.pkg.tar.xz (core, circa July 2012)
filesystem-2012.7-1-any.pkg.tar.xz (core, circa August 2012)
filesystem-2013.01-3-x86_64.pkg.tar.xz (core, circa February 2013)
glibc-2.16.0-1-x86_64.pkg.tar.xz (core, circa July 2012)
glibc-2.16.0-2-x86_64.pkg.tar.xz (core, circa July 2012)
glibc-2.17-3-x86_64.pkg.tar.xz (core, circa February 2013)
hwids-20120611-1-any.pkg.tar.xz (core, circa July 2012)
initscripts-2012.06.3-2-any.pkg.tar.xz (core, circa July 2012)
initscripts-systemd-20120412-1-any.pkg.tar.xz (community, circa July 2012)
libsystemd-185-4-x86_64.pkg.tar.xz (core, circa July 2012)
linux-3.4.4-2-x86_64.pkg.tar.xz (core, circa July 2012)
linux-api-headers-3.4.4-1-x86_64.pkg.tar.xz (core, circa July 2012)
lvm2-2.02.96-3-x86_64.pkg.tar.xz (core, circa July 2012)
mkinitcpio-0.9.2-2-any.pkg.tar.xz (core, circa July 2012)
net-tools-1.60.20110819cvs-3-x86_64.pkg.tar.xz (core, circa August 2012)
netcfg-2.8.5-1-any.pkg.tar.xz (core, circa July 2012)
openssh-6.0p1-3-x86_64.pkg.tar.xz (core, circa August 2012)
pciutils-3.1.10-1-x86_64.pkg.tar.xz (core, circa August 2012)
systemd-185-4-x86_64.pkg.tar.xz (core, circa July 2012)
systemd-sysvcompat-185-4-x86_64.pkg.tar.xz (core, circa July 2012)
systemd-tools-185-4-x86_64.pkg.tar.xz (core, circa July 2012)
usbutils-005-2-x86_64.pkg.tar.xz (core, circa July 2012)

Older systems or systems that have been more recently updated may not need all of these packages or they may need additional, older versions. 32-bit systems will require that you replace x86_64 with i686. Be sure to read through the remainder of the guide before you begin so you know what to expect.

For the rest of the guide, # at the start of the sample output indicates commands used in the root shell; $ indicates a user shell, with sudo as an optional alternative that I utilize hereafter for everything. If you prefer, you can do the update as root but be mindful of typos.

We’ll eventually be updating Arch to its latest state when we’re finished, so you should run the following commands to update your repository database and download all update files ahead of time.

$ sudo pacman -Syy
$ sudo pacman -Swu

We won’t be using the updates for a while, so keep them safe.

Next, if you’re running an AUR helper such as Yaourt, you should temporarily remove it to avoid noisome dependency conflicts during the update process. You don’t want an untimely distraction mucking up all your hard work:

$ sudo pacman -R pacman-color package-query yaourt

Obviously, you’ll need to substitute whatever AUR helper you have installed. I’ve illustrated this with Yaourt as it’s my preferred helper. Some users despise AUR helpers for various reasons–some realistic, some emotional–so removing them at this point helps avoid much of that debate if you get yourself into a bind. Although we’ll be using pacman for everything, we cannot update it until the end; pacman depends on the latest version of bash which should not be updated until after we’ve moved everything out of the way. It’s plausible to avoid some of this and update it from the start, but you’re on your own if you do so and will probably need to create a symlink to /usr/bin/bash at a minimum.

To reiterate:

• DON’T update pacman until the update process is finished.
• DON’T update bash until you’re told to do so.
• SOME warnings can be ignored, such as during an intermediate update between glibc 2.16 and 2.17, but…
• READ CAREFULLY all warnings the systems spits out at you. They happen for a reason and it’s up to you to understand what that might be. If you must fix something before proceeding (e.g. with mkinitcpio), fix it first. Don’t blindly continue the update process and hope for the best!

Important! Edit your /etc/rc.conf if you’re still using a pre-systemd installation and remove (or prefix with a “!”) all unnecessary services except for network (or whatever your network configuration requires, such as net-profiles) and sshd.

If you already have systemd installed, you can skip ahead.

If archlinux-keyring isn’t installed (pacman -Qs archlinux-keyring) or you’re unsure what state your local copy of the keyring is, you should install the haveged package, particularly if you’re connecting from remote. This will ensure gpg has sufficient entropy to generate your local signing key:

$ sudo pacman -S haveged

Next, verify that you’re running at least glibc 2.16.0-1:

$ pacman -Qs glibc

And if you’re not:

$ sudo pacman -U pacman -U glibc-2.16.0-1-x86_64.pkg.tar.xz linux-api-headers-3.4.4-1-x86_64.pkg.tar.xz

(glibc 2.16 relies on at least linux-api-headers >= 3.4.)

Then, upgrade the following packages. You may need to tweak this list depending on your specific versions as the idea is to simply bump the system state up to a point where we can install and enable systemd r185:

$ sudo pacman -U dbus-1.6.2-1-x86_64.pkg.tar.xz dbus-core-1.6.2-2-x86_64.pkg.tar.xz dbus-glib-0.100-1-x86_64.pkg.tar.xz device-mapper-2.02.96-3-x86_64.pkg.tar.xz hwids-20120611-1-any.pkg.tar.xz initscripts-2012.06.3-2-any.pkg.tar.xz initscripts-systemd-20120412-1-any.pkg.tar.xz libsystemd-185-4-x86_64.pkg.tar.xz linux-3.4.4-2-x86_64.pkg.tar.xz linux-api-headers-3.4.4-1-x86_64.pkg.tar.xz mkinitcpio-0.9.2-2-any.pkg.tar.xz net-tools-1.60.20110819cvs-3-x86_64.pkg.tar.xz netcfg-2.8.5-1-any.pkg.tar.xz openssh-6.0p1-3-x86_64.pkg.tar.xz pciutils-3.1.10-1-x86_64.pkg.tar.xz systemd-185-4-x86_64.pkg.tar.xz systemd-tools-185-4-x86_64.pkg.tar.xz usbutils-005-2-x86_64.pkg.tar.xz

If you’re using LVM, you may have noticed “/sbin/lvm2” errors, and you’ll need to upgrade it from the packages we downloaded earlier. If you’ve already upgraded LVM, you’ll need to re-install it.

$ sudo pacman -U lvm2-2.02.96-3-x86_64.pkg.tar.xz

And finally, regenerate the initramfs for your current (now 3.4.4 kernel):

$ sudo mkinitcpio -p linux

Pay particular attention to any errors that appear at this phase and fix them. If mkinitcpio doesn’t generate a bootable image, you won’t be able to continue. It is imperative that you have a bootable image at this point.

On some systems, the systemd target sound.target fails and hangs the system on boot, probably due libraries I’ve missed. Rather than fix it at this point by digging around for the appropriate archives, it’s easier if we just circumvent the problem entirely (and it’ll be reinstalled–and working–later) by moving it into our working directory:

$ sudo mv /usr/lib/systemd/system/sound.target .

Finally, modify /boot/grub/menu.lst and add init=/usr/lib/systemd/systemd to the end of your kernel= line. This will override what the kernel uses for init (normally /sbin/init on pre-2013-07 platforms) and boot the system using systemd.

Next, enable the sshd service. It should be started up by the initscripts-systemd bridge, but we want to be absolutely certain sshd will start on boot:

$ systemctl enable sshd.service

(Note: The *.service suffix is no longer needed on newer versions of systemd; however, r185 required that it be provided for enabling or disabling services. We’ll skip that nonsense later.)

Reboot the system. Ignore any syslog- or tmpfiles-related errors that the console spits out. We’re going to fix them soon enough.

Systemd is here, but what about that filesystem?

You should have a bootable systemd-initialized system. To verify, check the output:

$ systemctl --type service

It should indicate that several services have been loaded and are running. If you’re updating from a pre-systemd installation, you may notice that the tmpfiles service is in a “failed” state. That’s okay, because our system is still in flux. We’ll be fixing that shortly.

Now, we need to bump the filesystem up to the upgrade state it was when the symlinks /var/run and /var/lock were removed, finalizing the merge into /run:

$ ls -alh /var # verify that /var/run and /var/lock are indeed symlinks
$ sudo rm /var/run /var/lock # remove them
$ sudo pacman -U filesystem-2012.7-1-any.pkg.tar.xz # update

You’ll also need to upgrade Linux to move the /lib/modules directory out of the way and into /usr/lib. This should also take care of any firmware you have installed. Specialized firmware might require a separate update. If you’ve built anything from the AUR, you may need to upgrade it separately.

$ sudo pacman -S linux

If you have the development tools packages installed, you’ll need to either upgrade or remove a couple of their constituents that will be blocking the update (I suggest removing them; we’ll re-install these packages later):

$ sudo pacman -R libtool gcc

Other packages that may interfere are acl, attr, bzip2, e2fsprogs, fuse, mdadm, sysfsutils, and xfsprogs. You may encounter others that block the update process. Use

$ pacman -Qo /lib/*

To determine which offending packages need removing or updating. You may need to remove /lib/udev manually (but make sure it’s empty first!).

If glibc is the only package that owns files in /lib, you should be clear to update it again from what we downloaded earlier:

$ sudo pacman -U glibc-2.16.0-2-x86_64.pkg.tar.xz

Don’t panic if you receive the error “/lib exists in file system.” Simply repeat pacman -Qo /lib/* until no other packages beyond glibc owns files in /lib. If a file or empty directory is listed that does not have an owner, you can delete it manually.

When you’re finished, /lib should be symlinked to /usr/lib.

A systemd system with 2013-flavored filesystem layout and grub2

At this point, we can completely nuke initscripts. It’s unsupported anyway:

$ sudo pacman -R initscripts sysvinit initscripts-systemd
$ sudo pacman -S systemd-sysvcompat

And we’ll follow this update that requires filesystem and glibc to be updated at the same time.

$ sudo pacman -U filesystem-2013.01-3-x86_64.pkg.tar.xz glibc-2.17-3-x86_64.pkg.tar.xz

This will probably pull in the latest linux-api-headers package and may generate some pacnew warnings. Ignore them.

It’s also time to update your bootloader. Failure to do this may result in an unbootable system and grub-legacy (v0.9x) is no longer supported (BIOS instructions are provided; if you’re updating from an older Arch, you probably don’t have UEFI support anyway):

$ sudo pacman -S grub
$ sudo mv /boot/grub /boot/grub-legacy
$ sudo grub-install --recheck /dev/sd[X] # where [X] is your boot drive where grub was installed
$ sudo grub-mkconfig -o /boot/grub/grub.cfg

If grub-install fails with “path ‘/boot/grub’ is not readable by GRUB on boot,” grub-probe is probably failing and you may need to ln -s /usr/lib/libudev.so.1 /usr/lib/libudev.so.0. Be sure to remove the symlink after installing grub.

Network preparations

At this point, you have a choice. My personal preference is to use the newest network configuration package netctl. If you use NetworkManager or similar, that’s fine; skip this section and do some reading. Otherwise, if you’re migrating from a static configuration in your old /etc/rc.conf or from a previous netcfg configuration, follow these instructions.

First, remove netcfg and install netctl:

$ sudo pacman -R netcfg
$ sudo pacman -S netctl

Then, create a new profile for your network. I usually name it after the interface I’m configuring, e.g. /etc/netctl/eth0. You can name it anything you prefer. Some sample configurations below:

# Static IPv4
Connection="ethernet"
Description="eth0 - static assignment"
Interface="eth0"
IP="static"
Address="192.168.82.1/24"
Broadcast="192.168.82.255"
Gateway="192.168.82.1"
 
# IPv4 configured via DHCP with stateless IPv6 and static IPv6 assignment:
Connection="ethernet"
Description="eth0 - DHCP IPv4 and static IPv6"
Interface="eth0"
IP="dhcp"
IP6="stateless"
Address6=('2001:470:a:b:c:d:1:23/64')

To enable the profile you just created (the name of the profile is the file you saved under /etc/netctl) type:

$ netctl enable eth0

Updating to the latest

It’s time to update Arch to the latest binary path update (/bin and /sbin). If there are any other updates that require moving these paths, and there surely will be, then you may need to download the appropriate packages from an archival mirror. Following the instructions from the new page, we enter

$ sudo pacman -Su --ignore filesystem,bash

and skip the pacman updates both times. Once finished, we may now update pacman (you might have to leave the filesystem package out of this update cycle):

$ sudo pacman -S archlinux-keyring bash pacman

If you receive execv errors, you will need to issue the above command once more or at least re-install any package that triggered it.

If the pacman update fails with libarchive.so.12 not found, do this:

$ sudo ln -s /usr/lib/libarchive.so.13.1.2 /usr/lib/libarchive.so.12
$ sudo pacman -S archlinux-keyring bash pacman
$ sudo rm /usr/lib/libarchive.so.12

Should you be updating an Arch install from a much earlier revision that complains about missing keys and instructs you to run pacman-key –init, you may need to do the following:

$ sudo haveged -w 1024
$ sudo pacman-key --init
$ sudo pacman-key --populate archlinux

bash errors can be temporarily resolved by sym-linking /usr/bin/bash to /bin/bash and path errors (such as when running ls or ln) can be resolved by temporarily updating your $PATH:

$ export PATH=/usr/bin:$PATH

Just be sure to remove the symlink before finalizing the updates.

At this point, you’ll need to complete the update which should pull in only the filesystem package, and you’ll need to fix any other packages that generated errors during the “binary move update” (when you issued pacman -Su –ignore filesystem,bash):

$ sudo pacman -Su
$ sudo pacman -S fakeroot # fakeroot has generated errors on two systems for me during the update; fix it

Finally, you’ll need to run mkinitcpio and fix any warnings it spits out. In particular, you might need to change the HOOKS line in /etc/mkinitcpio.conf to something like:

HOOKS="base udev autodetect block modconf filesystems keyboard fsck"

And run:

$ sudo mkinitcpio -p linux

If you’re running LVM, you’ll need to add the lvm2 hook after modconf and before filesystems. You may also wish to re-run grub-mkconfig -o /boot/grub/grub.cfg to verify that grub picked up any changes from intermediate updates.

Cleaning Up

Before you reboot, you’ll want to ensure that sshd is running and set to start on boot:

$ systemctl list-unit-files | grep sshd
sshd.service                            enabled 
[email protected]                           static  
sshdgenkeys.service                     enabled 
sshd.socket                             disabled

If it says “enabled” next to sshd.service, you should be ready for a reboot. Otherwise, you’ll need to run systemctl enable sshd to enable it.

You’ll also need to verify that systemd-sysvcompat was in fact installed

$ sudo pacman -S systemd-sysvcompat

Otherwise you might discover a surprise when you enter “reboot.” systemd provides reboot facilities via systemctl reboot, so if you find reboot doesn’t exist, this give you an opportunity to do so without it.

Setting your hostname might also be desirable prior to a reboot

$ sudo hostnamectl set-hostname mycomputername

LVM users will also need to enable lvm-monitoring via systemctl enable lvm-monitoring before rebooting. Note that lvmetad won’t correctly start until after you’ve rebooted. It doesn’t matter anyway; your system won’t be able to make use of it until after a reboot.

At this point, if you’re satisified everything has been installed, reboot and finish the process. Don’t forget to install packages removed earlier in the process:

$ sudo pacman -S libtool gcc
$ sudo pacman -S yaourt # or whichever AUR helper you use, if any

You may also begin cleaning up your system by removing unused *.pacsaves and merging in any *.pacnews that have been installed with updated packages. My personal preference is to use yaourt -C to manage this automatically, running vimdiff where necessary, but there are plenty of other methods and tools you may use. Be sure to either outright replace old versions (/etc/services and /etc/protocols being two such examples) or merge in changes to updated configurations (/etc/php/php.ini comes to mind). Verify that the symbolic link /etc/localtime points to the correct timezone.

If you’re missing pacman-color, its functionality has been merged into pacman. See /etc/pacman.conf.

Finished!

First, I should apologize for the length of this post; the guide was written for a general audience with only a passing familiarity of Arch internals. My rationale is that those who neglect Arch updates for a year or longer are most likely the sort who run the OS based on recommendations from friends and are thus unfamiliar with the perpetual attention a rolling release-based operating system requires. Users who are otherwise familiar with Arch but simply want an upgrade path and happen to be hitting a wall ought to be able to extract what they want by simply skipping the superfluous rubbish.

If you’re reading through this guide to get a feel for the update process, I can’t stress enough the importance of collecting backups before you start. Although some upgrades such as Arch dropping support for MySQL in favor of MariaDB are non-forced changes, offering you sufficient time to create backups you may have otherwise missed, there are others that may require a more direct approach. Further, there’s no guarantee that versions of MySQL prior to the switch will continue to work as Arch evolves. In any case, you should meticulously backup and copy any data you consider important. This update isn’t guaranteed to work, and if it fails you may be left with an unbootable system. You should also have some means of obtaining console access in the event things don’t work out, but I’ve taken great care during this process to ensure that 1) networking will remain active and 2) SSH should remain enabled. I emphasize should; if you happen to be unlucky and support for your network card is dropped in newer kernels, then there isn’t much either of us can do.

For those of you who might happen upon this guide and use it successfully (or otherwise), feel free to drop me a note. I can’t emphasize enough that I’m no expert! I’m simply a user who happens to be something of an Arch zealot, and I hope that my experience updating a somewhat archaic install will be useful to others.

I love Arch precisely because of its simplicity and its lack of heavily pre-configured services and modifications. Although it has its warts (everything does–it’s a matter of picking the idiosyncrasies you can handle best), Arch is the best fit for my personality type, and I’ve been using it every day as both a workstation and home server (and HTPC!) for over a year. Here’s to many more years with Arch, and I hope you’ll be happy with your freshly updated install.

Just remember to keep it updated next time!