Linux on the Lenovo Y740 Legion (2019)

It’s easy.

No, really, it is. There’s a couple gotchas and some minor inconveniences (probably self-induced in my case), but provided you didn’t do anything particularly unusual with the system configuration at purchase, it should work.

First, I want to preface this with a brief overview of my configuration. I selected one with an NVMe SSD for its primary drive and a mechanical SATA drive for the secondary. I did not select one with Optane and with good reason, but I’ll get to that in a moment. All things considered, it’s a fairly banal, boring configuration with the exception of some of the features new to the 2019 lineup (notably the Corsair RGB keyboard and configurable fan LEDs). Interestingly, the behavior of this system’s EFI isn’t especially novel or noteworthy. It just works.

Caveat emptor: The configuration I discuss in this post may not work for you. I made decisions specific to how I wanted to use this system and performed some tasks manually to avoid overwriting defaults that shipped with the laptop. I’m also using Arch and other distros may present challenges specific to the software they ship or the utilities they package. Always back up any important files and do not perform these steps if you’re unwilling to lose data. I didn’t, but I’d only just gotten the laptop a couple days prior. Had I been using it for some months, I might have performed these steps differently. Some may also question why I didn’t perform a clean install of Windows; I considered it, but I didn’t feel the need to do so.

Now to the meat of the process: Before I started, I made certain to have available two USB sticks (both bootable): One with the Arch ISO image, and the other with a bootable Windows 10 installation via the Media Creation Toolkit. The latter was in my pile of tools in the event things went south and I needed to reinstall completely.

When booting to Linux on the Y740, you’ll note that the NVMe drive is not visible from Linux. This appears to be due to the storage configuration in BIOS. By default, it’s set to Intel’s Rapid Storage Technology; switching it to AHCI resolves the issue. One of the curious things to note when changing this configuration is the ominous warning that applying a different setting will delete all data on attached disks. I found this isn’t the case, but this is also why I’d recommend against selecting a model with Optane installed. I’m not certain on this point–you should certainly do your own research–but I believe with Optane installed, BIOS transparently configures it as a drive cache. Changing this on such a system may cause data corruption which is possibly what the warning implies. (The help text for this setting also mentions it’s specifically for Optane-equipped systems, hence my speculation.)

Once the drives were configured to use AHCI, the NVMe disk was accessible via Arch, and I proceeded to image it to the mechanical storage drive (using dd, of course). This image was then moved to permanent storage on my other systems in the event I did something incredibly stupid.

I look at it this way: If I didn’t have it available, I can almost guarantee I probably would have done something stupid. Always keep a backup!

(Speaking of stupid: This section is somewhat intentionally out of order but necessary for the story to flow; read below for potential video issues, because you will encounter them.)

Now, at this point, I had two choices. My initial thought was to partition the drives accordingly, and reinstall Windows completely. This would have been the ideal situation, but I wanted to save some time (and use the system in its stock state, minus some annoying cruft like McAfee). So, the next choice was to shrink the volumes on the SSD and the storage drive to roughly half. I gave Windows somewhat more storage, because it’s Windows, and because I plan to use this as a casual gaming system in addition to doing work. Doing so is easy enough: To resize the NTFS volumes, go to Computer Management -> Storage -> Disk Management. Then, you’re only a reboot away from getting started.

This is the easy part.

With the partitions resized, and a USB stick in hand, I feverishly pressed F12 to bring up the boot device selection screen–and promptly noticed that the USB stick wasn’t anywhere to be found. After unplugging it and putting it back in, BIOS appeared satisfied enough to present me with it, and off I went. I didn’t notice this with another stick I was using, and I had some difficulty replicating the problem. I’m not sure if this is a fault with the USB flash drive I had the Arch ISO written to or whether it’s just an idiosyncrasy of this BIOS. Either way, things appeared to work great…

Until the stick booted, that is.

Apparently the nouveau drivers that ship with Arch don’t work with the GeForce 2060 particularly well. I was greeted with what looked like vague video corruption and a line roughly one pixel high that appeared to be the bottom strip of whatever text was printed to the screen. Bummer. Rebooting, pressing F2, and getting into BIOS to examine whatever other configurations might help seemed to be my only salvation. Without much clue what else to pick, I noticed the graphics configuration had two states: Discrete graphics (that is the NVIDIA card) and “switchable graphics.” I knew from helping my girlfriend with her now-crippled Acer that the “switchable graphics” setting likely allowed the system to select between the integrated Intel UHD graphics on the CPU die and the discrete (NVIDIA) card; my theory at this point was to presume that doing so would allow Linux to boot up using the Intel chipset, hopefully avoiding the video corruption immediately after the kernel loaded up.

It worked, and from here we could progress.

The Arch installation was fairly pedestrian from this point: Setup the free space with partitions (I went with /boot and root on the SSD, although I should have left them merge–more on this in a moment–and the mechanical drive got a /storage mount point and swap), format as appropriate, install, configure, and… done!

Just kidding. That last part should be cry while you figure out how you want to setup your boot loader. You see, I’ve rarely used EFI systems outside virtualization. All of my systems are pre-2014-ish, and the one or two I’ve had the (dis)pleasure of poking at were all Windows machines. So, what are we to do?

(Aside: Wiping my girlfriend’s system and installing Linux would probably end with my face on a milk carton.)

First thing’s first: We need to figure out the time. No, I don’t mean how long the whole process had taken up to this point! I quite literally mean the time. One of the pain points dual booting Windows and Linux is how to handle the system clock. Once upon a time (sorry), Linux would begrudgingly chug along with the system clock configured to local time. This is a terrible idea, and I still have no idea why Windows insists, but fortunately, you can change Windows’ behavior easily enough. However, doing so requires the foresight to change this setting before getting started–something I didn’t have because I’m stupid and it only occurred to me when I got to the point of configuring the clock. Perhaps you’ll be luckier. You’re reading this after all!

Now, where were we? Oh, right! The bootloader. This is one of the deficiencies with using Linux on newer systems. Generally speaking, EFI, UEFI, or whatever your motherboard manufacturer has decided to call it, requires special attention that we Linux users haven’t had to give to boot loading since the 2000s. No, I wouldn’t use grub either–it does apparently have EFI support, but I have painful memories getting it working under VirtualBox with EFI enabled. Perhaps this is a VirtualBox-specific issue, but I’m inclined to believe we’re better off using tools designed for a specific purpose. In this case, rEFInd.

I won’t pretend to be a subject matter expert. I’ve never used rEFInd before. The Arch Linux wiki does have fantastic resources that can help you get started, but the thing I noticed with my particular configuration is that special attention had to be placed on configuring the OS declarations in esp/EFI/refind/refind.conf. If you’re following along at home, you should at least read this section on UEFI and the entry on rEFInd.

For my system, I did not follow the automatic installation with refind-install, because I didn’t want to overwrite the default EFI boot entry. Thus, I followed the manual installation process by copying rEFInd into the EFI system partition. Note that this alone is not enough to get rEFInd to work with the Y740’s BIOS. I’m not certain whether this is due to a step I’d skipped earlier in the installation or whether it’s an artifact of the manual install, but I found the only way the Lenovo BIOS would see rEFInd even if it’s in the EFI system partition is to configure the boot order via efibootmgr –bootorder 0004,0009,2001,2002,2003. I assume this should just work, but nothing I did would force the BIOS to recognize rEFInd (with the ID 0009). Changing the boot order did work, however, and with Windows (0004) as the primary–temporarily at least–and rEFInd (0009 on my system) as the secondary, I learned that BIOS had been forcibly configured to see rEFInd allowing me to change the UEFI boot order accordingly.

I also discovered that I could not get rEFInd to recognize the /boot/refind_linux.conf configuration. I’ve not investigated why, but I suspect it’s either due to my choice of partitioning (remember, I’m using a separate /boot and root!) or misunderstanding of rEFInd. However, configuring Arch via esp/EFI/refind/refind.conf has worked just fine. I should also note that by esp in the former example and above paragraphs, I mean the EFI System Partition. I mounted this (/dev/nvme0n1p1 on my system) to /boot/efi for ease of access. I’d suggest doing the same if you plan on going down this route.

Finalizing the Linux installation was relatively painless once the bootloader struggle was resolved, and I had NFS plus KDE working more or less out of the box. The system does quite well under Linux thusfar, although I’ve yet to configure wireless networking. I suspect the wireless NIC will work fine: I can see the card in ip link and the ridiculously named “Killer 1550i” cards are, as far as I know, simply rebranded Intel 9560NGW chips.

There is some unfinished business, and I’ve encountered at least one or two teething problems. First, this configuration doesn’t address secure boot. I was primarily focused on getting Linux working rather than working with secure boot. I’m hopeful this won’t be especially difficult, and from what I’ve read it appears the process is relatively straightforward. I’m planning on going the machine owner’s route with regards to kernel signing, with further plans to automate the process in a pacman hook. I’ve also noticed the wired network didn’t come up automatically in spite of being enabled via systemd (I prefer using systemd network configurations via /etc/systemd/network over NetworkManager), and there’s a large amount of i2c_hid spam in the journal. I suspect this may have something to do with some of the peripherals in the system (touchpad? wireless mouse?).

I’ll eventually write a part two once I get some of the remaining issues resolved, along with documenting whatever else I may encounter. If you’re using Linux and just bought one of these systems, please don’t feel overwhelmed with the installation process. Just be cautious, think things through, and have a plan of attack. Don’t forget to back things up, too. Oh, and as much as I don’t like Windows 10’s Microsoft account option, I would recommend logging in with one at least once because it ties the software license for that machine to your account. If you decide to reinstall Windows, this is a good thing, in my opinion!

No comments.
***

My Journey with ZFS on Linux

Toward the latter half of 2016, I decided to convert my home file server’s file system to ZFS as an experiment on real hardware in my own test environment. I knew before embarking on this journey that ZFS on Linux (hereafter referenced as ZoL) is stable but not production ready. The experiment is nearing its end, and I’ll be switching to an mdadm-based array with ext4 for reasons I’ll be exploring in this post. If your storage needs require the redundancy and self-healing properties of ZFS, consider other operating systems where it’s better supported and stable, like FreeBSD.

This post illustrates my thinking as of early 2017 and why I’m not going to continue with ZoL (for now).

Conclusions

I don’t wish to bore you with excess detail, so I figured I’d present my conclusions up front. You can then decide if the rest of this post is worth reading. Let’s begin exploring.

First, using ZoL in a home file server environment presents us with the following pros and cons:

Pros

  • Transparent compression can save a surprising amount of space with with LZ4 (useful!)
  • Lower administrative overhead (a wash; not always true in practice)
  • Data integrity guarantee (always make backups)
  • Self-healing (always have backups)
  • It’s not Btrfs (you’ll probably use your backups)

Cons

  • May require more extensive deployment planning
  • Some applications may require dataset tweaking
  • Administrative overhead not always as advertised (see above), but this is mostly the fault of the current state of ZoL
  • Poor performance for some workloads (especially databases; versus ext4)
  • Lack of recovery tool support limits possible restoration options when things go wrong (backups!)

Noteworthy bullet points when using ZoL:

  • Stable but not production ready (an important distinction!)
  • Upgrades will be painful!
  • /boot on ZFS is possible but…
  • DKMS ZFS modules are a horrible idea; don’t do this–don’t ever do this
  • Always create bootable media with ZFS tools installed in case something goes wrong (it will–don’t forget backups!)

The Meat and Potatoes of ZoL

I would not recommend ZoL for anything outside a test environment at this time (I’ll explain in a moment). ZoL may be useful for long term storage or a backup NAS box if you’re particularly brave. However, if you plan on deploying ZFS permanently or at least deploying it in a long term installation, I’d recommend using an OS with which ZFS is tightly integrated; recommendations include FreeBSD and FreeBSD-based distributions (including FreeNAS) or OpenIndiana-based platforms. I’d also shy away from using ZFS on systems that are intended to be used as general purpose machines; in my experience, ZFS really shines in a heavily storage-centric configuration, such as NAS, where throughput isn’t as important as integrity. Outside this environment, ZFS may be a performance liability, and current benchmarks demonstrate underwhelming behavior when used as a backing store on Linux for databases. In effort to work around this, ZFS requires planning and tuning for use with RDBMSes and may also impact write-heavy applications. Read-only loads are something of a wash–file system choice is better made with regards to workload requirements: Is raw performance or data integrity more important? Note that and the latter–data integrity–can be solved via other means, depending on use case, but ZFS’s automated self-healing capabilities are hugely attractive in this arena.

As of early 2017, ZFS on Linux is still exceedingly painful to install and maintain for rolling release distributions. In particular, on Arch Linux, ZFS will cripple most upgrade paths available for the kernel, itself, and applications that require ZFS dataset tuning. It’s almost impossible to upgrade to more recent kernels if you pride your sanity (and system stability) unless you expect to wait until such time as the latest kernel has been tested, new ZFS PKBUILDs are ready, and you’re open to an afternoon of potential headaches. Generally speaking, the upgrade process itself isn’t frustrating, but it should always be preceded by a fresh backup–and keep your rescue media handy!

Never use ZFS with DKMS in effort to shortcut kernel versioning requirements, even if upstream ZoL appears to support the new version–the package you’re using may not be updated yet, and the AUR DKMS PKGBUILDS for ZFS are not as stable or well maintained as the kernel-pinned packages. With DKMS, if there’s a mismatch, even slightly, you risk potential kernel panics and a long, painful recovery process. I was bit by this early on and discovered that some kernel panics aren’t immediate; instead, they may occur after an indeterminate period of time depending on file system activity, and the recompilation process of ZFS and SPL will consume a sufficient amount of time such that your system will likely panic before the build completes. Stick with the version-fixed builds; don’t use the DKMS modules. Of course, this introduces a new problem: Now you have to pin ZFS to a specific kernel version, leading to extra work on rolling release distributions like Arch…

It’s necessary to plan for a lot of extra work when deploying ZoL. Estimate how much time you’re likely (or willing) to spend and multiply it by two. I’m not even kidding. ZoL on distributions without completely pre-built packages (possibly others) require all of the following: Building SPL, its utilities; ZFS and its utilities; installing unstable versions of grub if you plan on keeping /boot on ZFS (with the implication that you’re one upgrade away from an unbootable system at all times); dataset tweaking per application; and potential bugs. Lots of them. When I first deployed ZFS on Linux, I was completely unaware of a deadlock condition which affected the arc_reclaim kernel process, and everytime I’d run rsync, arc_reclaim would hang, CPU usage would spike, and it’d be necessary to manually intervene. To say nothing of the RAM usage…

ZFS performance under Linux is also relatively poor for my use case. While read/write speeds are acceptable, it’s nowhere near ext4, and it’s likely slower than ZFS on the same hardware running FreeBSD. Furthermore, the memory pressure due to the ZFS driver’s ARC failing to release RAM back to the system (it’s supposed to, but I’ve yet to see it in practice) under memory-intensive operations can cause an out-of-memory condition, swapping, and a potentially fatal invocation of the Linux OOM killer. For this reason alone, I could never recommend ZoL on a general purpose server. If your deployment is intended exclusively as a NAS and you have literally no other services besides NFS/Samba, ZFS’ memory usage would be perfectly fine, provided it’s on a system with 8+ GiB RAM (though you should have more). Then there’s the possibility of L2ARC devices, caching devices, and so forth. If you’re planning on running other services in addition to a ZFS-backed NFS server, such as GitLab or Minecraft, you’ll quickly find that striking a balance between RAM allocation for the ARC versus other applications becomes a somewhat tedious chore. In fact, I might even venture as far as to suggest that you shouldn’t consider running ZFS plus other applications on anything less than 16 GiB RAM–preferably 32 to hand off a nice big chunk to the ARC, particularly if you plan on expanding drive capacity, and you still shouldn’t run anything you don’t absolutely need (seriously: Make it a pure NAS).

Tweaking ZFS for database loads doesn’t seem particularly noisome–certainly not on the surface–until you encounter one or more upgrade cycles that require more than just a dump/load. If you follow the default Arch Linux upgrade process for PostgreSQL, you’ll quickly find ZFS less flattering than alternatives. Not only is it necessary to tweak the recordset size in addition to a few other file system attributes for performance reasons (though you only do this at dataset creation), but suddenly, following the Arch upgrade guide by moving the old data directory and creating a new one is plagued with matters of shuffling around extra files, remounting previously tweaked datasets to the appropriate location, and more. In my case, I had to copy all of the original data to a new directory, wipe the old mount point for both the data directory and the write-ahead log, then create a new data directory in the old dataset mount point, copy the WAL into the WAL-specific mount point, mount it at pg_xlog, and only then could I complete the upgrade process. MySQL on ZFS is generally easier to upgrade, in my experience, but I also use it much less frequently. Be aware that MySQL still requires dataset tweaks, and the tweaks applied depend on whether you’re using primarily MyISAM or InnoDB. I’ve not experimented sufficiently to understand whether it’s possible to tweak datasets for individual storage engines.

Of course, there’s a few other relatively minor niggles that depend on your specific case. For example, grub2 naturally has no understanding of ZFS (or XFS for that matter), so it’s necessary to install grub-git if your /boot partition isn’t on a separate ext3/4 install. Under Arch, it’s also necessary to make certain your initrd is correctly configured via /etc/mkinitcpio.conf, and it’s almost always a good idea to re-run mkinitcpio even after upgrading or installing the kernel just in case it didn’t pick up your ZFS modules (you may need the binaries, too). Otherwise, you’ll be resorting to your emergency boot media to fix the dilemma (you did create it, didn’t you?).

A Less Optimal Solution

I consider the experiment with ZFS on Linux a fantastic success even though I’m already planning to migrate away from it. For my needs, I’m reluctant to run FreeBSD even though I’ve used it for similar purposes in the past. Thus, I’ll be reinstalling the machine with a combination of ext4 + mdadm (actually copying it back over, but there’s no functional difference insofar as downtime). In retrospect, I’ll probably miss ZFS’ transparent compression the most. Given my relatively modest data size and the fact that it defaults to lz4 compression (speed optimized), it’s surprising that it’s saved close to 200 GiB of storage! No other file system, save for Btrfs, provides transparent compression, and in spite of the integrity guarantees ZFS provides, I think compression is a far more pragmatic upside since its impact is real and immediate rather than theoretical.

Although I’d like to wax philosophical about ZFS’ touted benefits, I still can’t help but think it’s solving a problem that is gratuitously overblown. Perhaps bitrot is a real, material risk, but I’ve rarely been affected by it (ancient backup CDROMs notwithstanding). Has it affected my archives? Almost certainly so, but if it has, it’s never had an impact on photos or media, much less other, more critical data; the few times I’ve performed checksum validation of archives versus physical disk contents, I haven’t encountered a mismatch. Indeed, although it’s a problem ZFS is almost tailor-made to fix, it still doesn’t beat regular, extensive backups. Of course, that assumes you have a mechanism in place that would prevent your backups from being adulterated or overwritten by later, corrupted snapshots (and that your backups aren’t subject to bitrot as well), but I think Google’s solution here is far more apropos: Keep no fewer than three copies of your most important data. Surely one of them, statistically, will survive.

You’ll notice that I haven’t mentioned ZFS snapshots (or send/receive), because I’ve yet to encounter a circumstance (besides upgrades, perhaps) where they’re useful to me. While I’d like to use them with containers, there’s still the very real problem of running software inside a container that requires dataset tweaking, and there’s also the specter of lingering problems with ZoL’s implementation of ZFS which has had problems as recently as last year with snapshots (mostly with send/receive if memory serves). In my case, I tend to avoid advanced features if there’s a risk of causing damage because they’re either not well-tested, buggy, or have had a recent history of inducing failures. But alas, I’m conservative at heart; I’d happily poke about with ZFS snapshots in a virtual machine to see how they work, but I’m much less happy about using them on a real server that’s doing real work where downtime would likely interfere with important activities when those same kernel drivers are of dubious stability. I also have no other ZFS systems where send/receive would benefit me.

There is an alternative file system some of the more astute readers among you may have noticed in my list of omissions: Btrfs. I considered Btrfs for my server, testing it briefly, but at the time (mid-2016), I encountered some evidence that suggested Btrfs may not be particularly stable in spite of it being among the list of default file systems for some distributions. Btrfs’ tools feel lacking, dampening my confidence further.

The Btrfs authors have as recently as August 2016 admitted to substantial, possibly unfixable problems with then-current Btrfs’ RAID5/6 implementations. Although I’m running a simple mirror, the fact that such bug would be present in a file system some distributions have optimistically labeled as “stable” is worrisome (but just don’t use its RAID5/6 features–or whichever other features happen to be broken). I’ve seen comments from as early as 2014/2015 lauding the benefits of Btrfs as a stable, tested platform, but I have serious reservations substituting caution with optimism, particularly when 1-2 years later in 2016, it would appear such optimism is horribly misplaced. Consequently, I don’t see Btrfs as a viable alternative (yet!), and that’s without addressing Btrfs’ performance history with regards to PostgreSQL and other databases. This may change, eventually, and Btrfs does look promising. There are some valid criticisms that Btrfs is simply reinventing ZFS (poorly), but being as ZFS will likely never be included in the kernel due to licensing conflicts, Btrfs is better poised to reach parity with the likes of ext4 and company. I’m not optimistic about the Btrfs timeline, and while I’d be surprised if it attains feature completeness before 2020, I do believe many of its misgivings will be resolved well before then.

Back to ZFS: Will I revisit it in the future? Absolutely. If I do, I may put together a FreeBSD NAS for additional storage or the likes. For now, however, ext4 is suitable enough for my needs. It’s certainly good enough for Backblaze, and while I don’t share their requirements, I take a more pragmatic approach. If ext4 is good enough for them, it’s good enough for me.

No comments.
***

Updates Suck (Part 2): KDE Plasma 5.5 breaks task manager minimized application display

If you’ve recently updated to KDE Plasma 5.5, you may have noticed that minimizing your applications no longer produces a noticeable effect in the task manager. (In actuality, the status effect of minimized versus non-minimized applications is limited to a thin, gray outline around the application title–hardly something that conveys application state well enough to process visually at first glance.)

The fault of this change lies with this bug report posted almost 3 years ago. After much discussion, it was decided that 1) the “disabled” state of minimized applications was a mistake, 2) the minimized state makes it too difficult for some people to locate the gray scale application icon, and 3) no one wanted to add a configuration option to enable or disable this feature, so the next best thing was to rip it out. Now, I’ll be honest: Maybe I just happen to have better-than-average pattern matching capabilities in my visual cortex (doubtful), but the gray scale icons for minimized applications really don’t bother me. What does bother me is this change: It takes me twice as long to find minimized applications on the task manager. To say nothing of the fact that two or three people complaining incessantly for 3 years can effect such a substantial change on the rest of us who actually liked the feature…

Humorously, I didn’t notice this change during the initial update. Since I spread my work out across multiple desktops, I don’t often minimize my applications. This explains, at least in part, why it took three days for me to discover this problem and make a post about it (and why I blamed it on plasma-shell’s propensity to crash all the time, although that seemed like a reasonable explanation in lieu of anything more substantive).

The good news is that I traced it down to this commit which turned off the minimized gray scale display. As expected, removing this patch returns the task manager’s display of minimized applications to its previous state. The bad news is that, yes, you do need to recompile plasma-desktop from source. If you happen to use Arch Linux, I’ve put together a PKGBUILD containing the fix based off the PKGBUILD in ABS (hence why I haven’t yet changed any of the credits–I should note that this is not an official package). I’ll keep this updated as best as I can until such time that upstream finally settles on a better idea than a 1 pixel gray outline for non-minimized applications.

The rest of you will have to apply the patch and build the sources yourself. Sorry about that.

No comments.
***