Over the weekend, I made a number of changes to my NAS server hardware, and along the way decided to upgrade from Open Media Vault (which is based on Debian Wheezy), to something a bit more modern. This bought me to my first experience with systemd which was available in wheezy, but not installed by default.
In Open Media Vault, each drive is mounted by UUID in the directory
/media. On top of this, I pool some of my drives using aufs, which is also mounted in
/media using a UUID generated for the aufs pool. Rather than refer to each of the folders on my media drives via their long (and difficult to remember) UUID-based paths, I also create bind mounts under
/export. For example
/export/Books or whatever. That way if I shift content around different drives, I can simply re-point the bind mounts. Any software (eg, like Plex or whatever) points to the bind mounts.
Under OMV (and Wheezy) this all just “works”. But installing Debian Jessie, I suddenly found problems. The aufs pool seemed to be incomplete, and the bind mounts were completely empty. It took me a while to google and find the needle in the haystack that was causing me problems.
In a pre-systemd world, Debian mounts drives in
/etc/fstab in the order they are listed. If you want to create a pool mount using aufs, you would simply make sure it was listed after drives that aufs was pooling. Similarly, bind mounts would generally need to be listed last, since you are binding a folder from another drive, hence that drive needs to be mounted.
The problem is that systemd tries to streamline (and speed-up) the booting process by doing as much as possible in parallel. Under systemd, the order in fstab is not honoured and drives can potentially be mounted in any order. This means systemd might try and mount an aufs pool, before the constituent drives are available. In this case, this is why my my pool was incomplete.
Trying to diagnose this problem has proved to be a pain because there still isn’t a huge amount of knowledge out there around systemd, so I came across some inconsistent posts. Plus, I suspect it’s been something of a moving target over the years, with a lot changes and developments meaning the recommended solution from 2 years ago isn’t necessarily the best solution right now.
At boot, systemd takes the entries in fstab and converts them into ‘unit files’. It then uses the unit files to mount each drive individually. You can find the unit files in
/run/systemd/generator. Best as I can tell, one approach to resolving this issue is to create customised unit files. At boot time, systemd will then use the existing unit files rather than generating new ones from fstab.
The problem with this approach is that (a) depending on the number of drives you have, this could be quite annoying to setup and maintain, and (b) you are now splitting system configuration between fstab and a myriad of unit files.
The good news is that you can add options into your fstab that systemd will understand and parse. Two possible options to consider;
Automounting – by adding the options
noauto,x-systemd.automountto a fstab entry, a drive won’t be mounted on booting. Instead, it is only mounted when referenced. In the case, for example, of a pooled drive, it will only mount when the pool is referenced which is typically after the system has booted, and consequently after the constituent block devices have been mounted. However, in my case, I have bind mounts that have a dependency on a pool mount, which in turn depends on block devices… so I’m not sure how that chain of dependencies would work.
Specifying dependencies – you can add the option
x-systemd.requires=to a fstab entry (followed by the path of the required mount). This ensures that when the systemd unit file is generated, this dependency will be listed and resolved
Specifying the dependencies will result in a unit file (in
/run/systemd/generator) with a line entry for
Requires= (ie, which unit needs to be run first).
If you want to test this first, before rebooting (and potentially borking your system) – edit your fstab (making a backup first), and then run the
systemd-fsab-generator which can be found in
/lib/systemd/system-generators (in Debian). This will generate new unit files for mounting your drives. They can be found in
/tmp (and hence, won’t actually be used). You can read through them, to check the systemd has appropriately parsed your fstab.