We'd built an IoT prototype and we wanted its code to run automatically on startup - and I wasn't looking forward to it. In the past I’d used Upstart and init.d for this task on Raspberry Pi. I didn’t remember it as being either super easy or straightforward. But Linux had moved on since those days and so I needed to look into systemd. Once I'd gotten a service to start our main program it was actually fun to fit other services around it, such as a boot screen and shutting down screens. The addition of these make the experience of our IoT product feel really well finished.

Systemd is a Linux init system and service manager. You can control elements of systemd through the systemctl tool, for example to run programs as a service. Such a service is described in a unit configuration file with the .service suffix. Units can be added to a target, aka a collection of units, that is set to run at a key point of the kernel lifecycle, for example during network startup or halt or reboot. Generally speaking there is no order in which services within a target are run, but you can influence the order of events by setting a service to run specifically before or after a target. You can also specify when the service is installed, making it possible to run programs in environments with different resources than the current install environment has. An example of why that's useful: we were able to access assets in the file system during the installation of a service, when we knew that the filesystem wouldn’t be available anymore when the service was run.

For the purpose of this article I’ll assume you haven’t yet changed the pi username. Because I don't want to expand the scope of the subject too much I’m replacing our startup bash script with a simpler example of a such a file called 'iotproduct.sh' that starts a python module called wifi-leaf. Wifi-leaf isn't the product's real name, but because we can't yet talk about the product this name will have to work as a stand-in.

Create and edit the bash file. I used vim.tiny but if you don’t like vim any other available text-editor like nano will do just fine.

This is the contents of the bash script:

Make sure your bash script is executable:

Test it in the terminal. When you’re happy it works you can add it to a service. Create a file for the service and edit it.

Copy in the following content and save:

The After parameter within the Unit points to multi-user.target. Multi-user.target collects the units needed for setting up a non-graphical multi-user system. We’re running the script as the pi user, who is available after loading multi-user.target.

The default type for a service is Type=simple. The behaviour of Type=idle is similar, except the execution is delayed until all active jobs are dispatched. Doing this avoids interleaving shell outputs from services on the console.

The bit at the end of the shell call, 2>&1, redirects channel 2 (Standard Error) and channel 1 (Standard Output) to the current context, i.e. the log file.

If you want to double check what you’ve written in the file you don’t have to include the path for the file in order to cat it. Systemctl knows where to look for services. You can simply do:

Enable the service and start it with:

After starting my service and everything running smoothly I rebooted the system and my service didn’t start up. I checked the status with

The service came up as failed. 😣 Why? Log files came to the rescue in the form of journalctl. Journalctl, like systemctl, is a systemd utility, but instead of managing the kernel and user processes it manages logging. I used journalctl to check if systemd had tried to load my service with:

Once I could see what was happening I could fix my code and reload the service with:

My Raspberry Pi came with Plymouth pre-installed, which seems like the standard Pi way of doing splash screens but I figured it would be just as easy to run a splash screen with systemd. I originally tried to run one at sysinit.target but couldn’t get it to work. Trying to get as much info as possible I came across the following command:

Running this will give you output similar to:

This output gives you info on the system’s history. It also shows you where you are currently in the system, what is and isn’t available to you at different times during the startup sequence (file system, wifi, users etc) and how long each part of the sequence takes to start up. I figured that if I moved my service from sysinit.target to basic.target the timing difference would be negligible from a user point of view and I’d probably have more resources available.

Basic target was described as pulling-in all local mount points plus /var, /tmp and /var/tmp, swap devices, sockets, timers, path units and other basic initialisation necessary for general purpose daemons and it is meant as a synchronisation point for late boot services. Using basic.target as the place to run my service instead of sysinit.target solved the problems I was having. Interestingly because basic.target is the default target after which services run it turned out I didn't have to specify it as an 'After' as I originally had done with the sysinit.target, resulting in the following super short service description.

You enable the service and start it as follows:

In case you're interested in the framebuffer python script I've included it here:

Having created a service for a splash screen on boot I also wanted one for power down. If we'd simply ended our IoT device program and waited for halt, then the user would be looking at a black screen for a few seconds. To avoid this we had our program send a shutting-down image to the frame buffer as its last image before exit. When our shutting-down-image service kicks in on halt we see the same image with progress.

First shutting-down image that our program writes to the frame buffer as its last image before exiting

Second shutting-down image triggered by our service on halt

Our service looks like this:

Type=oneshot waits with the execution of all other programs until this one is finished running. Since the process run by this service only writes an image to the screen and then quits it's not likely to result in any delays.

DefaultDependencies=no makes sure that the service isn't stopped by the shutdown process. The documentation says that only services involved with early boot or late system shutdown should disable this option.

Enable the service and start it with:

As you probably know if you've worked with Raspberry Pi's it’s unsafe to remove the power while it’s running as you could corrupt the SD card. You can solve this by building your own little UPS using something like this but we were a bit late in the process to be adding modules to our device so we decided to handle it with good user messaging like vintage computers back in the day. In order to show the safe-to-unplug message I would have to make sure that the image was shown after the file system was unmounted to avoid corruption. It’s possible to do that by installing the unit when the filesystem is still available and run the service from RAM after unmounting. A good moment to run this service seemed to be around the time final.target is run which is defined as: “A special target unit that is used during the shutdown logic and may be used to pull in late services after all normal services are already terminated and all mounts unmounted.”

The resulting unplug-image service looked like this:

Enable the service and start it with:

Initially I didn’t find the documentation of systemd super clear, but its systemctl and journalctl tools are very good. Systemd facilitates an environment in which it is fairly straightforward to create a seamless end-to-end user experience around a core service, using boot images, shutting down and power down images. Despite my initial scepticism I now <3 systemd.

All is all that ends well right? Well... we weren't done just yet. To really clean up the product we needed to remove the messaging from the tty console on startup and shutdown. This will be discussed in my next blogpost: how tooling improved the quality of our IoT product development on Raspberry Pi.