RESTful design

ChatGPT answers "What does RESTful design refer to?":

RESTful design, or Representational State Transfer, is an architectural style for designing networked applications. It was introduced by Roy Fielding in his doctoral dissertation in 2000. REST is not a protocol but rather a set of constraints and principles that, when applied to a system, help create scalable, maintainable, and loosely coupled web services.

Here are the key principles of RESTful design:

1. Stateless Communication:

   • Each request from a client to a server must contain all the information needed to understand and fulfill the request. The server should not store any information about the client between requests. Each request from a client to a server must contain all the information needed to understand and fulfill the request.

2. Client-Server Architecture:

   • The client and server are separate entities that communicate over a network. The client is responsible for the user interface and user experience, while the server is responsible for processing requests, managing resources, and handling business logic.

3. Uniform Interface:

   • RESTful APIs should have a consistent and uniform interface. This principle is further divided into several constraints:
     • Resource Identification: Resources (such as data objects or services) are identified by URIs (Uniform Resource Identifiers).
     • Resource Manipulation through Representations: Resources are manipulated using representations, and these representations can be in various formats (e.g., JSON, XML).
     • Self-Descriptive Messages: Each message from the server to the client should include enough information to describe how to process the message.
     • Hypermedia as the Engine of Application State (HATEOAS): Clients interact with the application entirely through hypermedia provided dynamically by the application servers.

4. Stateless Communication:

   • Each request from a client to a server must contain all the information needed to understand and fulfill the request. The server should not store any information about the client between requests.

5. Cacheability:

   • Responses from the server can be explicitly marked as cacheable or non-cacheable. This improves efficiency and scalability by allowing clients to cache responses.

6. Layered System:

   • A client cannot ordinarily tell whether it is connected directly to the end server or to an intermediary along the way. Intermediary servers (proxies, gateways, etc.) can be added to improve scalability or enforce security without impacting the client.

RESTful design is commonly used in the development of web services, including APIs (Application Programming Interfaces) for IoT devices, web applications, and mobile applications. It emphasizes simplicity, scalability, and the decoupling of components, making it a popular choice for building distributed systems on the web.

Modern USB gadget on Linux & how to integrate it with systemd

Modern USB gadget on Linux & how to integrate it with systemd (Part 1)

Andrzej Pietrasiewicz
February 18, 2019

    

Reading time: 10 minutes

tl;dr: Automate your gadget creation. A look at how to implement USB gadget devices on Linux machines which have the necessary UDC hardware, automate the manual configfs process via declarative gadget "schemes", and use systemd for gadget composition at boot time.

The big picture

In order to understand what is a USB gadget we need to have a look at a broader picture. In USB there are two distinct roles: a host and a device. The purpose of USB is to extend the host with some functionalities provided by devices: be it a mass storage device, an Ethernet card on USB, a sound card or the like. On a given USB bus there can be only one host and many (up to 127) devices. The bus is host-centric, which means that all the activities happening on it are decided and directed by the host.

One way of implementing a USB device is to have a machine running Linux, equipped with a special piece of hardware called USB Device Controller (UDC), and appropriate software running on it. It is exactly this case we will be talking about in this post.

Linux machine as a USB device

Hardware

The question you likely ask yourself is whether your machine has a UDC. In case of desktop PCs you most probably need a dedicated add-on card, which is not a very popular thing: there are not so many users who might want to convert their desktop PC into a USB device. But such boards do exist. The hardware supported by the Linux kernel can be found in drivers/usb/gadget/udc/Kconfig (line 306 in v5.0-rc6). In the embedded world a UDC is very often a part of your system-on-chip (SoC), but beware: merely having a UDC inside SoC does not mean that it is actually connected to anything on the board.

For example Raspberry Pi Zero's SoC does contain a UDC and it is connected to one of the micro USB sockets on the board. Other examples of suitably equipped boards are Odroid U3 and XU3, or Beagle Bone Black. If you don't have access to any hardware of this kind, fear not! You still can play (to some extent) with USB gadgets using an emulated UDC which is a part of the dummy_hcd kernel module. The dummy_hcd combines an emulated host controller with an emulated device controller, so your machine acts as both a USB host and a device. More on dummy_hcd will be in another blogpost which I'm going to write soon.

Oh, and one more thing: you can often come across the term OTG, which stands for on-the-go device and refers to chips which are capable of being either a host, or a device. For the purpose of this post we will be using the term UDC.

Software

The Linux kernel provides drivers for various UDCs. But merely being able to drive a UDC is not enough to fully implement a USB device. What is missing is actual functionality, for example mass storage or Ethernet over USB. And here comes what USB standard says: a USB device can provide more than one functionality over a single USB cable at a time. A set of such functionalities is called a configuration. In fact the standard allows more than one configuration - only one can be active at a time, though - but devices providing more than one are rarely seen in practice.

In the Linux kernel an implementation of a USB device is called a USB gadget. This implementation of gadgets is nicely layered: there is a so called composite layer, which contains code common for all USB functionalities and allows composing gadgets out of several functionalities. The composite layer talks to the UDC driver. On top of the composite driver there are USB functionalities (such as mass storage or Ethernet), which are called USB functions. We will be focusing on the composite layer and functions.

A modern USB gadget

The traditional approach to gadgets composition was to create a kernel module for a given composition of a gadget. And even if you wanted only a slightly different set of USB functions in your gadget, you had to create another kernel module. Around late 2012 a new approach started appearing. The idea was to decouple information about gadget composition from code and only provide building blocks out of which the user composes their gadget at runtime. Very much in the spirit of "mechanism, not policy" philosophy. The interface chosen for userspace interaction was configfs (by default can be found in /sys/kernel/config). After about two years, all USB functions available in the Linux kernel had been converted to use the new interface.

The usage pattern is like this: the user creates a separate directory per each gadget they want to have, gives their gadget a personality by specifying vendor id, product id and USB strings (visible e.g. after running lsusb -v as root), then under that directory creates the configurations they want and instantiates USB functions they want (both by creating respective directories) and finally associates functions to configurations with symbolic links. At this point gadget's composition is already in memory, but is not bound to any UDC. To activate the gadget one must write UDC name to the UDC attribute in the gadget's configfs directory - the gadget then becomes bound to this particular UDC (and the UDC cannot be used by more than one gadget). Available UDC names are in /sys/class/udc. Only after a gadget is bound to a UDC can it be successfully enumerated by the USB host.

A working, minimal example of ECM (Ethernet) on an Odroid U3, which leaves some attributes at their default values:

# go to configfs directory for USB gadgets
CONFIGFS_ROOT=/sys/kernel/config # adapt to your machine
cd "${CONFIGFS_ROOT}"/usb_gadget

# create gadget directory and enter it
mkdir g1
cd g1

# USB ids
echo 0x1d6b > idVendor
echo 0x104 > idProduct

# USB strings, optional
mkdir strings/0x409 # US English, others rarely seen
echo "Collabora" > strings/0x409/manufacturer
echo "ECM" > strings/0x409/product

# create the (only) configuration
mkdir configs/c.1 # dot and number mandatory

# create the (only) function
mkdir functions/ecm.usb0 # .

# assign function to configuration
ln -s functions/ecm.usb0/ configs/c.1/

# bind!
echo 12480000.hsotg > UDC # ls /sys/class/udc to see available UDCs

Please note that your vendor id is assigned for a fee by USB Implementors Forum (USB IF) - this refers to products you want to put on the market. For your own tinkering you can choose whatever you like. However, these ids (vendor and product) can be used by the host to decide which host-side driver to use to talk to your device. 0x1d6b is for Linux Foundation and 0x0104 is for Ethernet Gagdet. If your USB host sees such ids it assumes it needs the cdc_ether host-side driver.

If your device is connected to a Linux host, then you shoud see output similar to the below in host's dmesg:

usb 3-1.2.1.4.4: New USB device found, idVendor=1d6b, idProduct=0104
usb 3-1.2.1.4.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 3-1.2.1.4.4: Product: ECM
usb 3-1.2.1.4.4: Manufacturer: Collabora
cdc_ether 3-1.2.1.4.4:1.0 usb0: register 'cdc_ether' at usb-0000:3c:00.0-1.2.1.4.4, CDC Ethernet Device, d2:c2:2d:b7:8e:6b

Note the Product and Manufacturer strings which are exactly what has been written to configfs.

A new usb interface should appear at the host side...

ifconfig -a

usb0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether d2:c2:2d:b7:8e:6b  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1  bytes 90 (90.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# why not configure it?
ifconfig usb0 192.168.1.2 up

...and at the device:

ifconfig -a

usb0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether f2:40:e6:d3:01:2c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# why not configure this one as well?
ifconfig usb0 192.168.1.3 up

# and ping the host?
ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=1.40 ms

Similarly, the device can be pinged from the host:

ping 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.06 ms

I want my modprobe g_ether back!

Poking around configfs is not difficult, but you need to know where to look, what to look for, what kind of directories to create and what values to write to particular files. And what to symlink from where. Creating a gadget containing just one function takes about 15-20 shell commands, which of course can be scripted. But it seems not a very nice approach. You can instead use an opensource tool called gt (https://github.com/kopasiak/gt) (requires https://github.com/libusbgx/libusbgx), which supports so called gadget schemes: instead of describing creating of your gadget procedurally (explicit shell commands) you describe it declaratively in a configuration file and the tool knows how to parse the file and do all the necessary configfs manipulation. This way modprobe g_ether can be changed to gt load my_ether.scheme, which is a very comparable amount of work :)

The scheme

A scheme corresponding to the above gadget is like this (let's call it ecm.scheme):

attrs : 
{
    idVendor = 0x1D6B;
    idProduct = 0x104;
};
strings = (
        {
                lang = 0x409;
                manufacturer = "Collabora";
                product = "ECM";
        }
);
functions : 
{
    ecm_usb0 : 
    {
        instance = "usb0";
        type = "ecm";
    };
};
configs = ( 
    {
        id = 1;
        name = "c";
        functions = ( 
            {
                name = "ecm.usb0";
                function = "ecm_usb0";
            } );
    } );

Now at the device side you can simply:

gt load ecm.scheme g1 # load the scheme and name the gadget 'g1'

and achieve the same gadget composition as with the above shell commands.

And systemd?

systemd is here. I'm not going to discuss whether it is a good thing or a bad thing. Instead, I want to share with you how you can have systemd compose your gadget, for example at system boot time. A typical example is to have your Ethernet connection over USB up and running and you want that controllable by systemd, so that you for example can systemctl enable/disable your gadget.

The event

The obvious event triggering our gadget creation is the appearance of a UDC in the system. https://github.com/systemd/systemd/issues/11587 points to a pull request, which adds a new udev rule for systemd:

SUBSYSTEM=="udc", ACTION=="add", TAG+="systemd", ENV{SYSTEMD_WANTS}+="usb-gadget.target"

The rule triggers reaching the usb-gadget.target, whose purpose is to mark the point when UDC is available and allow other units depend on it:

[Unit]
Description=Hardware activated USB gadget
Documentation=man:systemd.special(7)

The service

Now that we have a target to depend on, we can create a service which will be started once the target is reached. Let's call it usb-gadget.service.

[Unit]
Description=Load USB gadget scheme
Requires=sys-kernel-config.mount
After=sys-kernel-config.mount

[Service]
ExecStart=/bin/gt load ecm.scheme ecm
RemainAfterExit=yes
ExecStop=/bin/gt rm -rf ecm
Type=simple

[Install]
WantedBy=usb-gadget.target

Such a service can be controlled with systemctl:

systemctl enable usb-gadget.service
systemctl disable usb-gadget.service

This way the usb-gadget.target can be reached with or without actually composing the gadget, or a different gadget can be chosen by the system administrator for each purpose.

I want my own USB function!

The selection of USB functions available for composition is quite large (20), but you still might need something else, for example some custom USB protocol. Or, even not so custom (such as e.g. ptp), but a protocol which is not likely to reach upstream kernel. What to do? FunctionFS to the rescue! I will talk about that in the next installment of this post, so stay tuned.

Modern USB gadget on Linux & how to integrate it with systemd (Part 2)

Andrzej Pietrasiewicz
March 27, 2019

    

Reading time: 10 minutes

In the previous post I introduced you to the subject of USB gadgets implemented as machines running Linux. We also talked about modern style of USB gadget creation and integrating that with systemd. In this post, we look at how to implement your very own USB function with FunctionFS and how to integrate that with systemd.

The general idea of FunctionFS

The idea is to delegate actual USB function implementation to userspace, using a filesystem interface (read()/write() etc). This way, USB out traffic (from host) is available at a file descriptor ready for reading, and USB in traffic (to host) is accepted at a file descriptor ready for writing.

What you need to know before using FunctionFS

In USB all communication is performed through so called endpoints. At the device side the endpoints are (hardware) FIFO queues associated with the UDC. There are 4 types of USB endpoints: control, bulk, iso and interrupt. 

• Control endpoint is endpoint 0 and all USB devices shall have at least endpoint 0. This is a bi-directional communication channel between the host and the device used for enumerating and then controlling the device - remember, the USB is a host centric bus. So even if a device has some new data to be sent to the host it is the host that actually asks for data and the communication required to arrange this happens on the endpoint 0. All other types of endpoints are uni-directional. 
• Bulk endpoints are meant for tansferring (potentially) large amounts of data on a "best effort" basis, that is, as available bandwith allows, so there are no timing guarantees for bulk data. On the other hand bulk data is guaranteed to be transmitted error-free (perhaps using re-transmission under the hood). 
• Iso(chronous) endpoints are meant for time-sensitive data (such as audio or video) with guaranteed bandwitdh and bounded latency. Neither data delivery nor integrity is guarenteed, though, and it is on purpose: in multimedia transmission much more harm is done by non-timely data delivery rather than by occasional non-integrity or even loss of a frame. 
• Interrupt endpoints are for non-periodic data transfers "initiated" by the device (such as mouse movements or keystrokes) and have guaranteed latency. The "initiation" by the device is in fact queuing the data on an endpoint, and then the host polls it when it considers appropriate. In USB it's a host's world. Get used to it or quit using USB.

Why am I telling you this? You need to know that each USB function requires its own set of endpoints, which must not be used at the same time by any other function. This makes UDC's endpoints a scarce resource and imposes a limit on a number of functions which can be provided by your gadget at a time (in one configuration, but you can "overcommit" by specifying multiple configurations - you still remember, that only one of them can be active at a time?). If you want to roll your own USB function, you need to know how many and what kind of endpoints you need, and, especially if you want to compose it with other functions, whether the number of available endpoints is not exceeded. Modern UDCs usually do provide enough endpoints to compose 2-3 moderately "endpoint-hungry" functions into one configuration without any problems.

More detailed idea of FunctionFS

Now that you know about endpoints, you can learn more about FunctionFS. From the point of view of the gadget it is yet another USB function available for composing a gadget from. But it is special, because each instance of FunctionFS provides an instance of a 'functionfs' filesystem to be mounted. If you mount it you will notice there is only one entry: ep0. Its name suggests it is associated with endpoint 0. Indeed it is. However, the good news is that you don't need to implement all the endpoint 0 handling, because most of that is already taken care of for you by the composite layer. You only need to handle control requests directed at your particular USB function. ep0 is also (ab)used to specify the endpoints and USB strings. FunctionFS is not considered ready for binding until endpoint descriptors and strings are written to ep0. The gadget as a whole is not ready for binding if any of its functions is not ready. After the descriptors and strings are written, FunctionFS creates appropriate number of ep<number> files for you to use to implement the desired data streams between the host and the device.

An example function can be found in kernel sources: tools/usb/ffs-test.c The file also contains an example of how to specify USB descriptors and strings to be written to ep0 to make FunctionFS ready. Actually I'm using its modified and simplified version, which passes to the host whatever the host writes to it. We will get to its code later in this post.

And how to integrate THAT with systemd?

Compared to the previous scenario there are more steps involved before the gadget becomes operational:

• load gadget scheme and don't enable yet (because it contains FunctionFS)
• mount FunctionFS instance
• write descriptors and strings
• start the userspace daemon (e.g. ffs-test)
• enable the gadget (now that it's ready, bind it to UDC)

It turns out that systemd already has means to automate writing the descriptors and strings to FunctionFS, and starting a userspace daemon! However, the remaining steps need to be taken care of.

The scheme

Let's call the scheme ffs_test.scheme. It describes a minimal gadget with FunctionFS, leaving some attributes at their default values:

    attrs : 
    {
        idVendor = 0xABCD;
        idProduct = 0x1234;
    };
    strings = ( );
    functions : 
    {
        ffs_loopback : 
        {
            instance = "loopback";
            type = "ffs";
        };
    };
    configs = ( 
        {
            id = 1;
            name = "c";
            functions = ( 
                {
                    name = "ffs.loopback";
                    function = "ffs_loopback";
                } );
        } );

One important thing to note is that the instance name (here: "loopback") becomes a "device" name to be used when mounting this FunctionFS instance.

The service

Let's call our example service unit usb-gadget-ffs.service. This unit is almost identical to the usb-gadget.service, except the ExecStart line:

ExecStart=/bin/gt load -o ffs_test.scheme ffs_test

The difference is that we tell gt to only (-o) load gadget's composition but not bind it, which would fail anyway because the FunctionFS instance is not ready yet.

Another difference is that the Type cannot be "simple" this time, because we need systemd not to schedule loading of dependent modules until gadget scheme is fully loaded - and, consequently, FunctionFS instance registered and made available for mounting. Units of type "simple" are considered completely run immediately, so we need to change to:

Type=oneshot

This change ensures that the dependent modules will be started only after full completion of the gt command.

Complete usb-gadget-ffs.service code:

[Unit]
Description=Load USB gadget scheme
Requires=sys-kernel-config.mount
After=sys-kernel-config.mount

[Service]
ExecStart=/bin/gt load -o ffs_test.scheme ffs_test
RemainAfterExit=yes
ExecStop=/bin/gt rm -rf ffs_test
Type=oneshot

[Install]
WantedBy=usb-gadget.target

The mount

We can use a mount unit to automatically mount our FunctionFS instance. We choose to mount it at /run/ffs_test, so the mount unit must be named run-ffs_test.mount:

[Unit]
Description=Mount FunctionFS instance
Requires=usb-gadget-ffs.service
After=usb-gadget-ffs.service
Before=ffs.socket

[Mount]
# "device" name (FunctionFS instance name)
What=loopback
Where=/run/ffs_test
Type=functionfs
Options=defaults
TimeoutSec=5

[Install]
WantedBy=usb-gadget.target

and then use systemctl enable run-ffs_test.mount

The above two units are enough to load our gadget composition into memory at UDC's appearance and mount its accompanied FunctionFS instance.

Descriptors, strings and the userspace daemon

systemd supports socket units capable of listening to usb traffic directed at FunctionFS. Such a socket unit must be pointed at an already mounted FunctionFS instance. The socket unit contains ListenUSBFunction entry in its [Socket] section to specify exactly that. It also contains a Service entry pointing to the service unit associated with this unit and such a socket unit must have its associated service unit.

When the socket unit is started, it starts its associated service unit, which contains USBFunctionDescriptors and USBFunctionStrings entries. The two specify locations of files containing binary blobs representing USB descriptors and strings, respectively. The service unit's job is then to write those to ep0 of the corresponding FunctionFS instance. And here the systemd's lazy daemon start comes into play: the descriptors and strings are already passed to FunctionFS, so it becomes ready _but_ the daemon is _not_ started until some traffic directed at this FunctionFS instance happens on USB. And it is exactly the job of the socket unit to make it work. The daemon binary is specified with the usual ExecStart entry in the service unit.

The socket unit

Let's call it ffs.socket:

[Unit]
Description=USB function fs socket
Requires=run-ffs_test.mount
After=run-ffs_test.mount
DefaultDependencies=no

[Socket]
ListenUSBFunction=/run/ffs_test
Service=functionfs-daemon.service
# we will get to ExecStartPost later
ExecStartPost=/bin/gt enable ffs_test

[Install]
WantedBy=usb-gadget.target

And then systemctl enable functionfs.socket.

The service unit accompanying the socket unit

We call it functionfs-daemon.service as per the Service entry above.

[Service]
ExecStart=/root/bin/ffs-test
USBFunctionDescriptors=/root/descriptors-ffs-test.bin
USBFunctionStrings=/root/strings-ffs-test.bin

A question you likely ask yourself is where to get the .bin files from? include/uapi/linux/usb/functionfs.h in the kernel sources provides the format description (line 89 in v5.0-rc6). It is up to you how you create these blobs. An example of how to create the descriptors (and strings) in a C program can be found in tools/usb/ffs-test.c. Please note that you are supposed to use usb_functionfs_descs_head_v2, because the other format is now deprecated. You can modify the program so that it doesn't do anything except writing the descriptors/strings to standard output and then capture the result in a file.

Here is a hex dump of descriptors and strings blobs for a high-speed-only example funtion:

# hd descriptors-ffs-test.bin 
00000000  03 00 00 00 27 00 00 00  02 00 00 00 03 00 00 00  |....'...........|
00000010  09 04 00 00 02 ff 00 00  01 07 05 81 02 00 02 00  |................|
00000020  07 05 02 02 00 02 01                              |.......|
00000027

# hd strings-ffs-test.bin 
00000000  02 00 00 00 1d 00 00 00  01 00 00 00 01 00 00 00  |................|
00000010  09 04 55 53 42 20 46 69  6c 74 65 72 00           |..USB Filter.|
0000001d

Please note that the descriptors are created only in the unmodified version of ffs-test.c. As promised, we will get to the modified version later in this post.

Enabling the gadget

If the functionfs.socket did not contain the ExecStartPost entry, then at this point we would have a gadget ready to be bound, but not actually bound. The ExecStartPost contains a command which executes our gadget binding _after_ the service unit is started, which is the last missing piece in this puzzle. Your gadget is now composed and activated with systemd.

Can I use existing daemon code to integrate with systemd?

No, you can't. However, usually patching userspace code to be able to be passed open file descriptors by systemd is a trivial task. And to make it even easier for you to play with FunctionFS and systemd I have modified the ffs-test.c program for using with systemd (it purposedly contains some simplifications in order for you to be able to follow the code easily, so you can't take it as a full-fledged implementation):

https://gitlab.collabora.com/andrzej.p/ffs-systemd/tree/ffs-systemd

look for tools/usb/ffs-test.c on the ffs-systemd branch.

Giving it a try

In order to test the above mentioned modified function you need the following:

• Scheme of the example gagdet from this post
• All systemd units from this post
• ffs-test binary compiled from the modified sources
• FunctionFS endpoint descriptors and strings binary blobs
• usbserial module at host with generic serial support

Set up your gadget and connect it to the host, it should enumerate correctly. The idea is to bind generic usb serial driver at the host to our device. This results in automatic creation of ttyUSB<X> at the host side. And then whatever you write to the ttyUSB<X> you can read back from it. That's what the modified ffs-test.c does - a loopback function.

# at the host
echo 0xabcd 0x1234 > /sys/bus/usb-serial/drivers/generic/new_id

dmesg
<...>
usbserial_generic 3-1.2.1.4.3:1.0: The "generic" usb-serial driver is only for testing and one-off prototypes.
usbserial_generic 3-1.2.1.4.3:1.0: Tell linux-usb@vger.kernel.org to add your device to a proper driver.
usbserial_generic 3-1.2.1.4.3:1.0: generic converter detected
usb 3-1.2.1.4.3: generic converter now attached to ttyUSB1

while true; do dd if=/dev/ttyUSB1 bs=1 iflag=nocache status=none 2>/dev/null; done

And still at the host, at some other console:

man -P cat bash > /dev/ttyUSB1

The contents of bash man page should appear at the console where the "while" loop runs. You can further modify ffs-test.c to e.g. capitalize all the letters, or use some encryption algorithm to cipher your text, effectively creating a hardware crypto device on USB!

What's next?

In the next installment I will write about using dummy_hcd. And in yet another one we will add some systemd templatization to make the above solutions more generic.