systemd

About systemd in Hyperscale¶

We try to follow the systemd build from Fedora rawhide as closely as possible.

CI/CD¶

Contributing to the RPM spec / New Version Updates in the SIG¶

Make sure you're onboarded onto the SIG before following these steps.

For the following examples, most/all of the links will go to the c10s-sig-hyperscale branch when there are mentions of the RPM sources repo. Substitute as needed with other branches (e.g. c9s-sig-hyperscale).

When we're building for 2 releases (e.g. c10s and c9s), you can do most/all of the work on one branch (e.g. c10s) and merge the changes to the other branch (e.g. c9s) using the power of Git. Unless something happens, like a dependency on a newer release not being available on an older release, rarely will changes on the newer branch fail on the older one.

Steps:

Clone the CentOS RPM sources repo
- Checkout the c10s-sig-hyperscale branch
Add the Fedora RPM sources repo as a git remote
- git remote add fedora https://src.fedoraproject.org/rpms/systemd && git fetch fedora
Merge changes from Fedora Rawhide.
- git merge fedora/rawhide
- Solve any merge conflicts
- Carefully go through each commit from Rawhide. Take special care for commits that introduce changes in default behavior (e.g. https://src.fedoraproject.org/rpms/systemd/c/ba02e904964116b848080ca72243174f4ef3eced?branch=ba02e904964116b848080ca72243174f4ef3eced) as these can have a drastic impact on our users. Whenever finding such a commit, bring it up for discussion with the other members of the SIG to determine how we should handle the change.
- Another thing to look out for is newly added Obsoletes for the systemd package itself, for example, Obsoletes: systemd < 246.6-2. These prevent the old version from being installed anymore when the package with the Obsoletes is added to the rpm repository and should be removed until we can be reasonably sure that a newer version of systemd than is being obsoleted is installed everywhere.
Next, you can check the remaining differences between the Fedora rawhide spec and the CentOS Hyperscale spec by running git diff fedora/rawhide..HEAD while on the c10s-sig-hyperscale branch. Try to remove any differences between the two that aren't strictly necessary anymore.
Include patches that haven't been merged upstream yet or which haven't been backported to a stable branch
- Note that we're very hesitant on including downstream patches and there needs to be a very good reason to include downstream patches. Every downstream patch should have a comment to an upstream pull request where the functionality is proposed for inclusion in systemd itself.
- If the patch has been upstreamed but isn't in a stable release yet, you can ask upstream for a new stable release or prepare one yourself using the instructions found here. If there was a recent stable release and the issue that we need fixed only happens rarely in very specific use cases, the corresponding commit can be added as a patch to the rpm repository instead.
- Run git format-patch <commit> in the systemd repository
- Move the patch to the rpm repository
- Add something like PatchXXXX: <patch-file> to the rpm spec
Update the version and release.
- If we're building rpms for the same systemd stable release (from https://github.com/systemd/systemd/releases) as is shipped in Fedora Rawhide, no changes are required to the spec.
- If we're building a different release than the one shipped in Fedora Rawhide, update the default value for the Version field to match the version of the stable release that we want to build (e.g. 256.2).
- Update the Release value. This should match Fedora, except that we also append a dot number value to indicate which iteration of the Hyperscale release we're on (e.g. if Fedora has 256.2-1, Hyperscale should have 256.2-1.1).
Note that generally, aside from changing the version, release and including patches, all other changes should be implemented in the Fedora spec and backported to the Hyperscale spec to keep the diff between the two as small as possible.
- Exceptions can be made when Fedora makes a change that we want to explicitly want to deviate from.
- The upstream systemd repository builds rpms for CentOS Stream using the Fedora Rawhide spec, so the more our spec matches the Fedora Rawhide one, the better our test coverage.
Download the source tarball from the systemd github repository:
- spectool --define "_sourcedir $PWD" -g systemd.spec
Update the sources file with the new tarball name and SHA512 hash.
- Use sha512sum to calculate the SHA512 hash of the tarball.
Upload the new version tarball to the CentOS lookaside cache.
- Upload the tarball to the CentOS lookaside cache by calling the lookaside_upload_sig script found here.
- For example, the command could look as follows:
- centos-git-common/lookaside_upload_sig -f systemd-256.2.tar.gz -n systemd
Build locally with mock
mock -r centos-stream-hyperscale-10-x86_64 -D "%dist .hs.el10" --sources . --spec systemd.spec
The resulting rpms can be found in /var/lib/mock/centos-stream-10-x86_64/result
Test that the new rpms work as expected (See Testing section)
All updates are in and working as expected? Push your changes to a fork of the rpm sources repository, create a pull request and ask the other Hyperscale maintainers to take a look!

Testing¶

mkosi¶

We recommend testing changes using mkosi. We install mkosi from git to make sure we have the latest features and fixes:

git clone https://github.com/systemd/mkosi.git
sudo ln -s /usr/local/bin/mkosi $(pwd)/mkosi/bin/mkosi

To install the dependencies required by mkosi, we can either use a tools tree or install the required dependencies on our host system. To use a tools tree, write the following to mkosi.local.conf within the same directory containing the mkosi configuration that we're about to build (see below):

[Host]
ToolsTree=default
ToolsTreeDistribution=centos
ToolsTreeRelease=9
ToolsTreeRepositories=epel,epel-next,hyperscale-packages-main

To install all required dependencies on the host, run mkosi dependencies | xargs sudo dnf install. Note that your host system will need to have systemd 256 or newer packaged to be able to run the integration tests successfully.

Testing with the systemd integration tests¶

We'll start by testing our newly built rpms using the systemd integration tests.

Next, clone the upstream systemd repository, checkout the stable branch corresponding to the version we're going to release and configure meson (or use an existing checkout):

git clone https://github.com/systemd/systemd.git
cd systemd
git checkout <stable-branch>
meson setup build

Then, in the systemd repository, build an image including the downloaded rpms:

mkosi \
  --distribution=centos \
  --release=<centos-release> \
  --repositories=hyperscale-packages-main \
  --volatile-package-directory=<path-to-mock-rpms-dir> \
  --environment=NO_BUILD=1 \
  --selinux-relabel=auto \
  --force

Finally, run the integration tests:

SYSTEMD_INTEGRATION_TESTS=1 meson test -C build --no-rebuild --suite integration-tests --num-processes "$(($(nproc) / 3))"

Run through these steps for all supported CentOS Stream releases.

If you have access to a powerful machine, it's recommended to use it ro run the integration tests to speed up the process.

Testing SELinux with the systemd-releng repository¶

To test use cases not covered by the systemd integration tests, we have a separate set of mkosi configuration files in the systemd-releng repository.

Due to SELinux limitations, all mkosi commands in the selinux-releng repository have to be run as the root user as we cannot do arbitrary SELinux relabeling unless we're running with root privileges.

First, clone the repository (or use an existing checkout):

git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng.git
cd systemd-releng

Next, build a Hyperscale image, again including the downloaded rpms:

sudo mkosi --profile hyperscale -r <centos-release> --volatile-package-directory <path-to-mock-rpms-dir> -f

We can now try to boot into the image:

sudo mkosi qemu

If the boot succeeds and we end up in a root shell, great! Otherwise, we'll need to debug. If the boot failed this is likely due to SELinux denials. We can start by booting with SELinux in permissive mode:

sudo mkosi --kernel-command-line-extra=enforcing=0 qemu

This should get you into a root shell. Regardless of whether we're in enforcing or permissive mode, the first thing to check is whether there's any SELinux denials:

journalctl -t audit -g AVC

If there are denials, we can compare them with the denials on a Fedora Rawhide system by running the following:

sudo mkosi -d fedora -f qemu
journalctl -t audit -g AVC

If the denials don't pop up on Fedora Rawhide, that's a good indication that we should backport a newer version of selinux-policy. To backport a newer version, you can run the following:

git clone https://git.centos.org/rpms/selinux-policy.git
cd selinux-policy
git remote add fedora https://src.fedoraproject.org/rpms/selinux-policy
git fetch rawhide
git checkout rawhide
fedpkg sources
git checkout c<centos-release>s-sig-hyperscale
git merge fedora/rawhide

To double check that the newer version fixes the denials, first use mock to build the rpms:

mock -r centos-stream-hyperscale-<centos-release>-x86_64 --sources . --spec selinux-policy.spec

Then build and boot the image with the new selinux-policy rpms and check if the denials are fixed:

cd <path-to-systemd-releng>
sudo mkosi \
  --profile=hyperscale \
  --release=<centos-release> \
  --volatile-package-directory=<path-to-systemd-rpms-dir> \
  --volatile-package-directory=/var/lib/mock/centos-stream-hyperscale-<centos-release>-x86_64/result \
  --force \
  qemu
journalctl -t audit -g AVC

If the denials that were shown previously are gone, great! The new version of selinux-policy includes fixes for the denials you were seeing. In this case you can go ahead and build the new selinux-policy rpm in Hyperscale:

cd <path-to-selinux-policy>
<path-to-centos-git-common>/lookaside_upload_sig -f macro-expander -n selinux-policy
<path-to-centos-git-common>/lookaside_upload_sig -f container-selinux.tgz -n selinux-policy
<path-to-centos-git-common>/lookaside_upload_sig -f selinux-policy-*.tar.gz -n selinux-policy
git checkout c<centos-release>s-sig-hyperscale
git push
cbs build hyperscale<centos-release>s-packages-main-el<centos-release>s git+https://git.centos.org/rpms/selinux-policy.git#(git rev-parse HEAD)
cbs tag-build hyperscale<centos-release>s-packages-main-release selinux-policy-<version+release>.hs.el<centos-release>

If the denials also occur on Fedora Rawhide, file an issue for the selinux-policy repository on Github. Include the lines from the journalctl output showing the denials. Also include a reproducer to allow the maintainers to reproduce the issue. A reproducer will generally look as follows (you can copy paste this and modify it):

git clone https://github.com/systemd/mkosi.git
ln -s /usr/local/bin/mkosi $(pwd)/mkosi/bin/mkosi
git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng
sudo mkosi -d fedora -f qemu

You can extend the above with any extra commands required to reproduce the SELinux denials in the virtual machine. Now it's a matter of waiting and working with the policy maintainers to fix the denials, and backport the newer version of selinux-policy once it is released in Fedora Rawhide (see above steps).

Testing upgrades from stock CentOS Stream with the systemd-releng repository¶

To test upgrades from stock CentOS Stream to CentOS Stream + Hyperscale, you can build and boot without the Hyperscale profile and then upgrade from within the virtual machine:

sudo mkosi -f qemu
dnf upgrade

The packages enabling the CentOS Hyperscale repositories are automatically installed when building the image, so running dnf upgrade after booting is sufficient to upgrade the system to CentOS Hyperscale. After upgrading, run journalctl -p warning to see if anything went wrong during the upgrade.

CBS Build Commands¶

For CBS you will need to be onboarded to the SIG.

From inside the fork of the systemd Hyperscale RPM sources repo:

Once the specfile changes are pushed you can do a real build:

git checkout c<centos-release>s-sig-hyperscale

# Scratch build
cbs build --scratch hyperscale<centos-release>s-packages-main-el<centos-release>s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)"

# Official build
cbs build hyperscale<centos-release>s-packages-main-el<centos-release>s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)"

# Tag it for testing
cbs tag-build hyperscale<centos-release>s-packages-main-testing systemd-<version+release>.hs.el<centos-release>

# Tag for release (this will let it go to the mirrors)
cbs tag-build hyperscale<centos-release>s-packages-main-release systemd-<version+release>.hs.el<centos-release>

To do builds in the facebook tag, use the above commands but replace main by facebook.