systemd

About systemd in Hyperscale¶

We try to follow the systemd build from Fedora rawhide as closely as possible.

CI/CD¶

Contributing to the RPM spec / New Version Updates in the SIG¶

Make sure you're onboarded onto the SIG before following these steps.

For the following examples, most/all of the links will go to the c10s-sig-hyperscale branch when there are mentions of the RPM sources repo. Substitute as needed with other branches (e.g. c9s-sig-hyperscale).

When we're building for 2 releases (e.g. c10s and c9s), you can do most/all of the work on one branch (e.g. c10s) and merge the changes to the other branch (e.g. c9s) using the power of Git. Unless something happens, like a dependency on a newer release not being available on an older release, rarely will changes on the newer branch fail on the older one.

Steps:

Clone the CentOS RPM sources repo
- Checkout the c10s-sig-hyperscale branch
Add the Fedora RPM sources repo as a git remote
- git remote add fedora https://src.fedoraproject.org/rpms/systemd && git fetch fedora
Merge changes from Fedora Rawhide.
- git merge fedora/rawhide
- Solve any merge conflicts
- Carefully go through each commit from Rawhide. Take special care for commits that introduce changes in default behavior (e.g. https://src.fedoraproject.org/rpms/systemd/c/ba02e904964116b848080ca72243174f4ef3eced?branch=ba02e904964116b848080ca72243174f4ef3eced) as these can have a drastic impact on our users. Whenever finding such a commit, bring it up for discussion with the other members of the SIG to determine how we should handle the change.
- Another thing to look out for is newly added Obsoletes for the systemd package itself, for example, Obsoletes: systemd < 246.6-2. These prevent the old version from being installed anymore when the package with the Obsoletes is added to the rpm repository and should be removed until we can be reasonably sure that a newer version of systemd than is being obsoleted is installed everywhere.
Next, you can check the remaining differences between the Fedora rawhide spec and the CentOS Hyperscale spec by running git diff fedora/rawhide..HEAD while on the c10s-sig-hyperscale branch. Try to remove any differences between the two that aren't strictly necessary anymore.
Include patches that haven't been merged upstream yet or which haven't been backported to a stable branch
- Note that we're very hesitant on including downstream patches and there needs to be a very good reason to include downstream patches. Every downstream patch should have a comment to an upstream pull request where the functionality is proposed for inclusion in systemd itself.
- If the patch has been upstreamed but isn't in a stable release yet, you can ask upstream for a new stable release or prepare one yourself using the instructions found here. Urgent fixes or newer feature backports that aren't appropriate for stable releases can be added as patches to the rpm repository itself. To apply a patch to the rpm repository, you can do the following:
- Add something like PatchXXXX: https://github.com/systemd/systemd/pull/$PR.patch to the rpm spec
- If a PR doesn't apply cleanly to the stable branch, cherry-pick the commits into a subbranch of the corresponding stable branch in the systemd repository and open a PR on Github which you can then list in the rpm spec. If the patch is a feature backport, you can immediately close the PR as it won't be merged anyway.
Update the version and release.
- If we're building rpms for the same systemd stable release (from https://github.com/systemd/systemd/releases) as is shipped in Fedora Rawhide, no changes are required to the spec.
- If we're building a different release than the one shipped in Fedora Rawhide, update the default value for the Version field to match the version of the stable release that we want to build (e.g. 256.2).
- Update the Release value. This should match Fedora, except that we also append a dot number value to indicate which iteration of the Hyperscale release we're on (e.g. if Fedora has 256.2-1, Hyperscale should have 256.2-1.1).
Note that generally, aside from changing the version, release and including patches, all other changes should be implemented in the Fedora spec and backported to the Hyperscale spec to keep the diff between the two as small as possible.
- Exceptions can be made when Fedora makes a change that we want to explicitly want to deviate from.
- The upstream systemd repository builds rpms for CentOS Stream using the Fedora Rawhide spec, so the more our spec matches the Fedora Rawhide one, the better our test coverage.
Download the source tarball and patches from the systemd github repository:
- spectool --define "_sourcedir $PWD" -g systemd.spec
Update the sources file with the new tarball name and SHA512 hash.
- Use sha512sum to calculate the SHA512 hash of the tarball.
Upload the new version tarball to the CentOS lookaside cache.
- Upload the tarball to the CentOS lookaside cache by calling the lookaside_upload_sig script found here.
- For example, the command could look as follows:
- centos-git-common/lookaside_upload_sig -f systemd-256.2.tar.gz -n systemd
Build locally with mock
mock -r centos-stream-hyperscale-10-x86_64 -D "%dist .hs.el10" --sources . --spec systemd.spec
The resulting rpms can be found in /var/lib/mock/centos-stream-10-x86_64/result
Test that the new rpms work as expected (See Testing section)
All updates are in and working as expected? Push your changes to a fork of the rpm sources repository, create a pull request and ask the other Hyperscale maintainers to take a look!

Testing¶

mkosi¶

We recommend testing changes using mkosi. We install mkosi from git to make sure we have the latest features and fixes:

git clone https://github.com/systemd/mkosi.git
sudo ln -s /usr/local/bin/mkosi $(pwd)/mkosi/bin/mkosi

To install all required dependencies on the host system, run mkosi dependencies | xargs sudo dnf install. Note that your host system will need to have systemd 256 or newer packaged to be able to run the integration tests successfully.

Testing with the systemd integration tests¶

We'll start by testing our newly built rpms using the systemd integration tests.

Next, clone the upstream systemd repository, checkout the stable branch corresponding to the version we're going to release and configure meson (or use an existing checkout):

git clone https://github.com/systemd/systemd.git
cd systemd
git checkout <stable-branch>
meson setup build

Then, in the systemd repository, build an image including the downloaded rpms:

mkosi \
  --distribution=centos \
  --release=<centos-release> \
  --repositories=hyperscale-packages-main \
  --volatile-package-directory=<path-to-mock-rpms-dir> \
  --environment=NO_BUILD=1 \
  --selinux-relabel=auto \
  --force

Finally, run the integration tests:

SYSTEMD_INTEGRATION_TESTS=1 meson test -C build --no-rebuild --suite integration-tests --num-processes "$(($(nproc) / 3))"

Run through these steps for all supported CentOS Stream releases.

If you have access to a powerful machine, it's recommended to use it ro run the integration tests to speed up the process.

Testing SELinux with the systemd-releng repository¶

To test use cases not covered by the systemd integration tests, we have a separate set of mkosi configuration files in the systemd-releng repository.

Due to SELinux limitations, all mkosi commands in the selinux-releng repository have to be run as the root user as we cannot do arbitrary SELinux relabeling unless we're running with root privileges.

First, clone the repository (or use an existing checkout):

git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng.git
cd systemd-releng

Next, build a Hyperscale image, again including the downloaded rpms:

sudo mkosi --profile hyperscale -r <centos-release> --volatile-package-directory <path-to-mock-rpms-dir> -f

We can now try to boot into the image:

sudo mkosi qemu

If the boot succeeds and we end up in a root shell, great! Otherwise, we'll need to debug. If the boot failed this is either due to SELinux denials or because dracut hasn't been updated to accomodate changes in newer systemd releases. We can start by booting with SELinux in permissive mode:

sudo mkosi --kernel-command-line-extra=enforcing=0 qemu

If this still doesn't get you into a root shell, the problem might lie with dracut, see the next section for more details. Otherwise, regardless of whether we're in enforcing or permissive mode, the first thing to check is whether there's any SELinux denials:

journalctl -t audit -g AVC

If there are denials, we can compare them with the denials on a Fedora Rawhide system by running the following:

sudo mkosi -d fedora -f qemu
journalctl -t audit -g AVC

If the denials don't pop up on Fedora Rawhide, that's a good indication that we should backport a newer version of selinux-policy. To backport a newer version, you can run the following:

git clone https://git.centos.org/rpms/selinux-policy.git
cd selinux-policy
git remote add fedora https://src.fedoraproject.org/rpms/selinux-policy
git fetch rawhide
git checkout rawhide
fedpkg sources
git checkout c<centos-release>s-sig-hyperscale
git merge fedora/rawhide

To double check that the newer version fixes the denials, first use mock to build the rpms:

mock -r centos-stream-hyperscale-<centos-release>-x86_64 --sources . --spec selinux-policy.spec

Then build and boot the image with the new selinux-policy rpms and check if the denials are fixed:

cd <path-to-systemd-releng>
sudo mkosi \
  --profile=hyperscale \
  --release=<centos-release> \
  --volatile-package-directory=<path-to-systemd-rpms-dir> \
  --volatile-package-directory=/var/lib/mock/centos-stream-hyperscale-<centos-release>-x86_64/result \
  --force \
  qemu
journalctl -t audit -g AVC

If the denials that were shown previously are gone, great! The new version of selinux-policy includes fixes for the denials you were seeing. In this case you can go ahead and build the new selinux-policy rpm in Hyperscale:

cd <path-to-selinux-policy>
<path-to-centos-git-common>/lookaside_upload_sig -f macro-expander -n selinux-policy
<path-to-centos-git-common>/lookaside_upload_sig -f container-selinux.tgz -n selinux-policy
<path-to-centos-git-common>/lookaside_upload_sig -f selinux-policy-*.tar.gz -n selinux-policy
git checkout c<centos-release>s-sig-hyperscale
git push
cbs build hyperscale<centos-release>s-packages-main-el<centos-release>s git+https://git.centos.org/rpms/selinux-policy.git#(git rev-parse HEAD)
cbs tag-build hyperscale<centos-release>s-packages-main-release selinux-policy-<version+release>.hs.el<centos-release>

If the denials also occur on Fedora Rawhide, file an issue for the selinux-policy repository on Github. Include the lines from the journalctl output showing the denials. Also include a reproducer to allow the maintainers to reproduce the issue. A reproducer will generally look as follows (you can copy paste this and modify it):

git clone https://github.com/systemd/mkosi.git
ln -s /usr/local/bin/mkosi $(pwd)/mkosi/bin/mkosi
git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng
sudo mkosi -d fedora -f qemu

You can extend the above with any extra commands required to reproduce the SELinux denials in the virtual machine. Now it's a matter of waiting and working with the policy maintainers to fix the denials, and backport the newer version of selinux-policy once it is released in Fedora Rawhide (see above steps).

Testing the dracut backport with the systemd-releng repository¶

Because new systemd releases often require changes in dracut as well, we also maintain a backport of dracut from Fedora Rawhide. Every time a change in a new release of systemd requires a corresponding change in dracut, a newer release of dracut should be backported first before doing a new release of systemd in the Hyperscale SIG.

To backport a new version of dracut from Fedora Rawhide, you can run the following steps:

git clone https://git.centos.org/rpms/dracut.git
cd dracut
git remote add fedora https://src.fedoraproject.org/rpms/dracut
git fetch rawhide
git checkout rawhide
fedpkg sources
git checkout c<centos-release>s-sig-hyperscale
git merge fedora/rawhide

We can then build the new dracut rpms with mock:

mock -r centos-stream-hyperscale-<centos-release>-x86_64 --sources . --spec dracut.spec

And finally test whether the image built with mkosi boots with the new version of dracut as follows:

cd <path-to-systemd-releng>
sudo mkosi \
  --profile=hyperscale \
  --release=<centos-release> \
  --volatile-package-directory=<path-to-systemd-rpms-dir> \
  --volatile-package-directory=/var/lib/mock/centos-stream-hyperscale-<centos-release>-x86_64/result \
  --force \
  qemu

If everything works as expected, we can build and tag the new dracut version in CBS:

cd <path-to-dracut>
<path-to-centos-git-common>/lookaside_upload_sig -f <source.tar.gz> -n dracut
git checkout c<centos-release>s-sig-hyperscale
git push
cbs build hyperscale<centos-release>s-packages-main-el<centos-release>s git+https://git.centos.org/rpms/dracut.git#(git rev-parse HEAD)
cbs tag-build hyperscale<centos-release>s-packages-main-release dracut-<version+release>.hs.el<centos-release>

If there are still issues with booting, you'll need to debug the issue and resolve it together with upstream. Be sure to also extend this section with any useful debugging information to make the process easier for the next contributor.

Testing upgrades from stock CentOS Stream with the systemd-releng repository¶

To test upgrades from stock CentOS Stream to CentOS Stream + Hyperscale, you can build and boot without the Hyperscale profile and then upgrade from within the virtual machine:

sudo mkosi -f qemu
dnf upgrade

The packages enabling the CentOS Hyperscale repositories are automatically installed when building the image, so running dnf upgrade after booting is sufficient to upgrade the system to CentOS Hyperscale. After upgrading, run journalctl -p warning to see if anything went wrong during the upgrade.

CBS Build Commands¶

For CBS you will need to be onboarded to the SIG.

From inside the fork of the systemd Hyperscale RPM sources repo:

Once the specfile changes are pushed you can do a real build:

git checkout c<centos-release>s-sig-hyperscale

# Scratch build
cbs build --scratch hyperscale<centos-release>s-packages-main-el<centos-release>s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)"

# Official build
cbs build hyperscale<centos-release>s-packages-main-el<centos-release>s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)"

# Tag it for testing
cbs tag-build hyperscale<centos-release>s-packages-main-testing systemd-<version+release>.hs.el<centos-release>

# Tag for release (this will let it go to the mirrors)
cbs tag-build hyperscale<centos-release>s-packages-main-release systemd-<version+release>.hs.el<centos-release>

To do builds in the facebook tag, use the above commands but replace main by facebook.