Skip to content

systemd

About systemd in Hyperscale

We try to follow the systemd build from Fedora rawhide as closely as possible.

CI/CD

Links can be found on the CI/CD page.

Contributing to the Staging Repo

All of the patches we backport for the Hyperscale build of systemd are first merged to centos-sig-hyperscale/systemd, a Git repo where we stage patches. This allows people to use meson to build systemd with all of the Hyperscale patches, and run them against the systemd upstream unit tests without having to build a RPM.

We try to maintain as few patches as possible for each release, but we are open to including new features or bug fixes that are not yet available. Generally we prefer there to either be an open or merged pull request (PR) in systemd upstream before we will consider merging into the staging repo.

Steps:

  • Get a Fedora account here if you don't have one already.
  • Fork centos-sig-hyperscale/systemd.
  • Check out the latest Hyperscale branch there. It should be the last one alphabetically in the form of "hs-v<systemd-stable version number>". If you want to backport a patch to a stable systemd version that does not yet have a branch available in the staging repo, open an issue here to ask the maintainers to create a new branch for that version.
  • If you think your PR from upstream will apply cleanly, you can download it and git am <patch file> it directly. Otherwise you can use git cherry-pick and manual edits to ensure your change will build.
    • At minimum, do a meson test to ensure unit tests pass.
    • If you're backporting an upstream commit, update the BACKPORTS file with a link to the changes you're backporting
  • Open a PR against centos-sig-hyperscale/systemd with your change.
    • In the PR, include a link to your PR from systemd upstream.
    • Tag anitazha/daandemeyer for reviews.
  • Once merged, one of the systemd Hyperscale maintainers will update the RPM spec and publish a new version of systemd for Hyperscale with the patches.

Preparing a new release in the Staging Repo

  • Run create_or_checkout_release_branch.sh from the systemd-releng repo. Pass the name of the systemd stable release you want to base the new Hyperscale release on as an argument (e.g. ./create_or_checkout_release_branch.sh 250.3). This will push two new release branches to the pagure repository, one for the general Hyperscale release, one for the Meta specific release.
  • Look at the BACKPORTS file for the backports done for the previous release, find out which are still relevant for the next release and backport those changes to the new release branches. First, do the backports for the Hyperscale release branch, then merge the Hyperscale release branch into the Meta specific release branch and then do any Meta specific backports.
  • Add new entries to the BACKPORTS file for the new release branches, along with links to all the backports.
  • Push the branches to the pagure remote.

The new release branches are now ready to be used in the RPM spec. See the next section on how to do a new release for the RPM spec.

Contributing to the RPM spec / New Version Updates in the SIG

Make sure you're onboarded onto the SIG before following these steps.

For the following examples, most/all of the links will go to the c9s-sig-hyperscale branch when there are mentions of the RPM sources repo. Substitute as needed with other branches (e.g. c8s-sig-hyperscale).

When we're building for 2 releases (e.g. c8s and c9s), you can do most/all of the work on one branch (e.g. c9s) and push to the other branch (e.g. c8s) using the power of Git. Unless something happens, like a dependency on a newer release not being available on an older release, rarely will changes on the newer branch fail on the older one.

Steps:

  • Clone the CentOS RPM sources repo
    • Checkout the c9s-sig-hyperscale branch
  • Add the Fedora RPM sources repo as a git remote
    • git remote add fedora https://src.fedoraproject.org/rpms/systemd && git fetch fedora
  • Sync against changes from Fedora Rawhide.
    • The gist of this step is to figure out when we last synced from Fedora Rawhide, and get a diff between the latest Rawhide changes and the last time we synced. Make sure you have the Fedora rawhide branch available for these steps.
    • On the c9s-sig-hyperscale branch, scroll down to %changelog and find out what the the last non-CentOS Hyperscale log line was. You can usually tell by the version-release string in the changelog that doesn't have the additional dot number appended that we use for Hyperscale (e.g. 249.4-2 vs. 249.4-2.1).
    • Switch to the Fedora rawhide branch and use git blame to find out what commit corresponds to the last non-CentOS Hyperscale changelog line.
    • Use git diff <commit>..fedora/rawhide to get all the changes from the latest commit to the last time we synced to CentOS Hyperscale. To apply these changes to the Hyperscale branch, run git apply -3 < $(git diff <commit>..fedora/rawhide). This will apply the changes and generate merge conflicts when the upstream changes conflict with our own changes.
  • Next, you can check the remaining differences between the Fedora rawhide spec and the CentOS Hyperscale spec by running git diff fedora/rawhide..HEAD while on the c9s-sig-hyperscale branch. Try to remove any differences between the two that aren't strictly necessary anymore.
  • Update the source commits, version/release and changelog.
    • At the top of the specfile, update the hs_commit macro to point to the latest commits for the Hyperscale and Facebook branches for the current release on the staging repo respectively.
      • For the v250 release, these would be the latest commits on the hs-v250.3 and hs+fb-v250.3 branches respectively.
    • At the top of the specfile, update the Version and Release values. This should match Fedora, except that we also append a dot number value to indicate which iteration of the Hyperscale release we're on.
    • At the bottom of the specfile under changelog, add an entry about these changes.
  • Run the following commands to download the tarballs for the Hyperscale and Facebook branches:
    • spectool --define "_sourcedir $PWD" --define "facebook 1" -g systemd.spec
    • spectool --define "_sourcedir $PWD" -g systemd.spec
  • Update the sources file with the new tarball names and SHA512 hashes.
    • For example, the c9s-sig-hyperscale sources contains the SHA512 hashes along with the names of the tarballs.
    • Use sha512sum to calculate the SHA512 hashes of the tarballs.
  • Upload the new version tarballs to CentOS.
    • Upload the tarball to the CentOS lookaside cache by calling the lookaside_upload_sig script here.
  • Build locally with mock (See Build Commands section)
  • The resulting rpms can be found in /var/lib/mock/centos-stream-9-x86_64/result
  • Test that the new rpms work as expected (See Testing section)
  • All updates are in and working as expected? Push your changes to a fork of the rpm sources repository, create a pull request and ask the other Hyperscale maintainers to take a look!

Testing - Stock CentOS Stream

Basic Tests

Depending on how many versions of CentOS we need to support (typically 2 max), you will likely want to set up some virtual machines (VM) for testing. Head over to the CentOS Stream website to grab the 2 most recent versions and get your VMs set up. As soon as base CentOS Stream is installed, snapshot your VM! After this point, testing will begin and any number of things can go wrong.

Testing goes like:

  • sudo dnf install centos-packager # if you don't have it already for the cbs command
  • sudo dnf install epel-release centos-release-hyperscale
  • Use cbs download-build --task-id [id] to fetch the systemd rpms you built and tagged for testing.
  • At this point, you will want to make another VM snapshot.
  • sudo dnf upgrade to get the packges from EPEL and Hyperscale.
  • sudo dnf install systemd-networkd systemd-boot if it wasn't installed already.
  • sudo dnf upgrade *.rpm the RPMs you downloaded.
  • sudo systemctl start systemd-networkd
  • systemctl status to make sure the state is not degraded.
  • Reboot the host.
  • Start systemd-networkd again (if it isn't up) and check that the state is still not degraded.
  • Scan the journal and dmesg logs for any errors related to systemd after the timeframe of the upgrade.
  • If you made it this far without seeing errors or running into issues, you're good to tag the non-FB build for release!

It's very normal to make it successfully through the hs+fb rollout (Meta only) only to find that the RPMs won't install, services are broken, or the VM no longer boots with the hs version. Debugging time! Installation issues tend to come from building against a new dependency that is not yet in the mirrors. But the majority of the time, issues arise due to SELinux. CentOS Stream has SELinux enabled by default!

mkosi

You can also do testing using the mkosi image builder:

$ sudo dnf install python3 bubblewrap
$ git clone https://github.com/systemd/mkosi
$ sudo ln -s mkosi/bin/mkosi /usr/local/bin/mkosi
$ git clone https://pagure.io/centos-sig-hyperscale/mkosi hyperscale
$ cd hyperscale
$ mkdir mkosi.conf.d/
$ tee mkosi.conf.d/local.conf <<EOF
[Host]
ExtraSearchPaths=<path-to-systemd-build-directory>

[Content]
WithTests=yes # Change to no if you don't want rpmbuild to run the unit tests.
BuildSources=<path-to-systemd-checkout>:systemd
             <path-to-rpm-checkout>:rpm
EOF
$ mkosi -f qemu
Modifying SELinux Policies (and when your VM fails to boot)

To check that the SELinux isn't spewing denials everywhere:

journalctl -g AVC | audit2allow

You normally want to run this before you install your new systemd RPMs, and then after upgrade and after boot as well.

If your VM no longer boots, roll back to the last working snapshot (hopefully right before you installed the version of systemd you're testing) and try again with SELinux set to "permissive":

  • Open /etc/selinux/config in a text editor.
  • Modify the SELINUX=enforcing line to SELINUX=permissive.
  • Save, close, upgrade systemd, and reboot.

It should boot this time. It also means when you run audit2allow there should be a bunch of new lines. These show which policy denials. At this point you'll want to try modifying the SELinux policy. Inside your VM, git clone the systemd RPM repo from Hyperscale and checkout the corresponding branch that you're testing. Then:

  • sudo dnf install selinux-policy-devel needed to build policies
  • Copy the policy/denial lines from the audit2allow output that are relevant for systemd into systemd_hs.te in the repo checkout. Save and close.
  • make -f Makefile.selinux all to build the policy module.
  • sudo semodule -i systemd_hs.pp.bz2 to install the module you just built.
  • Now when you run audit2allow you'll see messages that the denials are now passing/allowed by the new policy.
  • I recommend rebooting with this new policy and seeing if there are any new/leftover denials you missed. It's possible that some rules just don't audit or some new things appear now that they're not being masked by the old denials. Whatever the case, rinse and repeat a few times until there are no more denials.
  • SNAPSHOT YOUR VM.
  • Set SELinux back to enforcing.
  • Reboot. Hopefully things keep functioning.
  • If they don't, restore your snapshot and keep tweaking the policy until it does.
  • Now you can open a PR with your changes to systemd_hs.te and get this RPM version tagged!

There are some other tools like setroubleshoot / setroubleshootd and sealert that you can try to help debug SELinux. audit2allow is very reliable at getting to the point and tell you how to make SELinux denials stop.

Build Commands

For CBS you will need to be onboarded to the SIG.

From inside the fork of the systemd Hyperscale RPM sources repo:

# Local build with mock. SIG chroots are from mock-centos-sig-configs.
mock -r centos-stream-hyperscale-9-x86_64 -D "%dist .hs.el9" --sources . --spec systemd.spec
mock -r centos-stream-hyperscale-9-x86_64 -D "%facebook 1" -D "%dist .hs+fb.el9" --sources . --spec systemd.spec

# CBS scratch builds (use the SRPMs generated by mock above)
cbs build --scratch hyperscale9s-packages-main-el9s <path to src.rpm>
cbs build --scratch hyperscale9s-packages-facebook-el9s <path to facebook src.rpm>

Once the specfile changes are pushed you can do a real build:

# CBS builds for CentOS Stream 8
git checkout c8s-sig-hyperscale
cbs build hyperscale8s-packages-main-el8 "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)"
cbs build hyperscale8s-packages-facebook-el8 "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)"

# CBS builds for CentOS Stream 9
git checkout c9s-sig-hyperscale
cbs build hyperscale9s-packages-main-el9s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)"
cbs build hyperscale9s-packages-facebook-el9s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)"

# Tag it for testing
cbs tag-build hyperscale8s-packages-main-testing systemd-<version num>.hs.el8
cbs tag-build hyperscale8s-packages-facebook-testing systemd-<version num>.hs+fb.el8
cbs tag-build hyperscale9s-packages-main-testing systemd-<version num>.hs.el9
cbs tag-build hyperscale9s-packages-facebook-testing systemd-<version num>.hs+fb.el9

# Tag for release; This will let it go to the mirrors
cbs tag-build hyperscale8s-packages-main-release systemd-<version num>.hs.el8
cbs tag-build hyperscale8s-packages-facebook-release systemd-<version num>.hs+fb.el8
cbs tag-build hyperscale9s-packages-main-release systemd-<version num>.hs.el9
cbs tag-build hyperscale9s-packages-facebook-release systemd-<version num>.hs+fb.el9