Lookaside/Git

Lookaside/git¶

This page documents how SIGs can upload the sources of their RPM into the lookaside cache and how they should set-up their corresponding git repository.

It should be noted that two model exists, the "traditional" model and the one based on CentOS Stream and Fedora. While both model can be intertwined in some places, it is recommended that SIGs pick a model and follow it. This documentation will not cover which aspect of which model is compatible with the other one.

New Package (from source)¶

If you SIG uses https://git.centos.org and the package you want to build is not there yet, you will have to create a ticket first at: https://pagure.io/centos-infra/issues to request that the git repo for /rpms/<your_package> be created.

If your SIG uses https://gitlab.com/CentOS, you, your SIG chair or people that have been granted access by the SIG chair, will be able to create the project for your package directly.

In the rest of the document we will use the following example : Build the package centpkg-minimal, as a member of the sig-core SIG group. We have to push two things:

the .spec file to build the rpm, to the git repository
eventually an archive (.tar.gz, .tar, .xz...) of the sources, to lookaside cache

Warning

To push to lookaside cache you need to have already a local check out from https://git.centos.org/centos-git-common

Pushing first to lookaside cache¶

Let's assume that my pkg centpkg-minimal that I want to build has an archive called centpkg.minimal.tar.gz. To be able to push to lookaside cache, we need the following :

a valid TLS cert (also needed to build on cbs.centos.org) obtained through centos-cert util
either script lookaside_upload or lookaside_upload_sig in your $PATH (coming from centos-git-common git repo, see above)

There are two path available for the lookaside cache, each is available using a different script:

lookaside_upload allows uploading to the traditional CentOS Linux structure: baseurl/pkgname/branch/hash (example)
lookaside_upload_sig allows uploading to the same lookaside structure as the one used by CentOS Stream and Fedora: baseurl/pkgname/tarball/hashtype/hash/tarball (example)

Both scripts need some paramters, simply use -h to see them.

Back to our example.

Pushing to the "traditional" lookaside structure¶

The pkg name is centpkg-minimal, file is centpkg-minimal.tar.gz and I'm member of the sig-core group, and want to build it for c7, so we'll call it like this :

lookaside_upload -f centpkg-minimal.tar.gz -n centpkg-minimal -b c7-sig-core
[+] CentOS Lookaside upload tool -> Checking if file already uploaded
[+] CentOS Lookaside upload tool -> Initialing new upload to lookaside
[+] CentOS Lookaside upload tool -> URL : https://git.centos.org
[+] CentOS Lookaside upload tool -> Source to upload : centpkg-minimal.tar.gz
[+] CentOS Lookaside upload tool -> Package name: centpkg-minimal
[+] CentOS Lookaside upload tool -> sha1sum: d6616b89617914a0dd0fd5cfa06b0afc7a4541c4
[+] CentOS Lookaside upload tool -> Remote branch: c7-sig-core
[+] CentOS Lookaside upload tool ->  ====== Trying to upload =======

################################################################################################################ 100.0%
File centpkg-minimal.tar.gz size 15178 CHECKSUM d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 stored OK
[+] CentOS Lookaside upload tool -> Validating that source was correctly uploaded ....
[+] CentOS Lookaside upload tool -> [SUCCESS] Source should be available at https://git.centos.org/sources/centpkg-minimal/c7-sig-core/d6616b89617914a0dd0fd5cfa06b0afc7a4541c4

Pushing to the CentOS Stream/Fedora lookaside structure¶

The pkg name is centpkg-minimal, file is centpkg-minimal.tar.gz, so we'll call it like this :

lookaside_upload_sig -f centpkg-minimal.tar.gz -n centpkg-minimal
[+] CentOS Lookaside upload tool -> Checking if file already uploaded
[+] CentOS Lookaside upload tool -> Initialing new upload to lookaside
[+] CentOS Lookaside upload tool -> URL : https://git.centos.org
[+] CentOS Lookaside upload tool -> Source to upload : centpkg-minimal.tar.gz
[+] CentOS Lookaside upload tool -> Package name: centpkg-minimal
[+] CentOS Lookaside upload tool -> sha1sum: d6616b89617914a0dd0fd5cfa06b0afc7a4541c4
[+] CentOS Lookaside upload tool -> Remote branch: c7-sig-core
[+] CentOS Lookaside upload tool ->  ====== Trying to upload =======

################################################################################################################ 100.0%
File centpkg-minimal.tar.gz size 15178 CHECKSUM d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 stored OK
[+] CentOS Lookaside upload tool -> Validating that source was correctly uploaded ....
[+] CentOS Lookaside upload tool -> [SUCCESS] Source should be available at https://git.centos.org/sources/centpkg-minimal/c7-sig-core/d6616b89617914a0dd0fd5cfa06b0afc7a4541c4

Now that we have uploaded to lookaside cache, we can reference it in our /rpms/centpkg-minimal git repository on git.centos.org, see below

Pushing to git¶

SIGs have two options when it comes to the git repositories which can be used as part of dist-git:

If your SIG wants to use gitlab, they will have to follow the instructions to request one. Here we'll cover both workflow, assuming the git repository exists on either platform.

Git clone¶

The first thing you'll need is a local checkout of the git repository for the pkg we'd like to work on. In our case, the git repo url is https://git.centos.org/rpms/centpkg-minimal or https://gitlab.com/CentOS/<sig>/rpms/<repo>.git

So the way to git clone/pull over ssh is either ssh://git@git.centos.org/rpms/centpkg-minimal.git or git@gitlab.com:CentOS/<sign>/rpms/<repo>.git:

git clone ssh://git@git.centos.org/rpms/centpkg-minimal.git
Cloning into 'centpkg-minimal'...
remote: Counting objects: 15, done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 15 (delta 2), reused 0 (delta 0)
Receiving objects: 100% (15/15), done.
Resolving deltas: 100% (2/2), done.

Important

On git.centos.org you can clone over https, but then it wouldn't let you push back to it, as it needs ssh with key verification and based on group membership for acls

Git branches¶

Repositories git.centos.org are used by both Red Hat and CentOS SIG, the structure of the git repository is used to control the access. SIG members are therefore only allowed to push to branches following the pattern: c<version>-sig-<sig_name>*

That means that if I'm member of the sig-core group (in ACO), I'll be able to commit/push to c7-sig-core, c8-sig-core or c9s-sig-core-whatever-I-want branches

For our example, on git.centos.org we'll have to create a new c7-sig-core branch (with on the assumption that we want to build for c7) :

git checkout -b c7-sig-core

Since SIGs have full control over their namespace on gitlab.com, that branching structure is not mandatory. It will, however, be necessary if you plan on using the "traditional" lookaside structure describe above as this relies on branch names.

Git repository layout¶

There are two possible structure possible for the repository layout:

The exploded SRPM layout
The flat dist-git layout

The exploded SRPM layout¶

This is the "traditional" layout used in CentOS Linux, it is articulated around two folders:

├── SOURCES
└── SPECS

The SPECS folder is meant to receive your spec file and all other text files and patches would go under SOURCES:

├── SOURCES
│   ├── <optional_file>
│   └── <optional_patch>
└── SPECS
    └── <pkg_name>.spec

Once done, we still need to link these files with the sources uploaded earlier in the lookaside cache. See the next section.

The flat dist-git layout¶

This is the layout used in the dist-git repositories in CentOS Stream and Fedora. All files (spec files, text files, patches...) are stored at the top level of the repository:

├── <optional_file>
├── <optional_patch>
└── <pkg_name>.spec

Linking to sources in the lookaside cache¶

Here as well, there are two ways you can link a dist-git repository to sources uploaded to the lookaside cache:

the "traditional" way using a .<pkg_name>.metadata file
the CentOS Stream/Fedora way using a sources file

Important

Even if your package doesn't contain any source pushed to lookaside cache (like for a package just having some small files in SOURCES/ dir, you need to have either a .<pkg_name>.metadata or sources file present and pushed in git repository

Using a `.<pkg_name>.metadata` file¶

To use this method, simply create a .<pkg_name>.metadata file with the following content:

<sha1 of the tarball> SOURCES/<filename>

For example:

d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 SOURCES/centpkg-minimal.tar.gz

Using a `sources` file¶

To use this method, simply create a sources file. The easiest way to achieve this is:

sha512sum --tag <tarball> > sources

Example output:

SHA512 (centpkg-minimal-1.1.0.tar.gz) = f7cfbc956199b1a0342f321a0e4cb055d6ac2c784a8faace43cf80772f0de34b58ede1a02a558cd03e25c14938ac6e17b3746b1919c0bbc1f5b2955472577e4f

Now that we have pointer to lookaside cache, and also .spec, we can push back to git and we should be able to proceed with the "build-from-git" on cbs.centos.org. Let's commit first.

Git push¶

Depending on if you picked the "traditional" structure or the CentOS Stream/Fedora one, this is how your repository should look like:

Traditional layout

.
├── .<pkg_name>.metadata
├── SOURCES
│   ├── <optional_file>
│   └── <optional_patch>
└── SPECS
    └── <pkg_name>.spec

CentOS Stream/Fedora layout

.
├── sources
├── <optional_file>
├── <optional_patch>
└── <pkg_name>.spec

We can now push to to git, as usual:

# git add <files> # if needed
git commit -a
git push origin <branch> # to create the remote branch if not existing yet

Where <branch> would be c7-sig-core for our example using git.centos.org.

Now that we have our sources pushed to both git.centos.org and lookaside cache, we can now proceed with a build in cbs/koji.