Date: 2021-09-11
So we want to reproduce Red Hat Enterprise Linux from the ground up. But how? Read on, and buckle up. Today we'll cover the 10,000 foot overview of what the "build steps" are: Where source comes from, how it gets to Rocky Linux, and how it gets (sometimes) patched, compiled, and produced as a Rocky package.
Rocky Linux, like RHEL, Fedora, CentOS Stream, etc. is an RPM-based distribution. This means that packages are stored in the RPM format and live in Yum/DNF repositories. Importantly, it also means that there are "source RPMs" (SRPMs) for each package. These SRPMs contain source code, patches, and a spec file "recipe" with instructions on how to compile that package.
Rocky Linux as a distribution is made up almost entirely of these RPMs. By and large, the business of creating Rocky mostly revolves around importing, compiling, and publishing these packages. There are more than 5000 of them in our base repositories alone, and they all must be built, maintained, tested, and released.
This isn't going to be an RPM tutorial, those are available elsewhere online. I'm assuming you are at least a little familiar with the basic concepts of RPM packages. For this article, the most important thing to know is the idea of creating source RPMs from source code, and then producing binary/final RPMs from those SRPMs.
Basic RPM compilation looks like:
Source Code (Git) ----> Source RPM (SRPM) ------> Binary RPM (final, goes in repository)
We'll list the broad steps here, then go into each one individually:
Import the source code for a RHEL package into Rocky's source control
When importing some packages, certain Rocky-specific patches need to be applied
Package must be compiled into an SRPM, then into a binary RPM, and internal tests run
New packages then get signed
Repository Compose: packages are transferred into the repository, according to a list
Repository metadata update and other tasks
dnf time!We'll explore each of these steps in a bit more detail below. To illustrate the concepts all the way through, I'll use Nginx (a popular web server package) as an example we can refer to.
I intend to write a follow-up article shortly after this one which dives a bit more into the actual software pieces that perform these steps, and how they do it. For now it's enough to know the basic tasks and why we're interested in doing them this way.
If we want to recompile RHEL, we just first obtain the sources for RHEL. Fortunately for us, the sources are well-organized and always available. They can be pulled from: https://git.centos.org/ (spec files, patches), and https://git.centos.org/sources/ (.tar source code archives).
In our Nginx example, the current version (as of this writing) is: 1.14.1-9. The RHEL source package is browseable here. We can tell from the ".nginx.metadata" file in that repo that the source tarball SHA hash is:a9dc8c5b055a3f0021d09c112d27422f45dd439c (for file nginx-1.14.1.tar.gz). That tarball (named after its hash) is downloadable at: https://git.centos.org/sources/nginx/c8-stream-1.14/. Rocky clones that repository and copies that source tar file into our own infrastructure.
The matching Rocky source repo for nginx-1.14 is here: https://git.rockylinux.org/staging/rpms/nginx/-/tree/r8-stream-1.14 and you can download the same source tarball from Rocky Linux here: https://rocky-linux-sources-staging.a1.rockylinux.org/a9dc8c5b055a3f0021d09c112d27422f45dd439c.
The Nginx package contains Red Hat logos and references in its sources, mostly for the default error and "welcome" HTML pages that are bundled with it. These must be replaced with Rocky Linux branding, we are NOT allowed to redistribute any trademarked material!
For packages that need it, Rocky hosts a per-package "patch/" repository at: https://git.rockylinux.org/staging/patch/. For example, the Nginx patch repository can be found at: https://git.rockylinux.org/staging/patch/nginx. Any package without a matching "patch/" repository gets imported 100% unaltered from its RHEL source.
About Automation: We obviously don't want to import these packages by hand, there are over 5000 of them! Rocky has a piece of in-house software, called srpmproc ( https://github.com/rocky-linux/srpmproc ), which helps automate this process. Srpmproc imports packages/sources, tags them appropriately, and applies Rocky patches if applicable.
Final note about sources: There is obviously much more to the source code / import process, this section has just skimmed the surface. Source code management, import, and patching will get its own article in this series soon, look for it!
Once the source has been imported, we can actually perform a build! RPMs are traditionally built with a program called rpmbuild. Rocky builds with this, but there are more tools and abstractions built around it to help us compile packages at scale.
The simple: Mock
Mock is a build wrapper tool. It will create a simple container/chroot environment and bootstrap a minimal system inside using DNF. It will install only what is needed to build the package inside, and then calls rpmbuild within this "clean" environment.
More complicated: Koji, MBS, Distrobuild
These are tools built to call Mock (and rpmbuild) against many packages, and organize the resulting RPM files.
Koji is the centralized build system of choice for Fedora-based distros.
Distrobuild is a web front-end tool built by the Rocky project to trigger Koji for builds, as well as call srpmproc to import RHEL sources.
MBS Is the Module Build Service, which is responsible for triggering Koji for modular stream builds. Modules and modular streams will get their very own article in this series later!
There's obviously much more depth to these tools than this short introduction, and they'll be covered in more detail in subsequent posts.
Live, Browseable Rocky Tools:
(Note: this png file has its diagram data embedded, and can be imported into https://app.diagrams.net/ for editing)
The build process flow of abstraction happens like this:
Rocky Release Engineers ---interacts with---> Distrobuild ---calls API---> Koji ---triggers---> Mock ---triggers---> rpmbuild ---triggers---> gcc/g++/automake/Cmake/Maven(Java)/etc
Once the packages have been compiled, they sit in several large DNF repositories held and organized by Koji. These packages are copied, submitted to Rocky's Sigul instance, which then signs it with the Rocky Linux private key. The freshly signed package is put in a special staging repository within Koji. Once signed, it should be ready for release.
The matching public signing key is distributed in every Rocky installation (you can find it on yours: /etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial). This makes it absolutely certain that the updates and packages you download from the Rocky repositories were in fact produced by the Rocky Linux project, and not forged or tampered with in any way! Your DNF package manager will refuse to proceed if it downloads an unsigned or mis-signed package from the official Rocky repositories.
Now that we have packages signed and ready, it's time to get them into the official Rocky repositories, and into the hands of users (and mirrors) around the world!
Publishing is done with a Fedora tool called Pungi. Pungi is configured to read a JSON "compose" file. The compose file is effectively a huge list of packages for Pungi to pluck out of Koji and place in the official Rocky repositories. It handles other compose tasks as well, such as generating the Rocky Linux ISO images.
The Rocky compose files are designed to match RHEL's exactly: we want the exact same packages without adding or removing anything. Maximum compatibility with RHEL is the #1 technical goal of the project (as per the Rocky mission statement and charter).
Example: Rocky Linux 8.4 compose: https://git.rockylinux.org/rocky/pungi-rocky/-/blob/r8/rocky-packages.json
There are a few final tasks to perform when a new/updated pacakge is pushed into the official Rocky Linux repository. These are:
I'm hoping to release a quick follow-up to this article, going through all the software tools (mock, koji, mbs, etc.) in a bit more depth.
Contact
Thanks for reading! If you have any questions about the material, or I guess questions in general, feel free to ping me: