UPDATE: (This article had a technical issue that has been fixed. I forgot to give instructions on modifying the Mock config to pull in non-default modules! Skip G., 2021-11-13)
Bind is the world's most popular DNS server, and we'll see that it's a bit harder to compile than bash. We'll find out first-hand about "hidden dependencies" and how we deal with them!
Note: this article assumes you have Mock and tools set up, and have completed the previous lab successfully (cloning and building Bash with Mock). If you haven't, then please do that first, or be familiar with how to do it! We'll be building on its concepts.
We begin our work very similar to the bash package above. First, clone the git repository for bind:
cd ~/rockysrc
git clone https://git.rockylinux.org/staging/rpms/bind
cd bind/
Next, We need to grab the binary source files, listed in that .bind.metadata file. There are 2 of them this time, which is a little different, but still easy:
[skip@RpiRockyDev bind]$ cat .bind.metadata
14064c865920842e48f444be2bda9dc91770e439 SOURCES/bind-9.11.26.tar.gz
a164fcad1d64d6b5fab5034928cb7260f1fa8fdd SOURCES/random.data
curl -o SOURCES/bind-9.11.26.tar.gz https://rocky-linux-sources-staging.a1.rockylinux.org/14064c865920842e48f444be2bda9dc91770e439
curl -o SOURCES/random.data https://rocky-linux-sources-staging.a1.rockylinux.org/a164fcad1d64d6b5fab5034928cb7260f1fa8fdd
Now, let's build our SRPM file from these sources. Again, works just like last time:
mkdir ~/rockybuild/bind
mock -v --resultdir=~/rockybuild/bind --buildsrpm --spec=~/rockysrc/bind/SPECS/bind.spec --sources=~/rockysrc/bind/SOURCES/
Tricky part: Let's attempt a build and see what happens!
time mock -v --resultdir=~/rockybuild/bind ~/rockybuild/bind/bind-9.11.26-4.el8.src.rpm
You'll notice that this build command errors out! Ruh Roh Raggy! We'll have to put on our Troubleshooting Pants to fix it:
The sharp-eyed among you will notice this dnf error message from your mock output:
No matching package to install: 'kyua'
As I explained in the previous article, not all build-time dependencies are included in the Rocky / RHEL 8 repositories, and this is one of them! We need this kyua package to proceed!
Fortunately, all dependencies are located in git, including kyua. So we can roll up our sleeves and compile it! First, the source code and SRPM generation:
git clone https://git.rockylinux.org/staging/rpms/kyua
[skip@RpiRockyDev kyua]$ cat .kyua.metadata
2e437ffdb96dfc6325e283e531a1a3e11bebe4d7 SOURCES/kyua-0.13.tar.gz
# Note that kyua has no SOURCES/ folder included, because there are no patches in the repository. That's ok, we'll just create one before downloading the tar.gz file:
mkdir SOURCES
curl -o SOURCES/kyua-0.13.tar.gz https://rocky-linux-sources-staging.a1.rockylinux.org/2e437ffdb96dfc6325e283e531a1a3e11bebe4d7
mkdir ~/rockybuild/kyua
mock -v --resultdir=~/rockybuild/kyua --buildsrpm --spec=~/rockysrc/kyua/SPECS/kyua.spec --sources=~/rockysrc/kyua/SOURCES/
Then, the compile step with Mock:
time mock -v --resultdir=~/rockybuild/kyua ~/rockybuild/kyua/kyua-0.13-1.el8.src.rpm
More Trouble : Another failure when trying to build kyua! This time we get:
No matching package to install: 'pkgconfig(lutok) >= 0.4'
Uh oh....
I'm going to save us a little time and "give away the ending" to this rabbit hole we've been tumbling down. The full dependency chain we need to compile bind looks like this:
bind ---depends on ---> kyua ----depends on ---> lutok ---depends on ----> atf
Only the final product, bind, is in the Rocky/RHEL repositories. The rest must be fetched and compiled from source.
But here's a conundrum: what do we do with these packages, even after we compile them? Let's say we successfully produce the atf RPM packages, for example. When we go to compile the next in the chain, lutok, that mock command won't automatically be able to see our atf packages and pull them in as dependencies! We must make it available somehow via DNF repo.
The answer is simple: we have to create our own (local) DNF repository! It's easier than you might think:
# We'll put our local repository in our home folder, under local_mock/ :
mkdir ~/local_mock
createrepo -v ~/local_mock
# (you installed "createrepo" when you installed mock. Did you notice? ;-) )
Now that we have a repository (even though it's empty), we need to edit our mock config to include it in builds. Append this option to /etc/mock/default.cfg:
# Be sure to swap USERNAME with your own user!
# (or just specify anywhere on the filesystem you want to put your repo)
config_opts['dnf.conf'] += """
[local_mock]
baseurl=file:///home/USERNAME/local_mock
gpgcheck=0
enabled=1
"""
Now we're ready! As we build dependencies, we can simply copy (cp) the produced RPMs to our repo folder, and run createrepo -v ~/local_mock. The next build should then "see" the dependencies we built. We are maintaining our own little repository!
Skip's Note: We could easily do this with a web server as well. Simply run the createrepo command pointing to a folder being served by your apache/nginx/whatever, and make the baseurl line start with http:// instead. You can share repositories with others this way. Do this enough, and next thing you know you'll have created your own Linux distro ;-) .
I've cloned my source into ~/rockysrc/atf , build the SRPM, and then built the binary RPMs into ~/rockybuild/atf. Adding the products to my local repo is simple. I copy RPMs, and run createrepo:
cp ~/rockybuild/atf/*.rpm ~/local_mock/
createrepo -v ~/local_mock
Now the next build in the chain (lutok) can "see" the results of this build! It won't fail, because I've given it atf package powers(!)
We rinse and repeat, following our chain all the way up to bind. We build lutok, copy those RPMs, run createrepo. Then build kyua, copy those RPMs, and again run createrepo. Now that we have all the RPMs from the atf -> lutok -> kyua "chain" in our repository, we are (finally) ready to build bind. We've collected every dependency that we need!
Once we get to bind, if you try it, we'll uncover yet another issue:
Problem: conflicting requests
package softhsm-2.6.0-5.module+el8.4.0+429+6bd33fea.x86_64 is filtered out by modular filtering
Doh! We'll fix this, and it will act as a kind of "preview" for when modules are covered on this blog.
Basically, the issue is this: Bind requires a certain package: "softhsm" to build successfully. Softhsm is available in Rocky (and RHEL), but not as part of a "default" module. It is part of the idm module, specifically the DL1 stream for that module. We can check this with dnf commands ("dnf module list idm")
[skip@RpiRockyDev ~]$ dnf module list idm
Last metadata expiration check: 0:00:19 ago on Sun 14 Nov 2021 11:32:12 PM EST.
Rocky Linux 8 - AppStream
Name Stream Profiles Summary
idm DL1 adtrust, client, common [d], dns, server The Red Hat Enterprise Linux Identity Management system module
idm client [d] common [d] RHEL IdM long term support client module
Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled
We see that the "idm:client" module stream is the default, but idm:DL1 is another option. Let's check the packages available in each one:
dnf module info idm
This will spit out a list of all the versions of these modules ever released, as well as which packages are included. If you scroll to some of the idm:DL1 entries, you'll see "softhsm" packages listed. These are not available in the default idm:client entries.
The solution is to tell our Mock config that we desire the idm:DL1, and NOT the default idm:client module enabled when we build Bind. Fortunately, there's an easy way to do this. Append this text to the end of your /etc/mock/default.cfg:
# This is used for enabling a proper non-default module for compiling bind
config_opts['module_setup_commands'] = [
('disable', 'idm'),
('enable', 'idm:DL1'),
]
Here we are telling Mock to first disable the idm module, and enable the non-default "idm:DL1" stream. Now the DNF inside the Mock build can "see" all the packages in idm:DL1. And your build can now pull in softhsm as a dependency.
Give bind a try again, and it should build this time! I recommend commenting out the those module_setup_commands by default in future builds though, and only stray from default modules when you need to. We'll dive more into modules later, but this gives you a way to enable/disable them in your builds, and to explore which packages are a part of each one.
Hope you learned something about package build chains and dependency management from this article. I'm encouraging engagement - feel free to chat with questions, tips, comments, etc. in the Reddit thread (under r/RockyLinux/), forums.rockylinux.org, or especially chat.rockylinux.org. I'm around decently often, and am usually down to chat about this stuff!
Next article, we'll leave the lab, but enter the factory! We'll take a (public) tour around the tools we use to trigger and manage builds at scale, mostly Koji and Distrobuild. We'll see what they're all about, and check out how much build activity is publicly available for all to see (hint: it's a bunch!).
As always, thanks for reading!
-Skip