ExoticSilicon.com - Jay explains the OpenBSD distributed ports builder

Reckless Guide

Part 1

In the first installment of our new series with Jay Eptinxa, we'll look at compiling software in the ports tree from source, and making our own binary packages.

After a basic introduction to getting dpb set up and working, Jay will show us some useful techniques to make our bulk building run as smoothly and efficiently as possible.

This article is part of a series - check out the index.

Website themes

The Exotic Silicon website is available in ten themes, but you haven't chosen one yet!

Answering the obvious fundamental question

“What if any advantages are there to building ports from source, when binary packages are available for the popular architectures?”

After all, even the official OpenBSD documentation recommends the use of the pre-built binary packages over building from source.

Indeed, for many use cases the pre-built binary packages are sufficient and especially for new and casual OpenBSD users they provide a way to get the job done with very little preamble.

Nevertheless, there are definite advantages to setting up and using dpb to build your own packages from source...

Reasons we might want to build ports from source

To edit the source code and make our own changes to it
To apply patches that are not already included in the OpenBSD port, or use different compiler options
To bulid packages for an architecture that doesn't have pre-built binary packages available
To learn and understand the ports building process
To reduce inter-package dependencies and overall requirements on constrained systems

As well as these common reasons, there are also numerous other instances where building from source is useful.

Some ports have not been updated for a very long time, three examples being archivers/bzip2, sysutils/dvd+rw-tools, and sysutils/cdrtools. Using the pre-built binary packages, you'd have needed to download a new package for each successive OpenBSD release, whereas working with the source code, you would have already had many of the source files to hand.

But why use dpb?

If you've set up the ports tree in /usr/ports, and have a working internet connection, then you can build any particular port by doing a simple make in it's directory, something like this:

# cd /usr/ports/archivers/bzip2

# make install

It seems so easy this way, but...

Unfortunately, whilst this method is simple and might appeal to new users without much experience of OpenBSD, it does has some severe drawbacks:

Drawbacks of just running make in the ports tree

By default, the entire build process runs as root
Each port's dependencies will be built sequentially
Only one CPU core will be used

The sequential building of each package combined with the use of only one CPU core often results in very long build times for complex ports with a lot of dependencies, time during which the CPU remains mostly idle if the machine isn't busy with other tasks.

It is possible to set up this method of building ports such that it runs as an unprivileged user, however if you're going to go to the trouble of doing this, you might just as well set up dpb anyway.

Fun fact!

Remembering ‘systrace’

An interesting anecdote for the curious: many years ago it was also possible to use systrace to reduce the risk of a malicious port doing something undesirable to your system. This greatly increased build times, which served as a fairly strong discouragement to it's use. Systrace was finally removed from OpenBSD during the 6.0 development cycle.

A few ports might successfully build using multiple cpu cores if invoked with a suitable -j parameter. Many will not.

Obviously there has to be a better way to build software from source, and of course there is, dpb.

The wonders of dpb

Parallel building of multiple ports using several CPU cores
Building process automatically performed as an unprivileged user
Optional privilege separation in the form of a chroot
Build logs kept in a central location
It's easier to fetch all required source distfiles first, for a later off-line build

With dpb we get all of these benefits, (and more!)

Whilst the name, distributed ports builder, might suggest that it's only intended or beneficial for those users who are running a compile farm with multiple machines on a LAN, in fact, dpb is a much better way to do just about any work involving the ports tree.

If you intend to do any serious work with ports, the initial investment in time and effort involved in learning to use dpb will almost certainly be outweighed by the added convenience and functionality gained.

So let's make a start setting it up!

Prerequisites for setting up dpb - extracting the ports tree

I'm going to assume that you already have the ports tree extracted into /usr/ports. If not, you probably want to do something like this:

# cd /usr/ports

# tar -xvzf /install_path/ports.tar.gz

Extracting the ports tree

Of course, you should have also already checked the integrity of the source archive with signify before un-tarring it:

# signify -Cp /etc/signify/openbsd-XX-base.pub -x SHA256.sig ports.tar.gz

Checking the integrity of the source archive using signify

Due to limits imposed by the version of tar in the base system on the length of file paths, some of the files will be missing if you use this method of populating /usr/ports.

Unless you actually want to build the few affected ports, this probably doesn't matter. If you do want to be sure of having a full copy of the ports tree, you can use cvs to fetch the missing files. If you're going to do this, note that it's likely still quicker to obtain most of the files from the tar archive and then use cvs just to update those that are missing, rather than do a cvs checkout of the full ports tree.

Since we're going to set up a chroot environment for the actual building processes, /usr/ports is basically only going to act as an initial source for populating the chroot, (and re-populating it if we ever have problems and want to start over). For this reason, /usr/ports doesn't need to be on it's own partition, and we don't need to extract the tar archive using the -p option to preserve the file attributes, (although doing so won't do any harm).

All of these instructions assume that you're using a -release version of OpenBSD. If you're running a -current system then you'll definitely want to keep your ports tree up to date using CVS and re-populate the chroot at regular intervals.

Prerequisites for setting up dpb - partitions for the chroot

Whilst /usr/ports doesn't need it's own partition, it is beneficial to create one or more partitions for the chroot that dpb will run in.

Within the chroot environment, two important paths are /usr/ports/distfiles, which is where the actual sourcecode archives will be stored, and /usr/ports/pobj, which is where they will be extracted and where the compilation will actually take place.

Since the /dev/ directory within the chroot needs to contain device special files, if you decide to create just a single partition for the chroot, it will need to be mounted with the dev option, (as opposed to the default nodev). It will also probably require the wxallowed flag, unless you happen to be compiling exclusively well-behaved ports which don't need it.

Alternatively, we can create separate partitions for the distfiles and pobj directories. Doing so has various benefits:

Handy hint

Advantages of separate partitions for distfiles/ and pobj/

We can mount the chrooted /usr/ports/distfiles and /usr/ports/pobj using the nodev flag, using dev only on the main chroot partition
We can restrict the use of the wxallowed flag to the pobj partition
We can optionally mount the pobj partition using the async flag for better performance
Alternatively, if we have sufficient physical RAM, we can put pobj on a ramdisk filesystem for even better performance
We can mount the distfiles partition read-only for times when we don't expect to, and don't want to, be downloading additional source files
We can re-format and re-populate the main chroot partition without having to backup and restore, or re-download the distfiles
It's easier to increase the size of the distfiles or pobj partitions at a later date, by deleting and re-creating them

The amount of disk space required for each partition obviously depends on exactly which ports you want to build. The amount of space required on pobj will also depend to a degree on how many ports you intend to build in parallel, since this requires more source files to be extracted at the same time.

Here is a table summarising the partitioning requirements, with some rough ballpark values for the sizes:

Partition	Typical size	User and group ownership	Mount options	Use
/proot	3 - 6 Gb	root, wheel	dev	The main chroot partition
/proot/usr/ports/distfiles	2 - 25 Gb	_pfetch, _pfetch		The source distribution files
/proot/usr/ports/pobj	5 - 10 Gb	_pbuild, _pbuild	wxallowed [1], async [2]	The actual build directory

Table footnotes

[1] The wxallowed flag is not required for all ports, however some commonly used ones do require it. If you are only building a subset of ports which do not, then you can successfully mount pobj without the wxallowed flag.

[2] Since the pobj directory is effectively just a temporary working directory which can be easily re-populated if it's erased, the main risks of using the async flag, namely data corruption in the event of power-loss or a kernel panic, are somewhat mitigated.

Once again, your own requirements for the partition sizes might be wildly different to those noted in the table, especially the distfiles partition.

Since the main /proot partition is the easiest to size, and the contents of the pobj directory can be erased between builds, by placing the partitions on the disk in the order that they appear in the table above it remains quite easy to re-partition and increase the space allocated to the distfiles and pobj partitions at a later date if this ever becomes necessary.

pobj on a ramdisk

If you have 32 Gb or more physical RAM in the build machine, you might want to create an mfs-backed ramdisk partition for the pobj/ directory:

# mount_mfs -s 12g swap /proot/usr/ports/pobj

Mounting an mfs ramdisk on the pobj/ directory

In our tests, performance was noticeably better using a ram-based mfs filesystem, even when compared to a reasonably fast SSD. However, this is probably going to be highly dependent on your specific hardware, so if performance is important to you then doing your own benchmarks is key.

Even if you don't have enough physical RAM to compile large ports in this way, it remains an option for lighter ports building sessions, so you might decide to create a large disk-based pobj partition, then mount a smaller mfs filesystem over it when you know that you are only going to be compiling a few small ports. Once again, experimentation is the key to getting best performance.

In the table above and in the following examples, we've used /proot as the location for the root of the choot. However you can choose just about any location for this.

With the relevant partitions created, newfs'ed, added to /etc/fstab with appropriate flags, and mounted, we're ready to move on to the next step.

Populating the chroot directory

In contrast to the normal approach of populating a chroot environment, in which you ideally want the fewest possible files present, for ports development we basically need to create a full copy of the base system.

Thankfully, there is a utility called ‘proot’ already in the base system specifically to do this:

# /usr/ports/infrastructure/bin/proot -B /proot/ actions=copy_ports

Using proot to populate the ports chroot directory

The above command will populate the chroot directory with the files that dpb needs from the base installation, as well as the contents of the ports tree from /usr/ports.

If it returns any errors, correct them before continuing.

Remember that any changes you make to configuration files in /etc, or anything in else in the base system after populating the chroot directory, will not automatically be applied to the chroot environment. So, for example, if you update the termcap database and change your default termtype, you would likely want to manually copy the changed files to the corresponding locations in /proot/. Alternatively, you could erase the whole partition with newfs and re-run the proot command above without losing the distfiles, although you would lose any packages that you had already compiled, along with their compilation logfiles, unless you backed them up first or created a fourth partition for /proot/usr/ports/packages/.

At this point, dpb is basically ready to use.

Invoking dpb

The dpb utility resides in /usr/ports/infrastructure/bin/dpb, which isn't in the shell's search path by default. We could just modify the search path, but most of the other programs in that location will be rarely used for simple building of existing ports, so to avoid them being matched when using tab-completion of filenames I prefer to simply add a shell alias for dpb. This has the advantage that we can also automatically specify two useful command line options.

# alias dpb="/usr/ports/infrastructure/bin/dpb -B /proot -D BUILD_USER=_pbuild"

Setting an alias for dpb using the korn shell

The -B option simply enables the chroot and specifies it's location. Invoked without this option, dpb would operate using the normal, non-chroot'ed, /usr/ports directory hierarchy. The -D command line argument allows us to specify various options, (detailed in the dpb manual page), but the one we are using here, BUILD_USER, ensures that as little as possible is actually run as root.

We can quickly test that dpb is working correctly by building a very small port that doesn't have any dependencies:

# dpb audio/metronome

Building a very small port to test dpb

If you don't already have the single distfile containing the source code for metronome in /proot/usr/ports/distfiles/, then dpb will download it. Next it will proceed to checksum, extract, configure, and build the port.

In the case of audio/metronome, this process will be almost instantaneous on any vaguely modern machine.

Once dpb has finished running, you'll find various log files in an architecture-specific directory in /proot/usr/ports/logs/. For example, if you are using the amd64 architecture, then you'll find the following in /proot/usr/ports/logs/amd64/:

DPB LOGFILES

# ls

awaiting-locks.log

build.log

built-packages.log

concurrent.log

debug.log

dump.log

engine.log

equiv.log

localhost.sig.log

locks/

packages/

paths/

performance.log

stats.log

summary.log

term-report.log

vars.log

That's fourteen ‘global’ log files, two directories, (packages, and paths), containing logfiles specific to each port that has been built, and one directory, (locks), which contains lockfiles, to avoid multiple instances of dpb working on the same port simultaneously.

If dpb actually fetched the distfile from a remote server, because it wasn't already present in the distfiles/ directory, you'll have two additional entries in /proot/usr/ports/logs/amd64/. The fetch/ directory contains two global logs detailing successful and unsuccessful download attempts, (good.log, and bad.log). The dist/ directory contains individual logs for each port, with slightly more verbose information, such as the actual invocation of the ftp command used for the download attempt, and the error message, if any, that it returned. You can use these logs to diagnose connectivity issues to particular mirror sites.

The log files in the packages/ directory are actually just symbolic links to those in the paths/ directory, so you can easy find the correct log given either the original path of the port, (such as audio/metronome in this example), or name of the resulting package, (metronome-2.log). If you experience a build failure for a specific port, these are probably the log files that you want to look at.

Dealing with build failures

If the build of a particular port fails due to an error, it will usually leave behind a lock file in /proot/usr/ports/logs/amd64/locks/. In the case of a distfile being unavailable, the name of the lock file will be the same as that of the file that dpb was trying to download to, which in our example would be metronome-2.tar.gz.dist.

If dpb is invoked again, this lock file will automatically be cleared and then re-created if the distfile remains unavailable. However, in the case of other errors, such as a required shared library being missing, the lock file that dpb leaves behind will be named according to the port path, with forward slashes replaced by periods. So, again, in the case of our example, we might see a file audio.metronome. This lock file will not be cleared automatically, and future invocations of dpb to compile the same port will error out almost immediately because of it.

Removing stale lock files is easy enough:

# rm -r /proot/usr/ports/logs/amd64/locks/*

Removing stale lockfiles

For interactive use of dpb on a single machine, you might even want to include this lock removal in the dpb alias we defined earlier:

# alias dpb="rm -r /proot/usr/ports/logs/amd64/locks/* && usr/ports/infrastructure/bin/dpb -B /proot -D BUILD_USER=_pbuild"

Updating our earlier shell alias to include removal of stale lockfiles before the invocation of dpb

Installing the compiled packages

Of course, compiling a port just creates a binary package, it doesn't automatically install it. Build dependencies of the port are installed automatically, but only within the chroot'ed environment, so don't be surprised when they are not listed if you run pkg_info normally, outside of the chroot.

The final binary packages are created in the /proot/usr/ports/packages/amd64/ directory hierarchy. In our case we will have four subdirectories, all/, ftp/, no-arch/, and tmp/. All of the packages can be found in the all/ subdirectory, whilst ftp/ and no-arch/ contain copies of some of these packages in the form of hard links.

Since the compiled packages are, by default, not cryptographically signed, we need to set the TRUSTED_PKG_PATH environment variable to enable pkg_add to install them:

# export TRUSTED_PKG_PATH=/proot/usr/ports/packages/amd64/all

Setting a trusted path from which to install unsigned packages

We can then install our newly compiled binary package with pkg_add:

# pkg_add metronome

Installing our newly compiled binary package

Signing locally built packages

We can sign packages with our own private key, and by doing this we eliminate the need to set TRUSTED_PKG_PATH on any machine that has the corresponding public key in /etc/signify/.

Packages are signed using the signify utility that has been in the base system since OpenBSD 5.5.

First we need to generate a public/private key pair:

# signify -G -p local_packages.pub -s local_packages.sec

Generating a new public/private key pair with signify

Then we copy the public key to /etc/signify/ on the local machine, and any other machines that we will be installing the packages on:

# cp local_packages.pub /etc/signify/

# scp local_packages.pub root@remote.machine:/etc/signify/

Installing the new public key in the relevant places

Finally, to actually sign a binary package that we have already built, we use signify with the -S and -z options:

# mkdir signed_packages

# signify -S -z -s local_packages.sec -m /portsroot/usr/ports/packages/amd64/all/metronome-2.tgz -x signed_packages/metronome-2.tgz

Signing a locally produced binary package

You might be wondering how the signature information is actually stored in the package file, as binary packages are simply tgz files.

Luckily, the gzip format has an option to store a free-form, zero-terminated comment, and signify uses this to embed the signature. In this way, the tgz files can still conform to the gzip format, and can be decompressed manually with normal programs if required.

Important

Be aware that the full path specified for the secret key in the -s argument to signify, is embedded in the signed package's gzip header. This potentially leaks information about valid file paths on the signing machine, and possibly even who signed the package, to anybody who has access to a copy of it.

For example, observe the following hexdump showing the gzip header at the start of a signed binary package, and the command used to sign it. The hexdump clearly reveals that the directory /home/jay/ exists on the signing machine, and implies that a user named jay signed the package:

Potential for data leakage when signing packages

00000000 1f 8b 08 10 00 00 00 00 00 03 75 6e 74 72 75 73 |..........untrus|

00000010 74 65 64 20 63 6f 6d 6d 65 6e 74 3a 20 76 65 72 |ted comment: ver|

00000020 69 66 79 20 77 69 74 68 20 6c 6f 63 61 6c 5f 70 |ify with local_p|

00000030 61 63 6b 61 67 65 73 2e 70 75 62 0a 52 57 53 6c |ackages.pub.RWSl|

00000040 6a 6b 54 4a 64 73 51 46 38 59 6a 48 51 2f 4f 4f |jkTJdsQF8YjHQ/OO|

00000050 2b 65 49 5a 74 47 4c 31 45 78 63 48 35 74 36 55 |+eIZtGL1ExcH5t6U|

00000060 67 65 44 4d 76 56 42 36 39 6d 67 45 6e 4e 39 42 |geDMvVB69mgEnN9B|

00000070 41 45 31 68 6b 49 35 6b 73 67 4a 68 75 63 6e 4f |AE1hkI5ksgJhucnO|

00000080 71 43 35 41 4b 71 53 2f 64 61 54 6e 30 6e 63 72 |qC5AKqS/daTn0ncr|

00000090 38 50 74 44 49 4e 6d 76 70 63 68 7a 65 41 30 3d |8PtDINmvpchzeA0=|

000000a0 0a 64 61 74 65 3d 31 39 37 30 2d 30 31 2d 30 31 |.date=1970-01-01|

000000b0 54 30 30 3a 30 30 3a 30 30 5a 0a 6b 65 79 3d 2f |T00:00:00Z.key=/|

000000c0 68 6f 6d 65 2f 6a 61 79 2f 6c 6f 63 61 6c 5f 70 |home/jay/local_p|

000000d0 61 63 6b 61 67 65 73 2e 73 65 63 0a 61 6c 67 6f |ackages.sec.algo|

000000e0 72 69 74 68 6d 3d 53 48 41 35 31 32 2f 32 35 36 |rithm=SHA512/256|

000000f0 0a 62 6c 6f 63 6b 73 69 7a 65 3d 36 35 35 33 36 |.blocksize=65536|

00000100 0a 0a 39 63 63 63 35 63 38 64 31 65 32 36 36 62 |..9ccc5c8d1e266b|

00000110 36 61 37 39 31 35 35 33 33 62 36 38 62 30 35 62 |6a7915533b68b05b|

00000120 35 63 62 37 34 34 64 62 35 33 64 66 34 34 63 31 |5cb744db53df44c1|

00000130 34 31 31 61 38 66 34 37 34 62 38 64 39 35 30 37 |411a8f474b8d9507|

00000140 34 61 0a 00 |4a.. |

# signify -S -z -n -s /home/jay/local_packages.sec -m /portsroot/usr/ports/packages/amd64/all/metronome-2.tgz -x signed_packages/metronome-2.tgz

The value following 'key=' is not actually used or required by pkg_add, but it is part of the data that has been signed. If we modify it with a hex editor in an existing signed package, then the package will fail signature verification. However, if we really want to create signed packages without this field, we can simply modify the zsign function in /usr/src/usr.bin/signify/zsig.c, which is where the field is inserted. Alternatively, we could just use a relative path when specifying the signing key.

Notice that we also used the -n option in the example above, to set the timestamp in the signature to 1970-01-01, instead of using the current system time. The signature timestamp field can also be completely suppressed by editing the same zsign function in zsig.c.

Forcing a clean build

If you're modifying the original source of a port by applying your own patches to it, then you'll want to re-build it from scratch. To do this, you'll probably want to use pkg_delete to remove the installed binary package first:

# pkg_delete metronome

Un-installing a binary package before compiling a newly modified version of it

Next, we can delete the package from each directory in /proot/usr/ports/packages/amd64/:

# rm /proot/usr/port/packages/amd64/*/metronome-2.tgz

Deleting the compiled binary package

Then remove the old package-specific logfiles. This isn't strictly required, but if we don't then the new logfile data will be appended to them next time the port is compiled:

# rm /portsroot/usr/ports/logs/amd64/dist/metronome-2.tar.gz.log

# rm /portsroot/usr/ports/logs/amd64/paths/audio/metronome.log

# rm /portsroot/usr/ports/logs/amd64/packages/metronome-2.log

Deleting old compilation logfiles

Finally we remove the plist entry:

# rm /portsroot/usr/ports/plist/amd64/metronome-2

After these steps, you can start a fresh compile with your modified makefile, additional local patches, or whatever other changes you've made to the port.

Advanced dpb techniques

Now that we've covered the basics of setting up dpb in a chroot environment and building a simple port, let's look at some tricks and techniques that might be useful for non-standard ports building configurations.

Pre-downloading distfiles for off-line building

Whilst it might seem convenient to be able to invoke dpb with a single command, and have it automatically fetch the source code distribution archives for several large ports along with perhaps tens or even hundreds of dependencies over the internet, this assumes that a couple of things are true.

Firstly, that the build machine is connected to a reliable, un-metered, and reasonably fast internet connection, and secondly that you have either already checked what the dependencies of each port are, or alternatively don't care what software is automatically installed on the target machines.

Whilst a cheap, fast internet connection can be taken for granted by many users in 2022, there are still many locations worldwide where internet connectivity is severely limited. In the case of a metered connection, such as a constrained cellular data plan, ensuring that you don't download un-needed distfiles and avoiding re-downloading distfiles that you already have can considerably reduce bandwidth usage.

In these and other cases, it's useful to be able to download the relevant files to prepare a suitably populated distfiles/ directory beforehand, then start the build on a machine with no access to the internet.

This also ensures that only the software you have already downloaded can be packaged and installed, so there will be no surprises with an updated port suddenly pulling in a multitude of dependencies that you're not familiar with, and might not be happy having installed on your machines.

To fetch the required files for a particular port or set of ports, we simply invoke dpb with the -F argument to indicate how many downloads can be started in parallel:

# dpb -F 5 archivers

Fetching the source for various ports, but not compiling them

This invocation of dpb would download all of the required distfiles to build every port in the archivers category. Note that this includes fetching distfiles for all of the different flavours of each port, and even ports which don't build on the architecture of the build machine. This is useful for building a local copy of all of the distfiles for the ports tree in any particular OpenBSD release. If this is not what you want, then you can be more specific in the path list supplied to dpb.

Handy hint

Preventing particular distfiles from being downloaded

If you want to prevent a particular distfile from being downloaded automatically, the easiest way is create a directory with the same name as the distfile but with '.part' appended to it.

This can be useful to avoid attempts to download large distfiles which you know that you won't need.

An example of where this might happen is when a particular port uses different versions of the C compiler depending on which architecture it's being built on.

The files in distfiles/ should be owned by user and group _pfetch. Dpb will do this automatically for files that it downloads, but if you add files manually to the distfiles/ directory, you will probably need to chown them manually as well. The only exception to this is the build-stats/ directory, which should be owned by user and group _pbuild.

Seeding distfiles/ with distfiles/ from a previous release

Assuming that you upgrade from one OpenBSD release to the next by re-installing, rather than trying to upgrade the system in-situ, you'll need to re-compile all of your ports after the upgrade. Since some, or even many, of the distfiles won't have changed between the two releases, you can save some time and bandwidth by initially copying the contents of the old distfiles directory to the new installation.

Then you can either allow dpb to fetch the remaining required distfiles automatically during the builds, or alternatively fetch them in advance by using the -F option. This will leave behind some old distfiles that were required for the previous release, but are no longer required for the new release, usually due to being replaced by newer versions.

A utility to identify and remove such un-needed distfiles is included in the base installation in /usr/ports/infrastructure/bin/clean-old-distfiles.

It relies on the history log created by dpb, so it's a good idea to make sure that that is up to date first:

# dpb -D HISTORY_ONLY

Updating the history log file in preparation for running clean-old-distfiles

This is particularly important if you've been manually copying distfiles around, or otherwise making local changes to the ports tree.

Since the clean-old-distfiles utility doesn't automatically operate within the chroot, we can either invoke it as:

# chroot /proot /usr/ports/infrastructure/bin/clean-old-distfiles -n -v

Running clean-old-distfiles within the chroot environment

Or alternatively we can just set PORTSDIR to the chroot directory:

# export PORTSDIR=/proot

# /usr/ports/infrastructure/bin/clean-old-distfiles -n -v

# unset PORTSDIR

Running clean-old-distfiles outside the chroot environment, but operating on the chroot directory

We invoke clean-old-distfiles with the -n and -v flags to see what would actually be removed. It can then be re-run without the -n flag to actually remove those files.

Downloading distfiles for a new release before installation

Whilst many people may be satisfied with installing the base sytem first and then just letting dpb fetch whatever it likes, there are other approaches.

When upgrading by re-installation, it's useful to have all of the required distfiles for the new release to hand locally before starting the fresh install. This is especially true if you only have one local machine running OpenBSD, if your internet connection is unreliable, or if you want minimise downtime and be back up and running as soon as possible. It's also a good idea if you've made local changes to any of the makefiles, as it gives you an opportunity to make and check corresponding changes to the makefiles in the new ports tree.

There are various ways to approach this.

Firstly, if you're building a very small number of ports, with few or no dependencies that are unlikely to have changed since the last release, you could simply look through the makefiles and manually download any distfiles that have changed. However, this is tedious and error-prone for more than a handful of ports, so not really recommended.

Next, although not guaranteed to work, it's often possible to parse the ports tree from one release on the previous release. Although many or most of the ports won't actually build correctly, due to mis-matched library versions and other changes, it's usually possible to check the required dependencies and fetch the required distfiles using dpb with the -F flag.

One way to set this up is to create another new partition which will eventually contain a chroot environment based on your current base system, with the ports tree from the next release. This can either be mounted on /proot, in place of the existing /proot partition, or alternatively mounted elsewhere, such as /newproot, and the -B parameter of dpb changed to match.

If you don't intend to do any more ports building on the existing system before you upgrade, you could simply erase and re-use the existing /proot partition, although remember to copy off any log files and modified makefiles that you want to preserve first. If you have made local changes to the makefiles, it will be useful to have both the original makefiles from your existing ports tree, and your custom versions to hand, so that you can see the changes that you made and forward-port them to the versions in the new release.

To populate the new chroot directory, we'll need to extract the new ports.tar.gz file somewhere. Since we only need it as a source for /usr/ports/infrastructure/bin/proot to copy the ports tree from, we can simply mount an mfs ramdisk filesystem over /usr/ports/ so that the files are in the expected location for copying, but the exising contents of /usr/ports/ is not lost.

A typical session might look something like this:

Archive the distfiles we used for the current release:

# tar -cvf distfiles_for_this_release.tar /proot/usr/ports/distfiles/

Unmount the existing partitions:

# umount /proot/usr/ports/distfiles

# umount /proot/usr/ports/pobj

# umount /proot

Create a new partition, which for this example we will assume is sd1p:

# disklabel -E sd1

# newfs /dev/rsd1p

Mount the new partition:

# mount -odev /dev/sd1p /proot

Mount an mfs ramdisk filesystem on /usr/ports:

# mount_mfs -s 2g swap /usr/ports

Verify and extract the new ports.tar.gz:

# signify -Cp /etc/signify/openbsd-XX-base.pub -x SHA256.sig ports.tar.gz

# cd /usr/ports

# tar -xzf /new_install_path/ports.tar.gz

Populate the new chroot:

# /usr/ports/infrastructure/bin/proot -B /proot/ actions=copy_ports

Done!

Now we can start making any desired local changes to the new makefiles, then hopefully eventually invoke dpb with the -F flag and fetch the required distfiles for the new ports tree to distfiles/. This should be everything we'll need after a fresh installation of the new release, to build and install the ports we were previously using.

The above method is quite useful when working on a single machine. However, if you have a spare machine to dedicate to the task, a spare disk that you can swap in to an existing machine, or the possibility to set up a VM, it might be quicker and easier to simply do a minimal fresh installation of the new release, quickly set up dpb just to fetch the distfiles, and then copy them to removable media such as a flash drive or optical disc. That installation can then be wiped, and the real installation performed, knowing that you already have a good set of distfiles to hand. This approach of using an actual installation of the new release for the fetching of the distfiles, has the advantage that you can reliably test that the ports actually compile with any local modifications, too.

Doing a temporary installation just to download the required distfiles with dpb might seem pointless, but it has some advantages once you start to make a lot of local changes to the ports tree. Here at Exotic Silicon, we do run fairly heavily customised installations of OpenBSD, and with each new release somebody has to forward-port all of our local changes. This can range from being trivial and almost no work at all, to a major re-factoring where changes to the base system or ports tree conflict with our own. Where updates to a port change it's dependencies, we need to check that all of the newly required software meets our needs and expectations, and doesn't conflict with anything else. We also consider whether we're going to follow the change or replace the software entirely with something written in-house. This whole process can involve building and testing various pieces of software that we haven't used before, and testing various experimental configurations locally. After all of this, it's quicker and easier to copy the new distfiles and makefiles elsewhere and do a fresh installation of OpenBSD, than to go through every single change that we've made and check that we haven't in-advertently adjusted something that will cause problems later on.

Summary

This week, we've set up the Distributed Ports Builder, built a port from source, learnt how to sign our own packages, taken a glimpse at some possible information disclosure and how to avoid it, then finally looked at some ways to optimise our downloading and management of distfiles. What a start to our new series!

In a future series, we might see how to modify ports makefiles to make local customisations and reduce dependencies. Next week, however, we'll be looking at custom kernel configurations. See you there!

IN NEXT WEEK'S INSTALLMENT, WE'LL TAKE A BREAK FROM PORTS AND FOLLOW JAY AS HE COMPILES A CUSTOM KERNEL. DON'T MISS IT!