The OpenBSD installation process is one of the fastest and most convenient of any modern operating system.
The base system install is easily completable in 15 minutes on any modern amd64 machine, even less time if you're doing a scripted installation and fetching the sources from a fast local server.
In reality, the pain of re-installing afresh is more likely to be felt in the process of re-configuring your system to be the same, or as near as possible, to how it was before.
Re-creating all of those tweaks to obscure configuration files. Discovering that you forgot to back up some hidden dot files in your home directory. Finding out that an on-line guide you followed a few releases back when you first installed, either no longer works or has been taken down. All of these situations, and more, can potentially cause hours of unwanted downtime and hassle. Especially so if they only become apparent after you've wiped your existing installation.
Moving by more than one release via the upgrade path is also not very convenient, as skipping intermediate releases is by no means guaranteed to be successful.
Downgrading is certainly not supported, and even if everything does seem to work, the process will likely leave files behind that wouldn't exist on a fresh installation. Whilst the system itself might not care about old libraries laying around, they might be a source of wasted time for any inexperienced system administrator trying to debug issues later on.
So wouldn't you feel much better being confident that you could re-install everything from scratch and get your system back up and running in a couple of hours, even if disaster strikes and a power surge destroys your main hard disk? Or maybe if you accidentally dd junk data over the wrong partition? A full re-installation might even become un-avoidable if you try to perform an update in situ using /usr/sbin/sysupgrade, and it fails spectacularly leaving you with a broken system that you don't have the knowledge to fix manually.
Ideally, you shouldn't ever be in a situation where re-creating your setup after a complete re-install would be an uphill struggle. And if it's not, then upgrading, (or downgrading), suddenly becomes very easy indeed, because we just use the same procedure.
By performing a fresh installation, we also have an easy opportunity to re-partition the root disk, move any legacy FFS-1 partitions to FFS-2, add or remove full disk encryption, perform a secerase on the root disk, and do countless other things that can't easily be done with a simple invocation of sysupgrade.
We can also move from any release to any other release, and completely eliminate any possibility that sysupgrade itself will fail and leave us unable to boot.
In short - a fresh install gives us a lot of freedom.
In principle, all we need to do is backup our existing configuration and data, re-install, and lastly restore the data, updating any configuration files to match the requirements of the new system.
Once we've taken the decision to perform a backup, re-install, and restore cycle, there are two main ways that we can go about doing this. Either we can put the existing installation to one side and use that as our backup, installing afresh on to a new physical hard disk, or alternatively we can copy the data elsewhere and do the new installation to the same physical disk, overwriting our existing system.
The first approach has numerous advantages, not least that in the event of unexpected problems during or immediately after the upgrade, the old disk can simply be swapped back in and the system immediately restored to it's previous state. For a virtual machine, this is almost certainly the best answer.
Unfortunately, not everybody has a spare internal hard disk laying around, and even if they do, it might not be of a suitable capacity or performance to swap into their workstation or a physical server as a substitute for the boot and root disk. In this case, we would need to back up the data to an external disk, locally networked server, or other appropriate storage device, so that the original disk can be wiped and re-used.
Of course, backing up and restoring user data is easy. However, backing up an entire running system and then restoring everything onto a blank disk so that the system boots correctly, whilst ensuring that all file permissions, hard links, symbolic links, and other special files are exactly how they were before, might be more difficult and error-prone.
Keeping notes about disk usage, disk layout, and installed packages
Before shutting down our exising installation for the last time, it's useful to create a few extra plain text log files containing information from the running system that we might want to refer to if we encounter problems once we're running the new version:
Preserving useful data from the old system
# mkdir /end_XX_logs
# df -h > /end_XX_logs/disk_usage
# for i in `sysctl hw.disknames | cut -d = -f 2 | tr ',' ' '` ; do disklabel -h ${i%:*} > /end_XX_logs/label_${i%:*} ; done
# pkg_info -m > /end_XX_logs/packages
# pkg_info > /end_XX_logs/packages_all
# sysctl hw.sensors > /end_XX_logs/sensors
# usbdevs -vv > /end_XX_logs/usb_devices
# pcidump -v > /end_XX_logs/pci_devices
# ifconfig > /end_XX_logs/network_interfaces
# for i in /home/*/.local/share/xorg/Xorg.0.log ; do echo $i >> /end_XX_logs/Xorg.0.log ; cat $i >> /end_XX_logs/Xorg.0.log ; echo >> /end_XX_logs/Xorg.0.log ; done
Obviously, replacing XX with the current version number.
The details of disk partitioning and usage of each partition on the existing system is useful as a guide when deciding how to partition the disk during the new installation. The lists of manually installed packages and of all installed packages including dependencies is useful to check which if any new dependencies have been introduced, and compare version numbers of packages installed.
The sensors, usb, and pci device information can be used if any regressions are suspected, to see if any un-expected output from the new system is indeed different, or whether it was in fact, ‘always like that’, and we just didn't notice.
Noting CPU and disk performance
“The new version is slower!”
Or does it just feel slower?
Sometimes, after an upgrade, the system can feel faster or slower. Whilst this is often a psychological effect, occasionally a bug fix, or a newly introduced bug might genuinely affect performance. Another possibility is any changes to BIOS settings that may have been made during the course of the upgrade.
To give ourselves at least a crude benchmark of CPU and disk throughput, we can create another logfile:
# uname -a > /end_XX_logs/benchmark
# date >> /end_XX_logs/benchmark
# for i in 1 2 3 ; do echo "Test $i" >> /end_XX_logs/benchmark ; dd if=/dev/rsd0c of=/dev/null bs=1m count=256 2>> /end_XX_logs/benchmark ; done
# md5 -tt >> /end_XX_logs/benchmark
Casual test of CPU and disk throughput
A typical logfile from the above benchmark will look something like this:
OpenBSD workstation.lan 7.0 CUSTOM.MP#90 amd64
Sat Mar 26 17:38:06 -00 2022
Test 1
256+0 records in
256+0 records out
268435456 bytes transferred in 1.263 secs (212420680 bytes/sec)
Test 2
256+0 records in
256+0 records out
268435456 bytes transferred in 1.143 secs (234822235 bytes/sec)
Test 3
256+0 records in
256+0 records out
268435456 bytes transferred in 1.126 secs (238334501 bytes/sec)
MD5 time trial. Processing 100000 10000-byte blocks...
Digest = 766a2bb5d24bddae466c572bcabca3ee
Time = 1.980000 seconds
Speed = 505050505.050505 bytes/second
Typical logfile generated from the commands above
In case of doubt, we can repeat the tests on the new system and compare the results. Whilst the above benchmarks are very simplistic, they will often be sufficient to demonstrate regressions.
Backing up to another disk
If you're going to swap out the existing disk and perform the new installation on a different one, then you can skip this section.
Your backup will effectively be the disk that you've just removed, or more likely, that you've re-configured as a secondary disk ready to copy data back from afterwards.
Assuming that you're not swapping the boot and root disk out, then you'll need to copy it's contents elsewhere. At that point, you'll effectively be in the same situation as if you had swapped it out, except that the backup disk won't be bootable without some extra tweaking.
Important note
Ensure you have the installation files for the existing system
If you decide to back up your existing installation to another disk, rather than swap the disk out and use this original copy as a backup, then be sure that you have a local copy of the distribution files for the existing version of OpenBSD that you are using, as well as any binary packages that you are using from ports, or the ports distfiles required to build them.
These files may be required if any newly introduced bugs in the new release cause the upgraded system to be unusable, and you want to restore the previous version by re-installing it.
The first step is to stop any running servers, and un-necessary processes. In most cases it's best to boot into single user mode.
To copy the contents of the partitions on the boot disk, whilst retaining the correct file permissions and links, we'll use /bin/pax. Don't try to do this with cp, or tar. Neither will correctly handle hard links, instead we would end up creating two or more identical copies of files that were once links to the same file.
It's fairly simple to use /bin/pax to copy a directory hierarchy and preserve all of the file attributes:
# mkdir /destination
# cd /source
# pax -rw -vpe . /destination
Copying a directory hierarchy with pax.
Note that whilst file attributes are preserved, file flags are not.
It's slightly more complicated when we want to copy everything from the root directory downwards, because we need to mount the target itself somewhere within the filesystem.
However, if we have the target disk partition mounted on /mnt, and we are using the korn shell, we can use filename patterns to avoid traversing /mnt:
# pax -rw -vpe /!(mnt) /mnt
Copying the root directory hierarchy to a destination which is mounted within it
Important note
File flags are not preserved
Although the above invocation of pax will preserve file attributes due to the use of the -pe argument, it will not preserve any file flags that may be set, unlike the similar -p argument to /bin/cp does.
No file flags are set during a default installation of the base system, and most third party software doesn't use them either, so this is unlikely to be an issue unless you have manually set file flags at some point in the past and have scripts that rely on them for correct behaviour.
If the target disk partition is blank, and we are prepared to start over with the copying in the case of a power failure or crash, we can speed up the backup process significantly by mounting the partition with the async flag:
The async flag will speed up copying at the risk of losing data on the target partition in case of a system failure.
Usually we would want to create a single large partition for the entire backup, rather than duplicate the separate partitions for /usr, /var, etc, that we have on the original disk. If we're not going to actually boot the system from this backup disk, then the main reasons for using separate partitions, namely the ability to set different mount options and avoid hard links being created with filenames that would otherwise be on separate partitions, don't really apply.
If we did intend to make the backup disk bootable, and potentially use it as a live system in the case that our new installation fails, it would obviously be advisable to create separate partitions on the same mount points. However, in most cases, the extra effort required to ensure that the backup disk is reliably bootable, isn't really worthwhile. At a minimum, we would need to run /usr/sbin/installboot, and also change all references to the duid of the old disk in /etc/fstab to that of the new one. Other changes may also be necessary, especially if the target disk is an encrypted softraid volume, as it may not be automatically assembled at boot time, meaning that the kernel will boot but be unable to find the root filesystem, (or even the device that it's on).
Readers who are confident in their ability to make the backup disk reliably bootable, probably don't need to be reading this article in the first place.
Optionally secerase the new boot disk
If the disk that we are using as the new boot and root disk is a SATA SSD that has been previously used, there might be an advantage to running the secerase command on it before performing the new installation.
Since OpenBSD does not support the trim command, performing a secerase is usually the only practical way to reset the disk's internal mapping of logical blocks to flash memory cells. On some SSDs, we have seen a noticeable performance increase after performing a secerase, and in at least one case, doing so resolved what seemed to be a firmware bug causing extremely slow read access, (to the point that the SSD was practically unusable).
Important note
Exercise caution with the secerase command
Use of the secerase function is not without risk, and can potentially damage your device in the case of firmware bugs, or hardware failure. We have seen at least one SSD which became temporarily bricked after having a secerase performed on it. In that specific case, we were able to revive the SSD by simply leaving it for 24 hours disconnected from any power supply. Upon re-connection, the disk had un-bricked itself and continues to function to this day.
However, due to the risk of permanent hardware damage, or temporary unavailability, exercise caution when using the secerase command.
To issue the secerase command to a disk, we can use /sbin/atactl. However, note that most disks will first require us to set a password using secsetpass.
Neither of these commands will succeed by default on OpenBSD, when issued to a disk connected via SATA, as the kernel freeze-locks all such disk devices that it enumerates during the boot process. The code that does this is in /usr/src/sys/dev/ata/atascsi.c and /usr/src/sys/dev/ata/wd.c.
Note that many BIOSes also freeze-lock disk devices that they detect as well, so some experimentation might be necessary to find a way to reliably issue the relevant commands to the disk.
A response such as the following from /sbin/atactl usually indicates that the disk is indeed freeze-locked, and therefore the issued command has not succeeded:
# atactl sd0 secsetpass master
Master password:
Retype master password:
atactl: ATA device returned error register 0
The drive is probably freeze-locked
Disk partitioning and separating user data from the system disk
One of the best ways to facilitate any future re-installations, and ensure that they go quickly and smoothly, is to restrict the role of the boot disk to just holding the system itself, and as little user data as possible.
User data which is kept on a separate physical disk, doesn't need to be restored to the system disk from a backup after the installation is complete, saving time.
A small partition, on a separate physical disk which is intended for user data, is also a convenient location to store a temporary copy of the existing system files for reference after the re-installation, as described above in the ‘backing up to another disk’ section.
If the boot and root disk contains no user data at all, then the currently installed OpenBSD system can be wiped and re-installed with very little preamble whatsoever.
However, rather than placing the main /home directory on a separate disk, it's often preferable to simply create a small /home partition on the boot and root disk, to hold limited user data such as configuration files. A much larger partition on the secondary disk can then be mounted elsewhere, such as /store, and users can then effectively be given a second home directory in which to place most of their other, larger files.
Initial tests for show-stopper issues
Obviously, personal preference will decide the exact order that some steps are carried out.
However, it's useful to check early on for some obvious issues that might require us to revert to the backup of the previous installation, before progressing too far with the configuration of the new system. This is especially true if avoiding excessive downtime for this machine is important. For example, if a bug has been introduced to a device driver since the version of OpenBSD that was previously installed, it might prevent normal operation of the hardware in this particular system, but a fix for the bug might not be immediately available. Such a situation could easily leave restoring the old version of OpenBSD as the only realistic option for the time being.
In such a case, it might also be desirable to avoid any of our user-data being touched by new versions of software installed from ports, given that any such software might update the file format of existing data files to new versions which will then be unreadable by the old versions of the software.
Important note
Softraid volumes can become unreadable by older OpenBSD versions
If you are using softraid volumes, check to see whether there have been any changes to the structure of the softraid metadata between the old version of OpenBSD and the new version, before allowing the new system to assemble any existing softraid volumes.
This was an issue between OpenBSD 5.9-release, and OpenBSD 6.0-release, for example. The on-disk format of the softraid metadata was changed, (from version 5 to version 6). As soon as a disk with version 5 metadata was assembled by the softraid code in OpenBSD 6.0-release, the metadata was updated to version 6. As a result, the volume became unreadable on any machine still running OpenBSD 5.9.
It's worth checking that the output of the following commands matches, or at least correlates with the output of the same commands on the old system, (which we saved in text files before the re-installation):
# sysctl hw.sensors
# usbdevs -vv
# pcidump -v
Some differences in the output of these commands are to be expected between different OpenBSD versions. However, significant differences in unexpected places could be an early indicator that code changes between the old and new versions have broken some functionality.
If this is a workstation, rather than a server, then we should also check that we can start X:
Starting X11 as a regular non-root user
All hardware devices that we expect to use should appear in the boot time dmesg output, and any differences in hardware detection between the two OS versions should be investigated.
Additionally, although any network cards that were not set up by the installation script won't yet be configured at the IP level, the output from ifconfig should still report all of those that are physically connected to another machine which is powered on as having a data carrier detected. They should also be listed as having negotiated the same link speed as before. An incorrect media type, incorrect link speed, or lack of a carrier, could indicate a newly introduced driver bug.
Restoring the old system in case of showstopper bugs
If any bugs or possible bugs have appeared, which don't have an immediate solution, then we might want to take the option of restoring the old system.
Before we do, it would be very worthwhile to save any information from the new installation that might be useful in diagnosing the problem.
This will likely be a subset of what we saved from the old installation before the re-install, and can either be saved to a removable drive, or another internal disk in the system.
Logging a subset of the information that we logged from the old system, from the new system before reverting it.
# dmesg
# ifconfig
# sysctl hw.sensors
# usbdevs -vv
# pcidump -v
This is useful for debugging the issue with running the new version on this hardware, if the system is a production system that can't be taken out of service.
If the boot and root disk of the previous installation was swapped out prior to re-installing, then the old system can quickly be restored by swapping it back in. In this case, assuming that no user data has been modified on other disks, no further steps should be necessary to restore the system to it's previous working state.
If, on the other hand, the contents of the old boot and root disk was copied elsewhere for backup, restoration will be slightly more complicated.
There are basically two options for restoring the original system. Either we can make the backup disk bootable, or we can perform a fresh installation of the previous version of OpenBSD and continue with the restoration of configuration files and user data in much the same way that we would with any new installation.
In most cases, a fresh installation of the version of OpenBSD that we were previously using is probably the best option. This has the advantage that we retain the ability to secerase the boot disk first, to size the partitions differently, and add or remove full disk encryption from the system.
Merging previous local changes into /etc/
Assuming that we are happy with the new system, and that no hardware devices have suddenly stopped working as a result of the upgrade, one of the first things we will want to do is to restore any configuration files in /etc/ that had local changes made to them.
Unfortunately, in most cases it isn't advisable to simply copy the contents of the old /etc/ directory to /etc/ on the new installation. Local changes made to the configuration files need to be merged with any changes included in the base system. For example, between OpenBSD 6.8 and OpenBSD 6.9, it became necessary to define pki entries explicitly in smtpd.conf, and between OpenBSD 6.9 and OpenBSD 7.0, various default settings for snmp changed.
As a result, for configuration files that were based on the examples provided in the base system, creating new configuration files in /etc/ usually means comparing our existing files with the corresponding stock files for the old system, understanding the changes that we made, and then creating them anew.
This process is much easier if we adopt a constant discipline for renaming the original files before editing them. So for example, if we decide to create our firewall ruleset by editing the supplied /etc/pf.conf, rather than creating a new ruleset from scratch, then we might keep the original copy of /etc/pf.conf that was included in the base system as /etc/pf.conf.base.
It then becomes easy to find any files that we have changed.
Assuming that the old system backup is again mounted on /previous/:
# find /previous/etc | grep '\.base$'
Locate backed-up copies of the unmodified versions of configuration files that we subsequently modified in the previous installation
We can also easily create a diff containing all of our changes:
# for i in `find /previous/etc | grep '\.base$'` ; do diff -u $i ${i%.base}; done
View the local changes that we made to configuration files in the previous installation, so that we can manually implement the same changes in the new installation
Side note
Filename extensions used in examples
Observant readers will remember that I usually use an extension of .dist for backups of files as they were in the base distribution. However, in the /etc/ directory hierarchy there are already files with a .dist extension, /etc/mtree/4.4BSD.dist, and /etc/mtree/BSD.x11.dist.
To avoid confusion in the example above, I used an extension of .base instead, but it's also trivial to use .dist and simply manually exclude such pre-existing files from any lists generated. The example using diff would also work fine, effectively skipping those files as long as files with the truncated names /etc/mtree/4.4BSD and /etc/mtree/BSD.x11 don't exist.
Configuration files which were created anew rather than being based on existing files, are easily identified by timestamp:
List the files in the old /etc directory ordered by timestamp, to easily find newly created configuration files.
Unless we had deliberately created files with a timestamp in the past, (which could occur due to using the -p argument to cp when copying configuration files from a previous installation), files which remain unaltered from the base system will be listed first, all with almost identical timestamps, followed by files which have been added or changed.
We can then easily produce a list of files that need re-creating with possible manual changes:
# cd /previous/etc/
# ls -1tr > /etc/manual_update_list
Store the above list of filenames in a file for use as a checklist
This will include the password files /etc/passwd and /etc/master.passwd, After manually removing directory entries, and transient files such as random.seed, as well as adding any files from subdirectories such as /etc/mail/ and /etc/iked/, we are left with a list of files to copy from the backup and manually update:
# cd /previous/etc/
# cp -v `cat /etc/manual_update_list` /etc/
# cd /etc
# vi `cat /etc/manual_update_list`
After manually editing the checklist to only include the relevant files, copy just those from the old installation to /etc/ on the new installation, and edit each one in turn using vi
Finally, if we've updated /etc/passwd and /etc/master.passwd, we should run /usr/sbin/pwd_mkdb, to re-create /etc/pwd.db and /etc/spwd.db.
This week we explored some different techniques for streamlining the process of upgrading by re-installing. However, these are only intended as a starting point for readers to develop their own personal workflow for upgrades. The real power comes when you can establish a procedure that caters exactly for your own needs.
In the next and final installment of this ten part series, we'll wrap things up by looking at some interesting, but seemingly under-used and under-appreciated features of OpenBSD.
IN THE FINAL INSTALLMENT, JAY UNCOVERS AN UNDOCUMENTED FILE FLAG, CREATES EXTRA VIRTUAL TERMINALS, AND LIBERATES THE FRAMEBUFFER CONSOLE FROM IT'S RATHER GLOOMY AND MOSTLY MONOCHROME EXISTENCE WITH A SPLASH OF COLOUR THANKS TO A CUSTOM TERMTYPE ENTRY! STAY TUNED!