“VT220 is so 1980s! Let's replace it with... Something just as old!”
Adding some xterm-compatible control sequences to the OpenBSD wscons code
Crystal has been busy hacking the console code in OpenBSD again.
Follow her progress as she teaches it some xterm-like moves, and find out why anyone would want to do so in the first place!
Do you sit back and stare in wonder every time you shell in to your OpenBSD machines from a Linux system? Or from any non-BSD system for that matter? Don't you think it's amazing how it all 'just works'?
If you have decades of experience in UNIX-like systems, you might do. Otherwise, probably not.
After all, it's all just ASCII characters, right? Or more likely unicode in the form of UTF-8 these days. Surely that ensures compatibility? What needs to be configured? What is there to go wrong?
Quite a lot, actually.
Once you start to scratch the surface, it quickly becomes clear just how complicated consoles can be.
Quick introduction for terminal newbies
If you've got no idea what we're talking about here, let me take a moment to explain.
We're not talking about configuring the connection at the hardware level. So this isn't about supported baud rates, parity, stop bits, 7E1, 8N1, hardware handshaking or anything along those lines.
Rather, we're talking terminal descriptions. As well as the printable alpha-numeric characters, every physical terminal or terminal emulator needs to understand certain control sequences to do things like move the cursor around, scroll, clear the screen, change colors, set underlining and bold fonts, and so on.
These control sequences are essentially different for every terminal, although admittedly there is now a lot of common ground for the simplest and most essential controls.
The problem is that it's the system at the remote end of the connection that needs to send the right codes for your terminal. Since the remote system usually wants to support as many different terminal types as possible, (because it has no idea who might be connecting to it), it needs to have a large database with descriptions of every possible terminal that just might connect one day.
This database is called the terminal capability database, (or terminfo database), and on an OpenBSD machine you'll find the source for it in /usr/src/share/termtypes/termtypes.master.
The problem
In many non-UNIX-like systems, the operation of the local display and keyboard is tightly bound to various functions of the OS. It's expected that you'll sit in front of the machine to use it. If remote access is possible, it's implemented quite separately from the low-level drivers for the local hardware.
Not so on a BSD system. The system's physical console is not really a special case. The local display and keyboard are abstracted from the system and a piece of kernel code uses them to emulate a terminal. That emulated terminal is then connected to the rest of the system in more or less the same way that any other connection is, such as a serial port or networked ssh socket.
Connecting to an OpenBSD machine from a different system over the network with ssh should work just fine, as long as the terminal that you're using to connect has an entry in the termtypes database, (and it almost certainly does).
If you connect via a serial port, you'll need to set the terminal type yourself. The vast majority of terminals understand enough sequences from the vt220 terminal type to make basic operations such as cursor movement and clearing the screen work, so for simple command line work setting it to vt220 might even be sufficient.
The real fun begins when you use the local keyboard and display.
Most people running OpenBSD on i386 or amd64 hardware in 2023 will find that their display hardware is supported by the framebuffer console, and it's been like this for about ten years now. Although the kernel starts booting in the 80 × 25 character VGA text mode set up by the BIOS, it quickly changes to a bitmapped display, such as 1920 × 1080, and draws each character graphically.
This not only allows for a much larger terminal size, (measured in characters), but also potentially allows for a full range of text effects and use of color with no restrictions. This is in contrast to the VGA text console which, for example, doesn't allow free use of color together with underlining due to hardware limitations.
All of these advanced facilities need to respond to different control sequences that are received from other programs by the terminal emulation code. If our system was working in isolation we could use whatever control sequences we liked, however if we connect to a remote system and run a console based program, then we are expecting the remote system to send the correct sequences for our terminal.
Key point
Terminal software for non-UNIX-like systems is often quite complicated, containing code to interpret the control sequences coming from the remote system and display it using the local operating system's text or graphics handling functions.
In contrast, on a UNIX-like system, the output from the remote system is usually passed more or less directly to the local terminal - or in this case, terminal emulation code in the kernel - and expected to work. As a result, programs like /usr/bin/cu can be very small indeed.
If we were to create an entirely custom and proprietary set of control sequences, then the chances are high that the remote system wouldn't have anything like it in it's database. On the other hand, if we use code sequences that are already used by a common terminal type, such as vt220, and supplement them with additional codes, we can expect that the remote system will at least be able to control the essential basic terminal functions. Color and different fonts might not work, but at least the text will be in the right place.
This is basically the situation today with the wscons code in OpenBSD. The terminal emulation code fully supports color, for example, but by default most programs running on the console won't use it, (unless they ignore the termtypes database and make their own assumptions about what the terminal supports).
The default terminal type, in other words the terminal type that the rest of the programs on the system are targeting, is set to vt220. This is a very safe default, and allows users to connect to just about any remote system without breakage.
However it's also what is preventing most local programs from using any features that are not supported by the vt220 terminfo entry, including color.
The terminfo file does include an entry specifically for the OpenBSD console, called pccon. This was added in 2011, and has seen a few minor updates since. However, it has a few issues:
Can't we just fix pccon?
Fixing the pccon terminfo entry might seem like the obvious answer at first sight, and it could indeed make everything work locally. However this naïve approach ignores the important fact that when we connect to a remote system, we would need the remote system to have an accurate pccon entry. It's probably reasonable to assume that any remote system running today that uses the terminfo file will be modern enough to have an entry of some sort for pccon, but if it were changed now, it could take many years for the changes to fan out to all of those systems.
The solution - fix the emulation
If the wscons code emulated a common, well-known terminal entry in the terminfo database, which also happened to provide more functions than vt220, then our problem would be solved. The changes to make this happen only need to be made at the local end, since by definition, if the chosen entry is common and well-known, then we can assume that any remote systems will support it.
One such possible terminal description to target is xterm. This has been in the terminfo database for over 20 years with some incremental changes, so it's effectively going to be present in any currently running system that we want to connect to.
What needs to be done?
For the rest of this article, we're assuming that you are using wscons with the regular vt100 emulation code. If you are running on the amd64 architecture and have not explicitly disabled it, then you will be.
If we boot in to a fresh installation of OpenBSD 7.2 and set the TERM environment variable to xterm, we can quickly see that support for some of the control sequences is missing. Simple terminal commands work, but launching vi and trying to edit a text file results in screen re-painting problems when we scroll.
Lots of programs from the ports tree will also show similar issues.
From looking at the output, it's fairly obvious that some kind of cursor movement controls are missing.
To find out more, we can compare the terminfo entries for vt220 and xterm. This is slightly awkward, as the terminfo format allows one entry to include parts from other entries, so we need to look at quite a few to build the whole picture.
Some of the capabilities that are defined in xterm but not in vt220 are already supported by the wscons code. This includes the color setting sequences, setab and setaf, as well as two that work to hide and show the cursor, civis and cnorm. These will automatically function as expected when we set TERM=xterm, without further modification of the code.
However, other capabilities that are defined in the terminfo entry but not yet implemented by wscons include:
Control sequenceTerminfo capability nameDescription
CSI 𝓍 dvpago to line 𝓍
CSI 𝓍 Ghpago to column 𝓍
CSI 𝓍 Trinscroll back 𝓍 lines
CSI 𝓍 Sindnscroll forward 𝓍 lines
CSI Zcbtreverse horizontal tab
These control sequences are currently ignored by wscons. However, adding support for them is trivial!
Adding support for new control sequences to wscons
If you've read my 'Candlelit Consoles' article, some of this will likely feel familiar. We're going to be looking at /usr/src/sys/dev/wscons/wsemul_vt100_subr.c again, because that's where a lot of the console emulation code resides.
Specifically, control is handed to the function wsemul_vt100_handle_csi in this file once the CSI has been detected. This function is basically one massive switch statement, with several smaller switch statements embedded, that either pass control to various other functions that implement the sequence in question, set flags, or do other small manipulations of the state of the console directly.
Adding the functionality listed above just requires us to add an extra case statement for each one, along with the code to implement it.
Absolute column and row addressing
Here is the code to implement the vpa and hpa capabilities:
case 'G': /* Go to absolute column */
edp->ccol = min(DEF1_ARG(0)-1, edp->ncols-1);
case 'd': /* Go to absolute row */
edp->crow = min(DEF1_ARG(0)-1, edp->nrows-1);
As might be expected, the code only needs to change one value. In this case, the current row and column are both stored in edp, which is defined as a struct wsemul_vt100_emuldata in wsemul_vt100var.h. This file also contains the definitions for various macros that are used throughout the CSI implementation code, such as DEF1_ARG as used above.
Since the, (unsigned), integer values for row and colum in the edp structure are zero-based, and the parameter specified in the control sequence is one-based, we subtract one from it. Then we use the min function to ensure that we don't set a value past the right hand side or bottom of the screen.
Scrolling forwards and backwards
This code is also fairly trivial:
case 'S': /* Scroll forwards X lines */
wsemul_vt100_scrollup (edp,DEF1_ARG(0));
case 'T': /* Scroll backwards X lines */
wsemul_vt100_scrolldown (edp,DEF1_ARG(0));
Unsurprisingly, the wscons code already contains functions to scroll the screen. The wsemul_vt100_scrollup routine is called when the cursor is already on the bottom line of the display, and needs to move to what would be the next line - this is just normal one-line scrolling.
What we want to do here, though, is allow these functions to be called directly by control sequences, and pass them the argument received as part of that sequence, as that represents the number of lines that we want to scroll.
Luckily, these functions already handle two complexities that we would otherwise have to deal with. The first of these is scroll regions, where a only a pre-defined set of rows is scrolled rather than the whole display, and the second is characters which are either double width, double height, or both.
So yet again, we just pass the argument received from the control sequence directly to the existing scroll functions. We don't even have to worry about bounds checking the passed value, as the called functions do this for us.
Reverse tabulate
Implementing reverse tabs actually requires some new logic, but it's still very simple:
case 'Z': /* Reverse TAB */
if (!edp->ccol)
if (edp->tabs) {
for (n=edp->ccol-1 ; n>0; n--)
if (edp->tabs[n])
} else {
n=((edp->ccol - 1) & ~7);
This is effectively the reverse of the algorithm used in the existing forward tab code, which can be found in wsemul_vt100.c, within the function wsemul_vt100_output_c0c1.
First we check to see if we are already in the left-most column. If we are, then we don't need to do anything at all, and the code just becomes a NOP.
To find the previous tab-stop, in principle we just need to step back from the current cursor position one column at a time until we either find a column that is flagged as a tab-stop, or we reach the leftmost first column.
The value of edp->tabs is usually expected to be a pointer to an area of memory, one byte for every column, containing flags that indicate whether that particular column is a tab-stop by being set to a non-zero value.
However, the normal forward tabbing code in wsemul_vt100.c also checks whether edp->tabs is a null pointer, (which can happen in wsemul_vt100_cnattach), in other words, no memory has been allocated. In this case, that code uses a simple fixed set of tab-stops calculated as being at every eighth column.
We use similar logic in the new code, and set edp->ccol to the computed value.
Whilst looking at wsemul_vt100_attach to try to work out the reasoning for the above detail, I noticed that there is a call to the KASSERT macro within an #ifdef DIAGNOSTIC block.
This #ifdef could be removed, as the KASSERT macro is defined in /usr/src/sys/lib/libkern/libkern.h as a NOP, unless DIAGNOSTIC is defined anyway.
In fact, this change was made to the equivalent code in the NetBSD tree back in 2017.
Early success
Patching the kernel with the above code, we can see that already a lot of programs will happily run and produce a correct display on the console with TERM set to xterm.
I was actually quite surprised at how little needed to be changed to get this far.
Next step - function keys
The biggest problem still remaining is that the F1 - F4 keys are not recognised by any program that uses the terminfo database to interpret them.
Amazingly, they don't even work with the default vt220 terminfo entry, so at least our switch hasn't broken this functionality.
The wscons code that defines the sequences sent by various function keys is in wsemul_vt100_keys.c. Here we can see that for F1 - F4, the control sequences are defined as ␛[11~, ␛[12~, ␛[13~, and ␛[14~. This is actually the most logical, natural, and technically correct way to extend backwards the set of sequences used for F5 onwards.
However, the terminfo entry for xterm expects ␛OP, ␛OQ, ␛OR, and ␛OS. These are the control sequences traditionally associated with the PF1 - PF4 keys, (and in fact they are defined as such further down in wsemul_vt100_keys.c). These PF keys were the only function keys on the original VT52 terminal, and were located at the top of the numeric keypad.
To fix this, we've got two choices. We could leave the definitions in wsemul_vt100_keys.c alone, and change the assignment in the various keyboard mappings found in /usr/src/sys/dev/pckbc/wskbdmap_mfii.c to make the physical F1 - F4 keys send the sequence designated as KS_KP_F1 - KS_KP_F4. Alternatively, we could just change the definitions for F1 - F4 in wsemul_vt100_keys.c to send ␛OP, ␛OQ, ␛OR, and ␛OS.
Note that the first approach won't affect the use of CONTROL-ALT-Fkey to switch virtual terminals, as the command function, (I.E. switching VT), for these keys is defined separately.
I've tested both methods of making F1 - F4 work with the xterm terminfo entry, and they both give satisfactory results.
If you're lucky enough to have a keyboard that sports F13 - F24 keys, (or if you've mapped these sequences to other key combinations, such as shift-F1 - shift-F12), there is additional work to do. The sequences expected when running with TERM=xterm are also different to those which wscons currently generates. In this case, changing them in wsemul_vt100_keys.c is the obvious solution.
A few non-function keys are also affected. Neither home nor end are correctly configured by either the vt220 or xterm terminfo entires. The arithmetic operator keys on the numeric keypad are correctly configured when using TERM=vt220, but break after setting TERM=xterm. There are also some differences in how each of the three entries treats the enter key on the numeric keypad.
Missing functionality
Xterm implements quite a few features which wscons doesn't yet support. If we're going to use the xterm terminfo entry with wscons, it would be nice to implement at least some of them, as obviously any program parsing that terminfo entry is going to assume that they are available. I demonstrated code to implement dim text in my Candlelit Console article, and if you've tried the accompanying patchset you'll know that it also includes sample implementations of strikethrough and double underline. These were only intended as a proof of concept, and only worked for displays using the 32bpp functions, but they do indeed show that the functionality is not difficult to add.
Support for blinking text would be more complicated, as it would require storing and updating information about the location of any characters with the flash attribute enabled, and then periodically updating the framebuffer. It's certainly not impossible, but it's also not trivial.
Today we've seen how, with just a few lines of code, it's possible to make the console broadly compatible with a widely used terminal description which supports far more features than a vt220.
Now we can enjoy color on the console, and compared to creating a custom terminfo entry this approach has the distinct advantage that it will work without further configuration when connected remotely to almost any modern system that uses terminfo.