“Avoiding the need to pull it out, only to push it in again!”
Implementing the MTRETEN ioctl on sd devices
Does issuing an eject command to a USB flash drive make it spontaneously fly across the room?
Sadly not, but we can still have some fun looking at the code that implements off-lining of sd devices and learn how to add a new feature to allow us to ‘re-load’ them without manually removing and re-inserting the device.
Most of you are probably familiar with using the eject command in conjunction with a CD-ROM or other optical device. In this case it physically ejects the loaded disc, either by opening the drive tray or in the case of a slot loading drive just pushing the disc out.
Tray loading drives can, (almost always), be told to insert a disc by using -t argument to the eject command. On the vast majority of optical drives, this closes the drive tray in the same way as pressing the eject button on the front panel.
# eject /dev/cd0cEject the drive tray
# eject -t /dev/cd0cRe-load the drive tray
Tape drives can usually be controlled in the same way, physically ejecting the media and in some cases being capable of re-loading it too.
The use of the -t option to the eject command to re-load removable media is not particularly portable across operating systems.
It's present function on OpenBSD has been implemented since 2005, so can safely be assumed to be available on any OpenBSD system in the wild.
On other systems, this option might have a completely different function - for example on NetBSD, (where /usr/bin/eject is implemented as a separate utility to /bin/mt), -t specifies the device type, and the -l option, (for load), is used instead.
Before relying on a particular behaviour in portable scripts, consult the eject manual page or source code for the operating systems that you expect to be running on.
What might not be as obvious is that eject can also be used with removable storage devices such as usb flash drives. In this case, there is almost certainly no way for the device to automatically physically remove itself from the usb socket. Even considering a usb storage device with removable media, such as a memory card reader, it's very unlikely that there is an automatic mechanism to load and unload it.
So what does the eject command do in these cases?
On the face of it, not a lot. Casual testing just seems to make the device inaccessible:
# eject sd7
# dd if=/dev/rsd7c
dd: /dev/rsd7c: Input/output error
But is this really all that it does, or there more going on behind the scenes?
It certainly seems like it, because checking with usbdevs, we can see that the system still considers the device to be connected:
# usbdevs -a 2
addr 02: xxxx:xxxx Generic, Mass Storage Device
Can we make use of this behaviour in some way, or is it just a vestigial curiosity?
The first thing to realise is that we can run the eject command against a device which is currently in use, in other words while I/O is being performed on it, or whilst it holds a mounted filesystem.
In this case, the eject, (or more accurately, the off-lining), is scheduled for when the I/O is finished, (or the last mounted filesystem is unmounted).
This has a clear practical use when used in conjunction with media which is actually ejectable, for example when writing a tar archive to a tape we may well want to remove the tape and put it into storage once the operation is completed.
In the case of a script writing a backup to a usb flash drive, it might be useful to ensure that no further writes are, (unintentionally), performed to the device between the backup finishing and the device being removed.
Although pretty much every storage device that plugs in to a usb port can be considered ‘removable media’ these days, not every such device actually presents itself to the sd driver as ‘removable’.
This matters here, as the eject command won't work with ‘non-removable’ devices, (technically, those without the SDEV_REMOVABLE flag set).
The reason for this apparent discrepancy is that, historically, devices such as hard disks were not hot-swappable. The device itself would always be present at the same hardware address once the machine had booted, and until it was powered down.
However, hard disk drives of that era may either have had fixed platters, in which case they would be considered non-removable, or exchangeable disk packs, in which case the media was indeed removable.
On a modern system that supports using usb storage devices, the usb subsystem is almost certainly going to implement hot-plugging of the devices themselves regardless of whether any particular connected device supports interchangeable media.
So basically, there are two different concepts of being removable to consider.
So why doesn't ‘eject -t’ work with usb flash drives?
Curious readers may have already tried to reverse the effects of an eject command against a usb flash drive without success:
# eject -t sd7
eject: sd7: Input/output error
Nope. It doesn't work.
Unfortunately, it doesn't make the device accessible again. Instead we're just greeted with an error message.
But why? Clearly this functionality could be implemented, as the device in question is still connected and recognised by the usb subsystem.
To find the answer, and eventually implement the missing functionality, we need to delve in to several parts of the source code.
In fact, the eject command in OpenBSD is just a shortcut for mt. This is a utility originally intended to issue various commands to magnetic tape drives, hence the name. It has since grown to support limited functionality with other storage devices such as optical drives.
Looking at the manual page for mt, we quickly learn that there is a whole wealth of interesting commands that we can potentially send to our devices, and that eject merely sends one of them, specifically the 'offline' command.
Sadly, most of these commands don't have any relevance to devices other than tapes. The mt program is certainly a good example of code from a past era which has to a large extent lost it's original purpose.
Operating systems other than OpenBSD usually implement eject as a separate utility, rather than an alias for mt.
The -t option
Historically, the -t option was used with mt to specify the device to use. Since this was always a tape device, the use of -t as a mnemonic made sense.
On modern systems, the use of -f to specify the device has become more common. OpenBSD removed support for the -t option from mt in 2003, although other systems such as NetBSD still retain it as an alias for -f.
When support for reloading removable media was added to eject in 2005, the same option letter was chosen for this function. This doesn't cause any conflict with the previous -t functionality of mt, as the -t option is only parsed when the mt code is invoked as eject. However, knowing this helps to avoid any confusion when comparing the source code for mt between systems.
Looking at the source for /bin/mt  
The manual page for mt tells us that eject invokes mt with the ‘offline’ command, but observant readers will have noticed that no mention is made of a ‘load’ command anywhere.
So what exactly is going on it the case of eject -t?
Now that we know that the source for /bin/eject is actually in /usr/src/bin/mt, we can open up /usr/src/bin/mt/mt.c and have a look.
It's actually quite simple. If the program is invoked as eject, then the eject flag is set. If the -t option is provided, then the insert flag is also set. When the insert flag is later checked, it switches between selecting COM_RETEN and COM_EJECT, which are both defined near the beginning of the file in the declaration of struct commands com.
So ejecting with the -t option is treated just like mt retension, and in fact this is noted in the cvs commit notes for revision 1.25 of mt.c.
Whichever command is chosen, and however it's sent, ultimately the mt code just ends up invoking an ioctl on the relevant device driver. It's then up to that individual device driver to implement the particular ioctls it can respond to.
Adding support for MTRETEN to sd.c  
If we want to add support for re-loading usb flash drives after ejecting them, there are two ioctls of interest to us.
These are in the #define's listed as the c_code elements in the struct commands com structure that we just saw in mt.c. The first is MTOFFL, as this is responsible for making the device inaccessible in the first place, and second is MTRETEN, which is what the sd driver will receive when we use the eject -t command against a usb flash drive.
Obviously we will need to add code to implement MTRETEN by reversing whatever actions MTOFFL performs.
The function we need to look at in sd.c is sdioctl, and here we can see that the only one of the mt ioctls which is currently supported is MTOFFL - anything else will just return EIO.
# mt -f sd0c rewind
mt: sd0c: rewind: Input/output error
Hardly surprising.
In the case of MTOFFL, the code just does a quick check that the SDEV_REMOVABLE flag is set, indicating that the drive supports this kind of operation, and then sets the SDEV_EJECTING flag. The SDEV_EJECTING flag is then checked by sdclose, and if it's set then a call is made to scsi_start which eventually leads to the appropriate command being sent to the hardware.
Handy hint!
It might seem a bit un-intuitive to be calling a function called scsi_start when we actually want to off-line a device.
However this is indeed correct, as the same basic SCSI command is used at the hardware level for both starting and stopping a connected device - it's just sent with different parameters.
These parameters are defined in scsi_disk.h, as SSS_STOP, SSS_START, and SSS_LOEJ.
Implementing MTRETEN
To implement our ‘reloading’ of usb flash drives using eject -t, we basically need to call scsi_start from within the ioctl handler for sd.c, in response to receiving a MTRETEN ioctl request.
This glosses over some of the finer details, but should be enough to make it work as a proof of concept.
Most of our new code can be added to the start of the existing case MTIOCTOP block. This alone isn't enough, though, because near the beginning of the sdioctl function we have a check for the SDEV_MEDIA_LOADED flag being set. If it's not set, then the switch statement that contains case MTIOCTOP is not parsed, because unless the supplied command is one of a few specific commands that can be executed without media already being loaded, the whole function just exits.
So at a minimum, we need to add a case MTIOCTOP to the first switch statement in sdioctl, to enable parsing of the following code which we'll add above the existing case MTIOCTOP in the later switch function:
if (((struct mtop *)addr)->mt_op == MTRETEN) {
scsi_start(link, SSS_START, 0);
scsi_start(link, SSS_START | SSS_LOEJ, 0);
goto exit;
During testing with various devices, some required SSS_LOEJ whereas others not only didn't require it but were actually prevented from reloading by including it.
There is probably a more elegant way to deal with this, but for testing purposes we can just call scsi_start twice, first without SSS_LOEJ and then immediately again with it.
Testing it
Amazingly, this code alone is enough to show that the idea works. Now, if we eject a usb flash device we can re-load it afterwards and read from it again:
# bioctl sd7
sd7: <Mass, Storage Device, \\001 \\001>, serial (unknown)
# eject sd7
# bioctl sd7
bioctl: BIOCINQ: Input/output error
# eject -t sd7
# bioctl sd7
sd7: <Mass, Storage Device, \\001 \\001>, serial (unknown)
It seems to work!
Not so fast...
The code above seems to work, but one situation that we've overlooked is both ejecting and re-inserting a device whilst it is being accessed.
In other words, doing something like this:
# mount /dev/sd7d /mnt
# eject sd7
# eject -t sd7
# umount /mnt
# mount /dev/sd7d /mnt
mount_ffs: /dev/sd7d on /mnt: No medium found
Not the desired result.
In this case, the code above isn't enough to ensure correct operation because we don't explicitly clear the SDEV_EJECTING flag. This didn't matter in the earlier example, because the sdclose function reset it for us. But if we want to re-load our media before sdclose gets a chance to run, we need to clear the SDEV_EJECTING flag ourselves.
The following version of the above code does exactly that:
if (((struct mtop *)addr)->mt_op == MTRETEN) {
if (ISSET(link->flags, SDEV_MEDIA_LOADED)) {
CLR(link->flags, SDEV_EJECTING);
goto exit;
scsi_start(link, SSS_START, 0);
scsi_start(link, SSS_START | SSS_LOEJ, 0);
goto exit;
Much better.
Interesting observation
Although implementing MTRETEN for sd devices was mainly an exercise in exploring how the eject command worked internally, whilst testing the code in conjunction with a memory card reader I did find a potential practical use for it.
# dd if=/dev/sd7c of=/dev/null count=1
dd: /dev/sd7c: No medium found
# dd if=/dev/sd7c of=/dev/null count=1
dd: /dev/sd7c: No medium found
# dd if=/dev/sd7c of=/dev/null count=1
dd: /dev/sd7c: No medium found
# dd if=/dev/sd7c of=/dev/null count=1
1+0 records in
1+0 records out
512 bytes transferred in 0.000 secs (724346 bytes/sec)
If the actual memory card is exchanged for a different one, (or even just removed and re-inserted), whilst the reader is left connected to the usb port, then the first few accesses to it fail.
# eject -t sd7
# dd if=/dev/sd7c of=/dev/null count=1
1+0 records in
1+0 records out
512 bytes transferred in 0.000 secs (724346 bytes/sec)
This doesn't happen if the newly implemented MTRETEN ioctl for sd devices is called first.
Today we've seen how to implement the basic functionality required to make the eject -t command work on usb flash drives.