EXOTIC SILICON
“Inbound SMTP without a static IP”
Tunnelling SMTP over IPSEC for traditional mail delivery from a VM
Dynamic IP?
Fed up with POP3 and IMAP?
Dream of real inbound SMTP right to your desktop?
We hear your pain!
But no worries, because the solution has arrived!
In this guide, Jay shows us how to set up an IPSEC tunnel between a local machine running OpenBSD, and a remote VM hosted at OpenBSD Amsterdam.
The idea behind this is to allow inbound SMTP connections from the VM to the local machine, for real, ‘traditional’, SMTP-based mail delivery all the way to your workstation, banishing tedious POP mailboxes and messy IMAP sessions to the bitbucket forever.
Introduction
Running your own SMTP server to collect and relay mail for your domain has some nice advantages over using a third party or ISP mail server.
Not yet running your own SMTP?
Fair enough, it's easy to just claim that running our own SMTP has numerous advantages, but what exactly are they?
Well, aside from becoming independent of large corporate email service providers, (which may in itself be enough of a reason for some users), some of the key technical advantages are:
  • Support for the latest standards, including IPv6, and TLSv1.3
  • Speed, flexibility, and local control of security settings
  • Dropping known junk connections before receiving junk message bodies
  • Advanced filtering based on connecting IP and remote user behaviour
  • Free choice of platform and SMTP software
  • Ability to set up custom configurations
If that's not enough, it's also an invaluable learning experience.
However, reliable mail delivery via SMTP requires a machine connected 24 hours per day to a high quality internet connection with guaranteed uptime, as well as a publicly accessible static IP address which is not listed on any widely used blacklists. For this, and other reasons, running your own SMTP server on a typical residential broadband connection is usually not a good idea. Since virtual servers are now available at low cost, running your own SMTP server in a VM makes a lot of sense, as the prerequisites we've just mentioned are much easier to obtain.
Setting up an SMTP server in a VM is not particularly difficult, although first time users might encounter a few difficulties along the way. Outbound mail doesn't really pose a problem, as you can easily connect to your own remote SMTP server from any IP to submit mail for relaying.
Inbound mail via SMTP, without a static IP?
Yes, it can be done!
The real problem, and the main issue we're going to solve here today, is how to collect inbound mail that is waiting for you on the VM.
If you've been used to using a desktop or mobile client program to access an IMAP or POP mailbox at your ISP, or even accessed all of your mail via a web-based interface, then you might be satisfied with setting up an IMAP or webmail server on the VM. At least initially.
However, many advanced and experienced users of BSD or Linux machines, as well as IT professionals, will desire a real SMTP connection all the way to their own workstation. This is, of course, impossible if that workstation doesn't have a fixed IP, since the VM will have no way to reliably initiate an inbound connection to it when mail arrives for onward delivery.
Although an SMTP connection can be simulated to a degree using programs such as fetchmail to download mail from a POP mailbox and then submit it into the local mail delivery system, it just isn't the same as having a setup with a real SMTP connection at every step.
If you have a static IP address on your broadband connection, then configuring the SMTP server running in your VM to deliver mail to an SMTP server running locally will usually work, (unless your ISP filters or blocks inbound connections). Even if the local IP is listed on public blacklists or your connection isn't solid 24 hours per day, mail delivery from the VM will be possible, as you are in full control of both machines and therefore able to configure them arbitrarily.
However there is still a need for a static IP address which allows inbound connections, and this remains a barrier for many users who simply cannot obtain a static IP at a reasonable cost.
Proposed solution - an ipsec tunnel
One solution to this whole problem is to set up an IPSEC tunnel between the local machine which is using a dynamic IP and the remote VM which has a static IP.
Once established by the local machine, the tunnel will be kept open and packets can flow from the VM to the local machine at any time. If the connection goes down, the tunnel will be quickly re-established once connectivity is restored, and data can once again flow across it. This works even if the dynamic IP changes. In fact, we can even use a different backup internet connection such as a 4G cellular link in place of the main broadband link, and the tunnel will promptly be re-established without difficulty.
Why not just use POP3 from the VM?
(and feed it into the local mailspool?)
Well, of course you can do. Programs such as ‘fetchmail’ allow exactly this, and many desktop clients even have POP and IMAP functionality built right in. So you can indeed download your mail from a remote server, (either your own server, or somebody else's), and save it in your local mailbox, completely bypassing your local system's mailspooling service.
But then you can't, (easily), use any of the features of the system's own mail spooler, either. You'll have created a non-standard system which, whilst it might work, is arguably more cumbersome than it needs to be.
On top of that, many of the popular POP and IMAP clients have frustrating arbitrary limitations. For example, whilst there might be an option to automatically delete collected mail from the server once it has been collected via POP, often no such option is available for IMAP based connections. There seems to be an assumption that the server is the final canonical resting place for received mail, and that via IMAP we are only supposed to be making local copies of it.
Connecting via POP is tedious, because whereas an IMAP session can be left open for an extended period of time, (hours), and supports pushing of new mail received during that time, POP does not. To simulate this using POP, we need to poll the server at short intervals, perhaps as short as 60 seconds. This is quite simply, messy.
So whilst mail collection via POP or IMAP might be convenient for a new and inexperienced user who just wants to get things up and running as quickly as possible, it has limitations.
It's also quite useless for anybody who wants to learn about and get experience of how SMTP mail delivery really works.
This whole process of bringing up and maintaining the IPSEC tunnel is opaque to the SMTP server, which simply delivers to the assigned IP address regardless of the fact that it is actually a private IP address rather than a publicly accessible one. If mail delivery is attempted whilst the IPSEC tunnel is down, the VM just sees a network error, and mail is held for the next delivery attempt.
In this guide, we'll be setting up an IPSEC tunnel between two machines both running OpenBSD 7.0. A local machine, and a VM hosted at OpenBSD Amsterdam. The principles should be easily applied to other hosting services and even other operating systems, but the specific examples given here are primarily aimed at users of OpenBSD.
Everything we need is included in a full installation of the OpenBSD base packages. That's to say, nothing whatsoever from the ports tree is necessary to set up SMTP delivery over an IPSEC tunnel.
Of course, SMTP isn't the only useful protocol that we can tunnel from the remote VM using IPSEC, and many of the principles that we're explaining here are equally applicable to other protocols.
A quick introduction to IPSEC
Since many readers will not be particularly familiar with IPSEC, a short introduction seems useful.
More information can be found in the manual page for ipsec(4) on an OpenBSD machine.
Why IPSEC?
(Can't we use wireguard instead?)
First of all, we would like to point out that when we first started setting up IPSEC tunnels here at Exotic Silicon, wireguard didn't even exist. Even when the first version of this guide was published, support for wireguard had only been in the OpenBSD kernel for about twelve months, making it a new and less well tested option.
Additionally, given the wide extent to which IPSEC is currently used in the IT industry, we expected that many readers who were not already familiar with it would be just as interested in the learning experience of setting up a practical, hands-on application of IPSEC, as they were for actually using the inbound SMTP facility.
Nevertheless, SMTP over a wireguard tunnel obviously can be done. Since we've been asked about this a number of times, and given that wireguard has continued to gain wider support and become a more credible alternative, we've now published an SMTP via wireguard tunnels article in the same spirit as this one.
Commonly, IPSEC is thought of as a standardised way to provide encryption of network traffic at the IP level, independent of higher level protocols. In fact, IPSEC actually provides much more than just encryption, including the ability to tunnel IP packets, as we will now see in our particular application.
IPSEC can be run over both IPv4 and IPv6, and was designed with the capacity to handle fairly large and complex networking setups. Some of this inherent complexity is evident even when setting up a simple point-to-point link as we will be doing. However, most of the complexity is in the terminology rather than the actual configuration.
Instead of configuring each individual network connection directly, IPSEC uses the concepts of flows and security associations. In very simple terms, a flow defines the IP addresses of two endpoints, (peers), and the corresponding security association is just a set of parameters that define things such as the actual encryption keys to be used.
Flows and security associations can be configured manually, but for various reasons, such as the fact that the encryption keys need to be changed on a regular basis, they are usually manipulated automatically by a daemon that implements the Internet Key Exchange protocol, or IKE. OpenBSD includes iked, which implements IKEv2. Since we will have full control of the configuration of both peers, there is little reason to use the older IKEv1, although this is still supported on OpenBSD using isakmpd.
This indirection might sound un-necessarily complicated, but it actually makes setting up and maintaining an IPSEC tunnel quite easy.
Block diagram of the proposed solution
Private IPv6 block for VPN: Public IPv6 block: 2001:db8:1234:1234::1/64
2001:db8:c1da::1/48 2001:db8:1234:1234::c155 namorar.example.com (Main IPv6 connectivity)
--------- 2001:db8:1234:1234::b17e smtp.example.com (IPv6 outbound mail to internet)
|NAMORAR| 2001:db8:1234:1234::6969 mx1.example.com (IPv6 inbound mail from internet)
--------- Static IPv4 192.0.2.2 smtp.example.com (IPv4 outbound mail to internet)
| | Static IPv4 192.0.2.2 mx2.example.com (IPv4 inbound mail from internet)
VPN endpoint vpn1.example.com / '-------------> INTERNET
2001:db8:c1da::2 \
/
V \ V
P / P
N \ N
/
\
VPN endpoint vpn0.example.com /
2001:db8:c1da::1 \ ,-------------> INTERNET
| | Dynamic IPv4
--------- --------- Dynamic IPv6 2001:db8:dd::dd
|MIMANDO| -------------------------->|CARINHO|
--------- ---------
2001:db8:6969::2 2001:db8:6969::1 Private IPv6 block for LAN: 2001:db8:6969::1/48
mimando, mimando.lan carinho, carinho.lan
The diagram above shows the desired arrangement of our three machines, two physical machines locally and a remote VM. The hostnames corresponding to the assigned IP addresses are also given.
Note that only the five entries listed at the top right need to be included in the public DNS. All of the other hostnames can simply be included in the local hosts file, as they are only used locally.
IMPORTANT NOTES
Hostnames in this example:
namorar.example.com
is our VM at OpenBSD Amsterdam.
carinho.lan
is a machine on our local network with internet access.
mimando.lan
is our workstation.
Configuring virtual network interfaces for the tunnel endpoints
With the preamble out of the way, we can now concentrate on configuring the new setup.
Since the IPs for the VPN endpoints are not going to be bound to physical network adaptors, the first step is to create virtual vether devices to use with them:
carinho# cat /etc/hostname.vether0
up
inet6 2001:db8:c1da::1
mtu 1024
namorar# cat /etc/hostname.vether0
up
inet6 2001:db8:c1da::2
mtu 1024
Style notes
To make this guide easier to follow, we've styled the console text output according to which of the three machines it relates to:
carinho# This is our local internet-connected machine, or LAN mailserver
namorar# This is our remote VM, or internet facing mailserver
mimando# This is our local workstation, which may or may not have internet connectivity
# This is generic output that is not specific to one machine
Additionally, individual lines of logging and debugging output from iked which exceed 500 characters will be wrapped on smaller displays for ease of reading. This is an exception to our usual style of using horizontal scrolling to avoid artificially breaking and wrapping long lines of console output.
Generating and installing keys for the IPSEC tunnel peers
Each peer of the IPSEC tunnel needs a way to prove it's identity to the other peer. The ‘traditional’ way is to use TLS certificates, but iked also allows us to use an alternative arrangement of directly sharing keys between the peers.
Sharing keys is slightly more straightforward to set up than a certificate-based system, so we'll be using this method in these examples.
It's fairly trivial to change a working iked configuration from using keys to using certificates, (or vice-versa), and this can easily be done once the SMTP delivery system is set up and working.
For more information about generating certificates for iked, (and about TLS certificates in general), please see our guide to encryption keys and TLS certificates.
Each peer requires a matching public and private key pair.
The private key lives in /etc/iked/private/local.key and the public key is in /etc/iked/local.pub.
The public key will need to be copied to the remote peer, renamed and stored under /etc/iked/pubkeys/fqdn/.
Looking at /etc/rc, you can see that OpenBSD generates a set of RSA keys on the first boot after installation, or more accurately any boot when /etc/isakmpd/private/local.key doesn't exist.
However, since the automatically generated keys are only 2048-bit RSA keys, we will replace them with an ECDSA key pair.
Creating an ECDSA keypair
First, we generate a file containing the key parameters
# openssl ecparam -out ec-secp384r1.pem -name secp384r1
Then we generate the new private key:
# openssl genpkey -paramfile ec-secp384r1.pem -out /etc/iked/private/local.key
The generated key will be written with permissions 0644, so we change this to 0640:
# chmod 640 /etc/iked/private/local.key
Finally we generate the matching public key
# openssl ec -in /etc/iked/private/local.key -pubout -out /etc/iked/local.pub
The ec-secp384r1.pem file can be deleted, as it is no longer needed.
# rm ec-secp384r1.pem
These are basically the same steps as described under ‘ECDSA key generation’ in our guide linked to above.
Note that the directory /etc/iked/private/ should have permissions of 0700, so there is no issue with the local private key file briefly being world readable.
Each local public key needs to be copied to the other peer, so in our example, /etc/iked/local.pub on the local peer carinho.lan needs to be copied to /etc/iked/pubkeys/fqdn/carinho.lan on the remote peer, the VM namorar.example.com. Likewise, /etc/iked/local.pub on the VM should be copied to /etc/iked/pubkeys/fqdn/namorar.example.com on the local peer, carinho.lan.
carinho# scp -p /etc/iked/local.pub root@namorar.example.com:/etc/iked/pubkeys/fqdn/carinho.lan
namorar# scp -p /etc/iked/local.pub root@carinho.lan:/etc/iked/pubkeys/fqdn/namorar.example.com
Copying the public encryption keys between the two peers.
The IKED configuration file
namorar# cat /etc/iked.conf
ikev2 esp from vpn1.example.com to vpn0.example.com local namorar.example.com peer any dstid carinho.lan ecdsa384
For the remote VM, the /etc/iked.conf file can be trivially simple.
This configures the remote peer to allow any traffic over the vpn, TCP, UDP and ICMP to any port. Connections sending ESP packets to the remote endpoint will be accepted from any IP address, since we have specified, ‘peer any’, but the other peer will be expected to identify itself as carinho.lan during the negotiation. Note that the IP address does not have to resolve to the dstid in any way, it's just convenient, (and default behaviour), to set the srcid to the hostname.
Since we haven't specified active mode, the VM will just wait for an inbound connection rather than try to bring up the tunnel immediately. This is obviously the only logical behaviour, as the remote VM doesn't know the dynamic IP that the local peer will be connecting from.
Filtering peer access by IP address
If we know the IP range that our ISP uses for dynamic IP allocations, we can restrict incoming isakmp packets, (UDP port 500), on the server to this IP range in it's firewall configuration.
Although this might seem superfluous since the correct key is needed to successfully negotiate an IPSEC tunnel anyway, doing so may help to reduce scanning attempts and provide some limited protection against any unknown bugs or exploits in the isakmp processing code itself.
Assuming that our server firewall is set to deny by default, a suitable pass rule for isakmp traffic might be something like:
pass in on vio0 proto udp from 2001:db8:dd::0/48 to namorar.example.com port isakmp
For the local machine, the /etc/iked.conf file is similar:
carinho# cat /etc/iked.conf
ikev2 active esp proto tcp from vpn0.example.com port 25 to vpn1.example.com peer namorar.example.com ecdsa384
ikev2 active esp proto tcp from vpn0.example.com to vpn1.example.com port 25 peer namorar.example.com ecdsa384
ikev2 active esp proto icmp6 from vpn0.example.com to vpn1.example.com peer namorar.example.com ecdsa384
Here we set three policies.
The first policy allows incoming connections from any port on the VM to port 25 on the local router, carinho.lan. This is obviously exactly what we need the IPSEC tunnel for in the first place.
The second policy allows connections from any port on the local router to port 25 on the VM, for submission of outgoing mail. We don't actually need to use the tunnel for this, as the VM has a publicly accessible static IP address that we could configure as our mail relay on carinho.lan. However, using the tunnel has several advantages and few disadvantages, so we might as well use it here too.
The third policy just allows icmp6 packets for testing connectivity.
We set each policy to active mode, so that as soon as iked is started on the local machine it attempts to bring up the tunnel with the VM. Once the tunnel is active it should, in ideal circumstances, stay open and allow the remote VM to make fresh connections to port 25 whenever mail needs to be delivered.
In reality, various things can cause the tunnel connectivity to drop, and we'll cover this shortly, in the ‘keepalives’ section.
Note that unlike in the first example above where we set dstid to carinho.lan and that was nothing to do with the connecting IP address, here we are specifying ‘peer namorar.example.com’, and in this way we are actually specifying a hostname that will be resolved to an IP address. So this either needs to be in the public DNS, or at least in the local hosts file.
This is a good reason not to try using literal IP addresses in the iked configuration file. It can be made to work, but generally speaking, configuration will be quicker and less error-prone if you stick to using hostnames.
By default, the expected dstid will be set to the same value, so it is as if we also specified ‘dstid namorar.example.com’.
Avoiding a common mistake
Having seen the example configuration file for the local machine, you might be wondering if we could be more specific in our configuration for the remote VM instead of allowing any traffic through, using a configuration such as this:
ikev2 esp proto tcp from vpn1.example.com port 25 to vpn0.example.com peer any dstid carinho.lan ecdsa384
ikev2 esp proto tcp from vpn1.example.com to vpn0.example.com port 25 peer any dstid carinho.lan ecdsa384
ikev2 esp proto icmp6 from vpn1.example.com to vpn0.example.com peer any dstid carinho.lan ecdsa384
Nope!
The answer is no, this will not work how you might expect it to.
If you read the manual page for iked.conf very carefully, you'll see why. Incoming connections are matched to policies based on the addresses, so in this case, each of the policies presented by the peer on our lan will match against all three of the policies configured above, and only the last one, (the policy allowing icmp6 packets), will be used in each case, resulting in no tcp traffic flowing across the tunnel.
If the reason for this isn't entirely clear, consider the first policy presented by the active peer. The addresses match all three policies defined on the passive peer, so the last one is used for the negotiation.
ikev2 active esp proto tcp from vpn0.example.com port 25 to vpn1.example.com peer namorar.example.com ecdsa384
ikev2 esp proto icmp6 from vpn1.example.com to vpn0.example.com peer any dstid carinho.lan ecdsa384
These two policies have nothing in common!
Clearly, since one policy only mentions TCP and the other one only mentions ICMP6, no flow will be established.
If we really wanted to make the configuration of the passive endpoint more restrictive, we could specify several traffic selectors in the same policy:
ikev2 esp proto tcp from vpn1.example.com port 25 to vpn0.example.com from vpn1.example.com to vpn0.example.com port 25 peer any dstid carinho.lan ecdsa384
However, setting up iked for the first time is probably confusing enough without creating extra complexity like this.
Keepalives
IPSEC is a stateful protocol, meaning that there is a concept of the tunnel being brought on-line, or ‘up’, before data can flow across it.
If packet loss, or any other network issue, causes data to stop flowing on this virtual connection, we would ideally like iked to recognise this situation and try to bring the tunnel up again automatically as soon as possible.
Based on our experiences of using IPSEC on OpenBSD machines in real-world applications, the default settings on their own will result in a tunnel that can be somewhat slow to come up again, and in some cases not even re-establish itself at all.
This might happen if your backup connection, (or even your main connection), is over a cellular data link and subject to intermittent packet loss due to variations in signal strength, or if your IPSEC traffic is suffering some kind of interference from intermediate network hardware, such as going over a IPv4 connection that has a many to one NAT applied to it by your ISP.
Luckily, there are ways to improve performance and reliability in this regard.
Reading the manual page for ipsec.conf, naïve users may assume that reducing the dpd_check_interval option in iked.conf would help us here, but infact it often doesn't.
This option is not a simple timeout before sending a keepalive probe packet to the remote peer, (in a similar fashion to the ServerAliveInterval and TCPKeepAlive options in ssh_config). As such, setting it to a low value will not ensure that packets are sent to the remote peer frequently enough to avoid intermediate network hardware from dropping the connection.
The logic behind dpd_check_interval can be found in ikev2.c, in function ikev2_ike_sa_alive, and there are two important things to note about it:
Firstly, in order for a probe packet to be sent, we have to have sent data to the peer but not received a reply. This might sound obvious, but if the tunnel is idle with no data needing to be sent across it, the local peer will not check that it remains up.
This is clearly an issue for incoming SMTP over a tunnel which is initially set up by the local peer, (because the local peer has a dynamic IP address), but will then sit idle waiting for incoming connections from the server.
Secondly, the actual timeout threshold is defined by the constant IKED_IKE_SA_LAST_RECVD_TIMEOUT, (defaulting to 300 seconds, or five minutes), and not defined by dpd_check_interval, which actually just defines the frequency at which the local peer calls the ikev2_ike_sa_alive function to do the local checks, (this is set up in function ikev2_enable_timer).
set dpd_check_interval 20
Adding this to iked.conf won't help very much at all
One way to ensure that isakmp data actually flows between the peers on a regular basis is to considerably reduce the re-keying intervals:
carinho# cat /etc/iked.conf
ikev2 active esp proto tcp from vpn0.example.com port 25 to vpn1.example.com peer namorar.example.com ikelifetime 30m lifetime 30 ecdsa384
ikev2 active esp proto tcp from vpn0.example.com to vpn1.example.com port 25 peer namorar.example.com ecdsa384
ikev2 active esp proto icmp6 from vpn0.example.com to vpn1.example.com peer namorar.example.com ecdsa384
With the above changes, packets will be sent out from our local machine every 30 seconds which should be sufficient to avoid problems with NAT performed by the ISP. Additionally, whilst we're here, we set up a re-keying interval of 30 minutes for the IKE SA.
Unfortunately, there are two issues with this approach. Not only is such frequent re-keying un-necessarily computationally expensive, but even with this arrangement we have seen instances where after a loss of tunnel connectivity, the subsequent re-negotiation of the tunnel repeatedly starts, but never completes.
Ultimately, we've found that the most reliable way to keep IPSEC tunnels up and data flowing during our tests has always been to simply run a continuous slow ping across the tunnel itself:
carinho$ ping -i 5 vpn1.example.com
Togther with a frequent re-keying interval, this slow ping ensures that ESP traffic as well as ISAKMP traffic is generated with sufficient frequency to avoid problems with NAT at the ISP in the majority of cases.
Bringing up the IPSEC tunnel
At this point, we should have our keys generated and put in place, and our iked.conf files correctly prepared. Now we can move on to bringing up the IPSEC tunnel for the first time.
Starting iked
Automatically or manually...
Start iked on every boot
# echo "iked_flags=" >> /etc/rc.conf.local
Start iked interactively
# iked -dv
Eventually, you will probably want to start iked automatically on every boot. This can be done simply by adding a single line to /etc/rc.conf.local.
However, for testing purposes, we can invoke iked from the command line and set it to log to the terminal rather than /var/log/daemon.
Having started iked on both peers, if everything is correctly configured then the local peer should start to produce a lot of debugging output on the terminal, (assuming that we started iked manually with the -v flag).
The quantity of output might appear somewhat bewildering for new users, but it's quite easy to understand if we divide it into sections.
If your output is completely different, see the following section, ‘but it didn't work for me!’, for suggestions on how to diagnose the problem.
The first block of output, we see the policies that we set in the configuration file echoed back to us, with the output has been expanded to include any parameters that were not explicitly set and have therefore taken a default value.
In this case, it's just the three policies from the example /etc/iked.conf given above:
Long lines have been wrapped
ikev2 "policy1" active tunnel esp proto tcp inet6 from 2001:db8:c1da::1 port 25 to 2001:db8:c1da::2 local any peer 2001:db8:1234:1234::c155 ikesa enc aes-128-gcm enc aes-256-gcm prf hmac-sha2-256 prf hmac-sha2-384 prf hmac-sha2-512 prf hmac-sha1 group curve25519 group ecp521 group ecp384 group ecp256 group modp4096 group modp3072 group modp2048 group modp1536 group modp1024 ikesa enc aes-256 enc aes-192 enc aes-128 enc 3des prf hmac-sha2-256 prf hmac-sha2-384 prf hmac-sha2-512 prf hmac-sha1 auth hmac-sha2-256 auth hmac-sha2-384 auth hmac-sha2-512 auth hmac-sha1 group curve25519 group ecp521 group ecp384 group ecp256 group modp4096 group modp3072 group modp2048 group modp1536 group modp1024 childsa enc aes-128-gcm enc aes-256-gcm group none esn noesn childsa enc aes-256 enc aes-192 enc aes-128 auth hmac-sha2-256 auth hmac-sha2-384 auth hmac-sha2-512 auth hmac-sha1 group none esn noesn lifetime 10800 bytes 4294967296 ecdsa384
ikev2 "policy2" active tunnel esp proto tcp inet6 from 2001:db8:c1da::1 to 2001:db8:c1da::2 port 25 local any peer 2001:db8:1234:1234::c155 ikesa enc aes-128-gcm enc aes-256-gcm prf hmac-sha2-256 prf hmac-sha2-384 prf hmac-sha2-512 prf hmac-sha1 group curve25519 group ecp521 group ecp384 group ecp256 group modp4096 group modp3072 group modp2048 group modp1536 group modp1024 ikesa enc aes-256 enc aes-192 enc aes-128 enc 3des prf hmac-sha2-256 prf hmac-sha2-384 prf hmac-sha2-512 prf hmac-sha1 auth hmac-sha2-256 auth hmac-sha2-384 auth hmac-sha2-512 auth hmac-sha1 group curve25519 group ecp521 group ecp384 group ecp256 group modp4096 group modp3072 group modp2048 group modp1536 group modp1024 childsa enc aes-128-gcm enc aes-256-gcm group none esn noesn childsa enc aes-256 enc aes-192 enc aes-128 auth hmac-sha2-256 auth hmac-sha2-384 auth hmac-sha2-512 auth hmac-sha1 group none esn noesn lifetime 10800 bytes 4294967296 ecdsa384
ikev2 "policy3" active tunnel esp proto ipv6-icmp inet6 from 2001:db8:c1da::1 to 2001:db8:c1da::2 local any peer 2001:db8:1234:1234::c155 ikesa enc aes-128-gcm enc aes-256-gcm prf hmac-sha2-256 prf hmac-sha2-384 prf hmac-sha2-512 prf hmac-sha1 group curve25519 group ecp521 group ecp384 group ecp256 group modp4096 group modp3072 group modp2048 group modp1536 group modp1024 ikesa enc aes-256 enc aes-192 enc aes-128 enc 3des prf hmac-sha2-256 prf hmac-sha2-384 prf hmac-sha2-512 prf hmac-sha1 auth hmac-sha2-256 auth hmac-sha2-384 auth hmac-sha2-512 auth hmac-sha1 group curve25519 group ecp521 group ecp384 group ecp256 group modp4096 group modp3072 group modp2048 group modp1536 group modp1024 childsa enc aes-128-gcm enc aes-256-gcm group none esn noesn childsa enc aes-256 enc aes-192 enc aes-128 auth hmac-sha2-256 auth hmac-sha2-384 auth hmac-sha2-512 auth hmac-sha1 group none esn noesn lifetime 10800 bytes 4294967296 ecdsa384
Next, we see the local peer sending the initial requests for each policy to the remote peer:
ikev2_init_ike_sa: initiating "policy1"
spi=0xa19af796130924cd: send IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500, 502 bytes
ikev2_init_ike_sa: initiating "policy2"
spi=0x43eef3cf1590a9a5: send IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500, 502 bytes
ikev2_init_ike_sa: initiating "policy3"
spi=0x5d7555acad9cbac7: send IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500, 502 bytes
If the isakmp packets are successfully flowing in both directions, we should now see acknowledgements of those requests from the remote peer:
spi=0xa19af796130924cd: recv IKE_SA_INIT res 0 peer 2001:db8:1234:1234::c155:500 local 2001:db8:dd::dd:500, 219 bytes, policy 'policy1'
spi=0x43eef3cf1590a9a5: recv IKE_SA_INIT res 0 peer 2001:db8:1234:1234::c155:500 local 2001:db8:dd::dd:500, 219 bytes, policy 'policy2'
spi=0x5d7555acad9cbac7: recv IKE_SA_INIT res 0 peer 2001:db8:1234:1234::c155:500 local 2001:db8:dd::dd:500, 219 bytes, policy 'policy3'
After that, authentication information is exchanged:
The local peer sends authentication information:
spi=0xa19af796130924cd: send IKE_AUTH req 1 peer 2001:db8:1234:1234::c155:500 local 2001:db8:dd::dd:500, 576 bytes
spi=0x43eef3cf1590a9a5: send IKE_AUTH req 1 peer 2001:db8:1234:1234::c155:500 local 2001:db8:dd::dd:500, 575 bytes
spi=0x5d7555acad9cbac7: send IKE_AUTH req 1 peer 2001:db8:1234:1234::c155:500 local 2001:db8:dd::dd:500, 575 bytes
And then receives authentication from the remote peer:
spi=0xa19af796130924cd: recv IKE_AUTH res 1 peer 2001:db8:1234:1234::c155:500 local 2001:db8:dd::dd:500, 467 bytes, policy 'policy1'
spi=0x43eef3cf1590a9a5: recv IKE_AUTH res 1 peer 2001:db8:1234:1234::c155:500 local 2001:db8:dd::dd:500, 468 bytes, policy 'policy2'
spi=0x5d7555acad9cbac7: recv IKE_AUTH res 1 peer 2001:db8:1234:1234::c155:500 local 2001:db8:dd::dd:500, 468 bytes, policy 'policy3'
Finally, we see a confirmation that the policies have been established, and that the tunnel is ready for user data to flow across it:
spi=0xa19af796130924cd: ikev2_childsa_enable: loaded SPIs: 0xe03e552b, 0x04c66c48 (enc aes-128-gcm esn)
spi=0xa19af796130924cd: ikev2_childsa_enable: loaded flows: ESP-2001:db8:c1da::1/0=2001:db8:c1da::2/0(6)
spi=0xa19af796130924cd: established peer 2001:db8:1234:1234::c155:500[FQDN/namorar.example.com] local 2001:db8:dd::dd:500[FQDN/carinho.lan] policy 'policy1' as initiator (enc aes-128-gcm group curve25519 prf hmac-sha2-256)
spi=0x43eef3cf1590a9a5: ikev2_childsa_enable: loaded SPIs: 0x804a1ae2, 0x5ecf4532 (enc aes-128-gcm esn)
spi=0x43eef3cf1590a9a5: ikev2_childsa_enable: loaded flows: ESP-2001:db8:c1da::1/0=2001:db8:c1da::2/0(6)
spi=0x43eef3cf1590a9a5: established peer 2001:db8:1234:1234::c155:500[FQDN/namorar.example.com] local 2001:db8:dd::dd:500[FQDN/carinho.lan] policy 'policy2' as initiator (enc aes-128-gcm group curve25519 prf hmac-sha2-256)
spi=0x5d7555acad9cbac7: ikev2_childsa_enable: loaded SPIs: 0x3eeb8769, 0x0dd82693 (enc aes-128-gcm esn)
spi=0x5d7555acad9cbac7: ikev2_childsa_enable: loaded flows: ESP-2001:db8:c1da::1/0=2001:db8:c1da::2/0(58)
spi=0x5d7555acad9cbac7: established peer 2001:db8:1234:1234::c155:500[FQDN/namorar.example.com] local 2001:db8:dd::dd:500[FQDN/carinho.lan] policy 'policy3' as initiator (enc aes-128-gcm group curve25519 prf hmac-sha2-256)
Debug output from the remote peer
The output from the remote peer will be similar, and follow the same pattern.
First, we see the single policy defined in /etc/iked.conf repeated back to us, as before:
Long line has been wrapped
ikev2 "policy1" passive tunnel esp inet6 from 2001:db8:c1da::2 to 2001:db8:c1da::1 local 2001:db8:1234:1234::c155 peer any ikesa enc aes-128-gcm enc aes-256-gcm prf hmac-sha2-256 prf hmac-sha2-384 prf hmac-sha2-512 prf hmac-sha1 group curve25519 group ecp521 group ecp384 group ecp256 group modp4096 group modp3072 group modp2048 group modp1536 group modp1024 ikesa enc aes-256 enc aes-192 enc aes-128 enc 3des prf hmac-sha2-256 prf hmac-sha2-384 prf hmac-sha2-512 prf hmac-sha1 auth hmac-sha2-256 auth hmac-sha2-384 auth hmac-sha2-512 auth hmac-sha1 group curve25519 group ecp521 group ecp384 group ecp256 group modp4096 group modp3072 group modp2048 group modp1536 group modp1024 childsa enc aes-128-gcm enc aes-256-gcm group none esn noesn childsa enc aes-256 enc aes-192 enc aes-128 auth hmac-sha2-256 auth hmac-sha2-384 auth hmac-sha2-512 auth hmac-sha1 group none esn noesn dstid carinho.lan lifetime 10800 bytes 4294967296 ecdsa384
Since the remote VM is the passive peer, the negotiation starts with a received request from the local peer, which is promptly replied to.
spi=0x526fe3d096cc3eb5: recv IKE_SA_INIT req 0 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 502 bytes, policy 'policy1'
spi=0x526fe3d096cc3eb5: send IKE_SA_INIT res 0 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 219 bytes
spi=0x6f0f49d9495e60f6: recv IKE_SA_INIT req 0 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 502 bytes, policy 'policy1'
spi=0x6f0f49d9495e60f6: send IKE_SA_INIT res 0 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 219 bytes
spi=0xdaa38c7f4c38ade4: recv IKE_SA_INIT req 0 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 502 bytes, policy 'policy1'
spi=0xdaa38c7f4c38ade4: send IKE_SA_INIT res 0 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 219 bytes
Note that each of the IKE_SA_INIT requests matches policy1, the only policy defined on this peer.
Authentication information is exchanged as before, and the peer is established, matching policy1 on the VM:
spi=0x526fe3d096cc3eb5: recv IKE_AUTH req 1 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 577 bytes, policy 'policy1'
spi=0x526fe3d096cc3eb5: send IKE_AUTH res 1 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 468 bytes
spi=0x526fe3d096cc3eb5: ikev2_childsa_enable: loaded SPIs: 0x8c4eab0f, 0x3d9d366a (enc aes-128-gcm esn)
spi=0x526fe3d096cc3eb5: ikev2_childsa_enable: loaded flows: ESP-2001:db8:c1da::2/0=2001:db8:c1da::1/0(0)
spi=0x526fe3d096cc3eb5: established peer 2001:db8:dd::dd:500[FQDN/carinho.lan] local 2001:db8:1234:1234::c155:500[FQDN/namorar.example.com] policy 'policy1' as responder (enc aes-128-gcm group curve25519 prf hmac-sha2-256)
Finally, we see the second and third policies on the local peer being authenticated and negotiated by the remote VM. In this case the messages regarding the establishment of the last two policies are interleaved. This is simply due to timing differences, network lag, or other spurious factors, and it doesn't matter at all from a diagnostic or debugging point of view.
spi=0x6f0f49d9495e60f6: recv IKE_AUTH req 1 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 577 bytes, policy 'policy1'
spi=0xdaa38c7f4c38ade4: recv IKE_AUTH req 1 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 576 bytes, policy 'policy1'
spi=0x6f0f49d9495e60f6: send IKE_AUTH res 1 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 468 bytes
spi=0x6f0f49d9495e60f6: ikev2_childsa_enable: loaded SPIs: 0x62822c59, 0x917b9a93 (enc aes-128-gcm esn)
spi=0x6f0f49d9495e60f6: ikev2_childsa_enable: loaded flows: ESP-2001:db8:c1da::2/0=2001:db8:c1da::1/0(0)
spi=0x6f0f49d9495e60f6: established peer 2001:db8:dd::dd:500[FQDN/carinho.lan] local 2001:db8:1234:1234::c155:500[FQDN/namorar.example.com] policy 'policy1' as responder (enc aes-128-gcm group curve25519 prf hmac-sha2-256)
spi=0xdaa38c7f4c38ade4: send IKE_AUTH res 1 peer 2001:db8:dd::dd:500 local 2001:db8:1234:1234::c155:500, 467 bytes
spi=0xdaa38c7f4c38ade4: ikev2_childsa_enable: loaded SPIs: 0x8a87e98b, 0x20b37742 (enc aes-128-gcm esn)
spi=0xdaa38c7f4c38ade4: ikev2_childsa_enable: loaded flows: ESP-2001:db8:c1da::2/0=2001:db8:c1da::1/0(0)
spi=0xdaa38c7f4c38ade4: established peer 2001:db8:dd::dd:500[FQDN/carinho.lan] local 2001:db8:1234:1234::c155:500[FQDN/namorar.example.com] policy 'policy1' as responder (enc aes-128-gcm group curve25519 prf hmac-sha2-256)
The important thing to see in each case is the final, ‘established peer’ message.
At this point, the corresponding traffic as defined in the relevant policies should be able to flow across the tunnel.
Testing the tunnel
At this point, we can test the connectivity over the tunnel just as if it was any other regular network connection.
Since we included a policy that allows ICMP packets through, we can try pinging each host from the other:
carinho# ping6 vpn1.example.com
PING vpn1.example.com (2001:db8:c1da::2): 56 data bytes
64 bytes from 2001:db8:c1da::2: icmp_seq=0 hlim=64 time=270.801 ms
64 bytes from 2001:db8:c1da::2: icmp_seq=1 hlim=64 time=278.679 ms
64 bytes from 2001:db8:c1da::2: icmp_seq=2 hlim=64 time=312.592 ms
64 bytes from 2001:db8:c1da::2: icmp_seq=3 hlim=64 time=270.840 ms
--- vpn1.example.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 270.801/283.228/312.592/17.254 ms
Pinging the remote peer from the local peer.
namorar# ping6 vpn0.example.com
PING vpn0.example.com (2001:db8:c1da::1): 56 data bytes
64 bytes from 2001:db8:c1da::1: icmp_seq=0 hlim=64 time=284.213 ms
64 bytes from 2001:db8:c1da::1: icmp_seq=1 hlim=64 time=252.207 ms
64 bytes from 2001:db8:c1da::1: icmp_seq=2 hlim=64 time=250.453 ms
64 bytes from 2001:db8:c1da::1: icmp_seq=3 hlim=64 time=252.667 ms
--- vpn0.example.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 250.453/259.885/284.213/14.070 ms
Pinging the local peer from the remote peer.
Having tested ICMP connectivity, the next step is to try making a TCP connection.
By default, smtpd will not be bound to port 25 of the IP for the VPN endpoint. So unless we've already changed the default /etc/mail/smtpd.conf on the VM we can test TCP connectivity using netcat to listen on that port, then connecting to it from the local machine:
namorar# nc -l vpn1.example.com 25
carinho# nc -v vpn1.example.com 25
Connection to vpn1.example.com (2001:db8:c1da::2) 25 port [tcp/smtp] succeeded!
Outbound
And of course, most importantly, it works in the reverse direction too, since making inbound connections to TCP port 25 is the main purpose of this tunnel:
carinho# nc -l vpn0.example.com 25
namorar# nc -v vpn0.example.com 25
Connection to vpn0.example.com (2001:db8:c1da::1) 25 port [tcp/smtp] succeeded!
Inbound
At this point, we have the connectivity we need via the IPSEC tunnel to be able to configure smtpd on both machines to relay inbound mail from the internet by opening smtp connections to our local machine.
Even if it's connected to the internet via a dynamic IP, behind firewalls, NAT, or other tedious networking hardware at the ISP, mail delivery should work just fine.
But it doesn't work for me!
Since even a slight mis-configuration can break the setup and stop things from working, we'll cover some basic debugging in this section.
So if you didn't see the correct output from iked, specifically the final, ‘established peer’, messages, then don't worry.
First, we'll consider the case where the remote peer is down, or we have a network connectivity issue preventing the ISAKMP packets being exchanged on UDP port 500. In this instance, we'd see messages such as the following from iked on the local, active peer:
ikev2_init_ike_sa: initiating "policy1"
spi=0xf84dad5836e092ba: send IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500, 502 bytes
ikev2_init_ike_sa: initiating "policy2"
spi=0xb5f0b6b75d2a8c29: send IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500, 502 bytes
ikev2_init_ike_sa: initiating "policy3"
spi=0x837c5a94946136ac: send IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500, 502 bytes
spi=0xf84dad5836e092ba: retransmit 1 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0xb5f0b6b75d2a8c29: retransmit 1 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0x837c5a94946136ac: retransmit 1 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0xf84dad5836e092ba: retransmit 2 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0x837c5a94946136ac: retransmit 2 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0xb5f0b6b75d2a8c29: retransmit 2 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0xf84dad5836e092ba: retransmit 3 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0xb5f0b6b75d2a8c29: retransmit 3 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0x837c5a94946136ac: retransmit 3 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0xf84dad5836e092ba: retransmit 4 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0x837c5a94946136ac: retransmit 4 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0xb5f0b6b75d2a8c29: retransmit 4 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0xb5f0b6b75d2a8c29: retransmit 5 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0x837c5a94946136ac: retransmit 5 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0xf84dad5836e092ba: retransmit 5 IKE_SA_INIT req 0 peer 2001:db8:1234:1234::c155:500 local :::500
spi=0xf84dad5836e092ba: sa_free: retransmit limit reached
spi=0x837c5a94946136ac: sa_free: retransmit limit reached
spi=0xb5f0b6b75d2a8c29: sa_free: retransmit limit reached
Isakmp packets are not being received by the active peer.
These messages will repeat endlessly, with different spi identifiers, as iked repeatedly re-tries to establish the tunnel.
The remote peer will log nothing at all by default, as it waits for the incoming connection. Even invoking iked with the -v flag, all we will see is the first line confirming the policy configuration, and nothing more:
Long line has been wrapped
ikev2 "policy1" passive tunnel esp inet6 from 2001:db8:c1da::2 to 2001:db8:c1da::1 local 2001:db8:1234:1234::c155 peer any ikesa enc aes-128-gcm enc aes-256-gcm prf hmac-sha2-256 prf hmac-sha2-384 prf hmac-sha2-512 prf hmac-sha1 group curve25519 group ecp521 group ecp384 group ecp256 group modp4096 group modp3072 group modp2048 group modp1536 group modp1024 ikesa enc aes-256 enc aes-192 enc aes-128 enc 3des prf hmac-sha2-256 prf hmac-sha2-384 prf hmac-sha2-512 prf hmac-sha1 auth hmac-sha2-256 auth hmac-sha2-384 auth hmac-sha2-512 auth hmac-sha1 group curve25519 group ecp521 group ecp384 group ecp256 group modp4096 group modp3072 group modp2048 group modp1536 group modp1024 childsa enc aes-128-gcm enc aes-256-gcm group none esn noesn childsa enc aes-256 enc aes-192 enc aes-128 auth hmac-sha2-256 auth hmac-sha2-384 auth hmac-sha2-512 auth hmac-sha1 group none esn noesn dstid carinho.lan lifetime 10800 bytes 4294967296 ecdsa384
The passive peer will stop here, waiting for isakmp packets that never arrive.
In this case, it's probably easiest to use tcpdump on the remote peer to monitor inbound traffic on UDP port 500, whilst adjusting firewall rules and testing for connectivity with netcat.
Less commonly, isakmp traffic might be able to get through to the remote peer, but ESP traffic will find itself blocked. In this case, the IPSEC tunnel will be successfully brought up and we'll see the ‘established peer’ messages in the logs at both ends. However, trying to make connections or send data through the tunnel will fail.
A tell-tale sign that this is indeed the problem in hand are icmp port unreachable messages which, will be reported by tcpdump:
09:39:14.557258 2001:db8:dd::dd > 2001:db8:1234:1234::c155 esp spi 0xc2eefa9b seq 146 len 140
09:39:14.822434 2001:db8:1234:1234::c155 > 2001:db8:dd::dd: icmp6: 2001:db8:1234:1234::c155 protocol 50 port 64155 unreachable
Typical output of tcpdump showing ESP packets being blocked from reaching the remote VM.
This is likely due to a firewall blocking ESP traffic, but it could also possibly be a broken or misconfigured router or other piece of network somewhere between the two peers.
Firewall notes
If we're running our firewalls in a ‘block inbound by default’ mode, and only allowing required inbound traffic through, (which is usually a good idea), then we shouldn't need any special configuration on the local, active peer, to send out the initial IKE_SA_INIT requests, nor to receive the ISAKMP replies back, as state will have been created for them by the corresponding outbound packets.
Although the local peer will always initiate the ISAKMP exchange, either peer could be the first with user payload data to send over ESP. We should, therefore, explicitly allow ESP in from the remote peer, as there is no guarantee that state will have been created for this from ESP packets having already been sent out.
To do this, we can use a rule such as the following, taking care to remember that the ESP traffic is coming from the real public IP of the peer and not the VPN endpoint address:
pass in quick on if0 proto esp from namorar.example.com
In this example, we're using the fictitious interface specifier 'if0', which should obviously be replaced with the real, (hardware-dependent), interface identifier.
Shell trivia
How to discover the interface identifier in a script
Most users are probably either aware of what hardware is in their machines, or know how to identify it from the boot-time dmesg output, making the task of substituting if0 for the correct interface identifier fairly trivial.
However, there might be occasions when you want to do this from within a shell script. This can be done by using the get function of the route command:
route get -inet6 namorar.example.com | grep interface | cut -d ':' -f 2 | cut -c 2-
The -inet6 modifier is only required if a symbolic hostname is provided that resolves to an IPv6 address. Literal IPv6 addresses will be handled automatically.
On the remote peer, we need to allow both ISAKMP and ESP traffic in.
This can be done using two pass rules, again remembering to use the real public IP address and not the VPN endpoint IP.
pass in on vio0 inet6 proto udp to 2001:db8:1234:1234::c155 port isakmp
pass in on vio0 inet6 proto esp to 2001:db8:1234:1234::c155
Use the real public IP addresses!
When setting up your firewall ruleset at the same time as configuring the IPSEC tunnel, it's important to check that that the new ruleset works after state has been cleared.
Otherwise, it's very possible that things might appear to work just fine until the first reboot, only to then mysteriously fail. This can happen if ESP traffic has been allowed to flow between the peers whilst the firewall was in a permissive configuration, thus creating state during the initial testing.
Adding rules afterwards that block ESP between the same addresses will not have any effect whilst the state table still holds an entry for ESP traffic between those two peers.
Flushing the firewall state table is simple enough, using the ‘-F states’ option to pfctl:
# pfctl -F states
Flush the state table after making the firewall ruleset more restrictive.
Configuring smtpd
Now that our IPSEC tunnel is, (hopefully), working correctly, we can move on to configuring smtpd.
Since our smtp connections will be encrypted by the ESP encapsulation, we could simply run the plain text smtp protocol over the VPN and still avoid third parties from intercepting the data. This was, after all, one of the design considerations of IPSEC, to allow existing protocols to be secured in a standardised way, rather than implementing encryption directly into each protocol separately. An smtp over TLS configuration is more complicated, so if you want to start by configuring plain text smtp over the tunnel just to see it all working, fair enough.
However, there are several reasons to prefer not leaving this as a production setup, and to set up a propper smtp over TLS configuration instead. Firstly, if the opportunity to obtain a publicly accessible static IP for the local machine becomes available in the future and therefore using an IPSEC tunnel becomes un-necessary, it's convenient to have a TLS-based setup already fully tested and working so that the switch can be made immediately. Secondly, future mis-configurations or even as yet undiscovered bugs in the IPSEC code could theoretically result in smtp traffic being sent in the clear over the internet. If our SMTP session uses TLS within the tunnel, the chances of this happening are greatly reduced.
Configuring smtpd on the remote VM
The following configuration file will allow the relay of outbound email to other internet hosts via our remote VM, namorar.example.com, from any user as long as the sending machine, our local mail router carinho.lan, presents any valid TLS certificate and uses the credentials stored in /etc/mail/secrets.
No user-level authentication is done. Of course, the only way to connect to the IP address that listens for these submissions is via the IPSEC tunnel, so further machine-level authentication will also have been done by this point, in the form of verifying the iked keys.
Example configuration file for smtpd on namorar.example.com
# Aliases to expand for local mail delivery only, E.G. automatic messages from daemons.
table aliases file:/etc/mail/aliases
# Permitted sending IPs. Should be smtp.example.com for IPv6, and our only IPv4 address.
table ips_out { 2001:db8:1234:1234::b17e, 2001:db8:1234:1234::b17e, 2001:db8:1234:1234::b17e, 2001:db8:1234:1234::b17e, 2001:db8:1234:1234::b17e, 192.0.2.2 }
# Users that accept inbound email
table valid_users_example_com { test, postmaster }
# Credentials for authentication with carinho.lan over the VPN
table outbound_auth file:/etc/mail/secrets
# Define pki names, certificate and key paths
# mx1.example.com and mx2.example.com are presented to external clients connecting to us to send mail to us
pki mx1.example.com cert "/etc/ssl/private/mx1.example.com.crt"
pki mx1.example.com key "/etc/ssl/private/mx1.example.com.key"
pki mx2.example.com cert "/etc/ssl/private/mx2.example.com.crt"
pki mx2.example.com key "/etc/ssl/private/mx2.example.com.key"
# vpn1.example.com is presented to carinho.lan as our identity when we connect to deliver inbound mail from the internet
# This is what carinho.lan knows the remote end of the VPN as, according to it's hosts file
pki vpn1.example.com cert "/etc/ssl/private/vpn1.example.com.ssc"
pki vpn1.example.com key "/etc/ssl/private/vpn1.example.com.key"
# smtp.example.com is presented to carinho.lan as our identity when carinho.lan connects to us to send mail to the internet
# carinho.lan is actually connecting on the IP that is listed in it's hosts file as vpn1.example.com
pki smtp.example.com cert "/etc/ssl/private/smtp.example.com.crt"
pki smtp.example.com key "/etc/ssl/private/smtp.example.com.key"
# Set queue expiry time to 20 days
queue ttl 20d
# Listen on various interfaces. Listen on socket is also done by default.
# Outbound mail from carinho.lan via the VPN
# Present the hostname smtp.example.com, even though we are really listening on the VPN IP address for vpn1.example.com
# This avoids the VPN details appearing in Received: headers on outbound email. The next relaying step onwards will use
# smtp.example.com as the source IP, and this makes the headers look consistent. However, it does mean that we need to
# present the certificate for smtp.example.com rather than the certificate for vpn1.example.com.
listen on 2001:db8:c1da::2 pki smtp.example.com tls-require verify hostname smtp.example.com mask-src auth <outbound_auth>
# Inbound external mail from the internet via IPv4. DSN disabled to avoid showing details of the internal network.
listen on 192.0.2.2 pki mx2.example.com hostname mx2.example.com no-dsn
# Inbound external mail from the internet via IPv6. DSN disabled to avoid showing details of the internal network.
listen on 2001:db8:1234:1234::6969 pki mx1.example.com hostname mx1.example.com no-dsn
# Delivery actions
# Local mail, E.G. from daemons
action "local_mail" maildir "%{user.directory}/mailspools/in/" alias <aliases>
# Inbound mail from the internet
action "relay_in" relay pki vpn1.example.com host smtp://vpn0.example.com
# Outbound mail from carinho.lan via the VPN
action "relay_out" relay helo smtp.example.com src <ips_out>
# Match rules
# Allow pure local mail for delivery via maildir
match from local for local action "local_mail"
# Inbound mail for example.com to relay via carinho.lan
match from any for domain "example.com" rcpt-to <valid_users_example_com> action "relay_in"
# Outbound mail from carinho.lan for any destination is accepted for relay by us as smtp.example.com
match from src 2001:db8:c1da::1 for any action "relay_out"
We specify our IPv6 address multiple times in ips_out, because otherwise smtpd will frequently fall back to using IPv4 when talking to a dual-stacked host.
Since there is no obvious way to force a preference for IPv6 in these cases, specifying the IPv6 address multiple times manually at least helps to increase the frequency with which IPv6 is used.
Bounces
We set the queue ttl to a much larger value than the default of four days, to avoid mail being permanently deleted if for some reason an outage prevents it being relayed from the remote VM to the local mail server. Unlike an IMAP or POP server which will usually keep stored mail indefinitely, as it is considered to have reached it's final destination and therefore have been delivered, an SMTP server considers it's queued mail to be ‘in transit’. If none of the specified relays can be contacted before the queue ttl expires, the mail will be bounced back to the sender.
With the configuration above, the bounce mails will intentionally fail, with the result that the incoming message is simply deleted without informing the sender. The failed bounce mails will appear in /var/log/maillog with a message such as the following:
Nov 7 16:38:02 namorar smtpd[8493]: warn: PermFail injecting failure report on message 52866920 to <sender@other.host.example.com> for 1 envelope: 550 Invalid recipient: <sender@other.host.example.com>
Warning messages generated as per the configuration of bounce warn-interval will also intentionally fail with a similar error.
If the link between the local smtp server on the lan and the remote VM is not up 24 hours per day, this behaviour with regards to not sending bounce messages is probably desirable. The reason they fail is simply because we don't have a match rule that allows locally generated mail to be relayed to other internet hosts.
If we really did want such bounce messages to be sent to external hosts, this could be done with a single line in smtpd.conf:
match from local for any action "relay_out"
In this case, we would probably also want to check that bounce warn-interval is set to a reasonable value based on the expected level of connectivity between the two peers, to avoid sending excessive warning messages just because the VPN link is down at certain times of the day.
Creating the credentials file
The credentials file on the remote VM, namorar.example.com, simply consists of a username and an encrypted password:
namorar# cat /etc/mail/secrets
mailuser $2b$10$TMCTeP.xVdU8YLGJkyJrTOz6VU0.xWRXFvCp90vGfNGFmvBDLC9P6
Credentials file
The encrypted password string used above can be generated using smtpctl:
namorar# smtpctl encrypt
foobar
$2b$10$TMCTeP.xVdU8YLGJkyJrTOz6VU0.xWRXFvCp90vGfNGFmvBDLC9P6
Generating the encrypted password string
The same file on the local mail relay, carinho.lan, contains the username and unencrypted password, prefixed by an arbitrary label, which in this case is the same as the username:
carinho# cat /etc/mail/secrets
mailuser mailuser:foobar
This time the password is in plain text
The credentials file is stored in a world-readable directory, so it is important to create it with appropriate permissions.
Setting the shell's umask to 027 before creating /etc/mail/secrets will ensure that the file is created with 0640.
Creating the keys and certificates
The remote VM has four identities that we need to create keys and certificates for:
Remote VM identities
  • smtp.example.com
  • mx1.example.com
  • mx2.example.com
  • vpn1.example.com
For correct operation sending mail to and receiving mail from other internet hosts, the first three certificates should be real TLS certificates signed by a real certificate authority. If you're using a free certificate issuing service, that likely means that you will need to renew them every three months.
The certificate for vpn1.example.com will only be used between the local server and the remote VM, so we can simply create a self-signed certificate with a long validity period of ten years, and install it in the root certificate bundle on the local peer.
Creating the encryption keys for smtpd is a similar process to that which we used to create the keys for iked. In this case we only need the private key, so we don't bother to create the matching public key.
Once again, TLS key generation is covered in much more detail in our guide to encryption keys and TLS certificates, so refer there for more information.
namorar# cd /etc/ssl/private
namorar# openssl ecparam -out ec-secp384r1.pem -name secp384r1
namorar# openssl genpkey -paramfile ec-secp384r1.pem -out /etc/ssl/private/vpn1.example.com.key
Creating the encryption keys
The generated key will be written with permissions 0644, so we change this to 0640. Note that since it is created in a directory that is neither world nor group readable, we don't need to concern ourselves with the key file itself being momentarily world-readable between it's creation and the change of permissions.
namorar# chmod 640 /etc/ssl/private/vpn1.example.com.key
Setting suitable permissions
Now we need to create a self-signed certificate using the key we've just generated.
Traditionally, certificates for a single hostname supplied that hostname in the Common Name, or CN field. Modern systems often now require the Subject Alternative Name or SAN field of the certificate to be populated with the hostname too. This is easily done by passing some extra options to openssl, which can be stored in a very small configuration file:
namorar# echo "subjectAltName=DNS:vpn1.example.com" > vpn1.example.com.ext
Creating a configuration file to ensure that the certificate's SAN field is populated
Next, we generate a certificate signing request:
namorar# openssl req -key vpn1.example.com.key -new -out vpn1.example.com.csr
Generating a signing request
This command will prompt for various pieces of information interactively, which we can either fill in or leave blank as desired. The only really important field is the hostname, which in this case should be vpn1.example.com.
If a password is set in the password field, it will need to be entered every time the server is restarted. To avoid this requirement, and allow the server to be restarted automatically, the password field can be left blank.
With the certificate signing request, we can now generate the actual self-signed certificate:
namorar# openssl x509 -sha256 -req -days 3650 -in vpn1.example.com.csr -signkey vpn1.example.com.key -extfile vpn1.example.com.ext -out vpn1.example.com.ssc
Generating the actual certificate
To check the specifications of the certificate that we just created, we can display it in text format:
# openssl x509 -text -noout -in vpn1.example.com.ssc
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
bd:e3:53:66:ea:7d:a2:f0
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN=vpn1.example.com
Validity
Not Before: Nov 23 16:25:52 2021 GMT
Not After : Nov 21 16:25:52 2031 GMT
Subject: CN=vpn1.example.com
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (384 bit)
pub:
04:d5:3c:9a:ed:25:4b:cb:01:dc:3b:dc:6b:58:1d:
04:34:62:29:40:7e:9e:94:3f:8b:86:8c:b6:e4:71:
17:bb:b7:b4:d0:e5:7b:73:6b:cd:66:9e:12:bb:b3:
b9:d1:9a:38:27:a8:bf:c1:16:30:51:30:ef:1f:55:
37:7f:a4:29:a9:3f:e1:0b:14:68:6f:4c:5b:d3:5d:
42:08:c7:56:6c:4f:84:8e:8b:6f:22:25:0d:fd:d5:
23:8e:15:e4:3d:9d:fd
ASN1 OID: secp384r1
NIST CURVE: P-384
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:vpn1.example.com
Signature Algorithm: ecdsa-with-SHA256
30:66:02:31:00:f2:8a:e3:45:00:74:87:68:5e:52:c2:52:4d:
74:3e:df:03:0d:0b:c2:51:39:c0:6b:59:9e:79:71:e7:03:d3:
28:f5:79:5b:45:93:4b:de:33:54:5c:9b:48:ef:d9:01:8b:02:
31:00:e9:5a:1a:a5:fa:d6:0e:22:45:65:44:2f:10:92:6f:69:
36:91:ac:75:61:a2:d9:02:88:1f:c7:77:1c:4e:9b:99:d9:da:
d2:21:e6:87:0f:6d:d4:78:c4:58:f5:cb:4c:b8
Displaying the new certificate in text format
If we intend to use this self-signed certificate as-is, which is perfectly reasonable for vpn1.example.com, we'll need to add the certificate to /etc/ssl/cert.pem on every machine that we want to accept it as valid:
# cat vpn1.example.com.ssc
-----BEGIN CERTIFICATE-----
MIIBijCCAQ+gAwIBAgIJAL3jU2bqfaLwMAoGCCqGSM49BAMCMBsxGTAXBgNVBAMM
EHZwbjEuZXhhbXBsZS5jb20wHhcNMjExMTIzMTYyNTUyWhcNMzExMTIxMTYyNTUy
WjAbMRkwFwYDVQQDDBB2cG4xLmV4YW1wbGUuY29tMHYwEAYHKoZIzj0CAQYFK4EE
ACIDYgAE1Tya7SVLywHcO9xrWB0ENGIpQH6elD+Lhoy25HEXu7e00OV7c2vNZp4S
u7O50Zo4J6i/wRYwUTDvH1U3f6QpqT/hCxRob0xb011CCMdWbE+EjotvIiUN/dUj
jhXkPZ39ox8wHTAbBgNVHREEFDASghB2cG4xLmV4YW1wbGUuY29tMAoGCCqGSM49
BAMCA2kAMGYCMQDyiuNFAHSHaF5SwlJNdD7fAw0LwlE5wGtZnnlx5wPTKPV5W0WT
S94zVFybSO/ZAYsCMQDpWhql+tYOIkVlRC8Qkm9pNpGsdWGi2QKIH8d3HE6bmdna
0iHmhw9t1HjEWPXLTLg=
-----END CERTIFICATE-----
# cat vpn1.example.com.ssc >> /etc/ssl/cert.pem
Adding our self-signed certificate to the local machine's certificate bundle
It would be good practice, although not actually necessary, to add the textual form of the self-signed certificate to /etc/ssl/cert.pem too, following the style of the existing entries in that file.
This whole process of key and certificate generation can be repeated to generate key/certificate pairs for the other identities, smtp.example.com, mx1.example.com, and mx2.example.com, if we're happy to use self-signed certificates for these too.
However, for maximum compatibility and interoperability with other internet mailservers, these identities will ideally use globally recognised CA-signed certificates usable on the internet at large. Such certificates can be obtained from various CAs without charge for the certificate itself, using tools that are included in the OpenBSD base installation, specifically acme-client and httpd.
Details on how to do this are in the above linked guide.
Configuring smtpd on the local mailserver
The configuration of the local mailserver is slightly simpler:
# Example configuration file for smtpd on carinho.example.com
# Set queue expiry time to 20 days
queue ttl 20d
# Define pki names, certificate and key paths
pki carinho.lan key "/etc/ssl/private/carinho.lan.key"
pki carinho.lan cert "/etc/ssl/private/carinho.lan.ssc"
pki vpn0.example.com key "/etc/ssl/private/vpn0.example.com.key"
pki vpn0.example.com cert "/etc/ssl/private/vpn0.example.com.ssc"
# Aliases to expand for local mail delivery only, E.G. automatic messages from daemons.
table aliases file:/etc/mail/aliases
# Credentials for authentication with carinho.lan over the VPN
table secrets file:/etc/mail/secrets
# Listen on various interfaces. Listen on socket is also done by default.
listen on 2001:db8:6969::1 tls-require verify pki carinho.lan mask-src
listen on 2001:db8:c1da::1 pki vpn0.example.com tls-require verify
# Local mail, E.G. from daemons
action "local_mail" maildir "%{user.directory}/mailspools/in/" alias <aliases>
# The local hosts file contains an entry for smtp.example.com directing it to vpn1.example.com
# Therefore, following rule really connects to vpn1.example.com, the VPN address, and not the address of smtp.example.com listed in the public DNS.
# This is done to avoid details of the VPN appearing in Received: headers.
action "outbound_example" relay tls pki vpn0.example.com host smtp+tls://mailuser@smtp.example.com auth <secrets>
action "forward_to_mimando" relay pki carinho.lan tls host smtp://mimando.lan
# Allow pure local mail for delivery via maildir
match from local for local action "local_mail"
match from local for domain "mimando.lan" action "forward_to_mimando"
match from src "[2001:db8:6969::2]" for local action "local_mail"
match from src "[2001:db8:6969::2]" for any mail-from regex ".*@example.com$" action "outbound_example"
match from any for domain "example.com" action "forward_to_mimando"
match from local for domain "example.com" action "forward_to_mimando"
Typical local mailserver configuration
The relevant keys and certificates can be created using the process described above for the remote VM. We only need to use self-signed certificates on the local server, as it is only speaking SMTP to our other machines and not to other hosts on the internet at large.
The hosts file on the local server should contain an entry for smtp.example.com with the IP address for vpn1.example.com:
carinho# grep smtp.example.com /etc/hosts
2001:db8:c1da::2 smtp.example.com
Adding the hostname for the remote end of the vpn to the local machine's hosts file
This is simply to avoid details of the VPN appearing in the Received: headers of outbound email.
If you only have one local machine...
If the local mailserver is in fact also your local workstation, where you want incoming mail to be finally delivered to, then you can simply change the match lines to:
match from local for local action "local_mail"
match from local for any mail-from regex ".*@example.com$" action "outbound_example"
match from any for domain "example.com" action "mail_from_example"
and add another delivery action:
action "mail_from_example" maildir "%{user.directory}/mailspools/in/" alias <aliases_example>
as well as create an appropriate aliases table, mapping usernames at example.com to corresponding local usernames that should receive the mail.
At this point, you should be finished, and both inbound and outbound email delivery using SMTP over the IPSEC tunnel should be working.
Configuring smtpd on the local workstation
The configuration of smtpd on the local workstation is very straightforward:
table aliases file:/etc/mail/aliases
table aliases_example file:/etc/mail/aliases_example
pki mimando.lan key "/etc/ssl/private/mimando.lan.key"
pki mimando.lan cert "/etc/ssl/private/mimando.lan.ssc"
listen on 2001:db8:6969::2 tls-require verify pki mimando.lan
action "local_mail" maildir "%{user.directory}/maildir/in/" alias <aliases>
action "local_mail_example" maildir "%{user.directory}/maildir/in/" alias <aliases_example>
action "outbound" relay host smtp://carinho.lan pki mimando.lan
match for local action "local_mail"
match for any action "outbound"
match from src "[2001:db8:6969::1]" for local action "local_mail"
match from src "[2001:db8:6969::1]" for domain "example.com" action "local_mail_example"
Example configuration file for smtpd on mimando.example.com
The /etc/mail/aliases_example file just needs to contain mappings between users at the example.com domain, and local users on mimando.lan.
For example, if there is a single local user with username jay, who should receive mail for test@example.com, and postmaster@example.com, we would create a simple aliases_example file with just two lines:
mimando# cat /etc/mail/aliases_example
test jay
postmaster jay
Mail for test@example.com and postmaster@example.com will be delivered to the local user jay
At this point, we should be done!
Let's enjoy real SMTP mail delivery without the expense of an internet connection with a static IP address!
Tweaking smtpd for SMTP connections that are not available 24/7
By default, smtpd waits an increasing amount of time between successive delivery attempts. This is reasonable behaviour when delivering to most SMTP servers, as publicly accessible SMTP servers on the internet are generally expected to have good network connectivity and good uptime. If you intend to keep your IPSEC tunnel open 24 hours a day, and your connectivity is reliable enough to do this, you might not need to change smtpd's scheduling behaviour at all.
However, if your IPSEC tunnel is likely to be more intermittent, it's useful to increase the frequency of delivery attempts so that queued mail comes through the tunnel promptly.
Intermittent connectivity could be due to reasons as simple as only bringing the link up during business hours, or the general un-reliability of home broadband connections.
To increase the frequency of delivery attempts, we can edit the file /usr/src/usr.sbin/smtpd/scheduler_ramqueue.c, then re-compile and restart smtpd:
# cd /usr/src/usr.sbin/smtpd
# vi scheduler_ramqueue.c
# make clean
# make obj
# make
# make install
# /etc/rc.d/smtpd restart
The changes you will probably want to make are to the define for BACKOFF_TRANSFER, which defaults to 400 seconds, and the function scheduler_backoff(), removing one of the multiplications by step from the returned value, so that re-delivery attempts are made at constant rather than increasing intervals.
Final notes and observations
We've tested this system of using an IPSEC tunnel between a local server and a VM at OpenBSD Amsterdam to allow mail delivery over inbound SMTP connections for over a year now, and at least for us it has proven reliable handling hundreds of emails per day. Performance seems quite reasonable, even over a 4G cellular internet connection.
One of the advantages of this particular setup, is that the workstation, mimando.lan in our example, does not need direct internet access in order to send and receive internet email, as it only submits to and receives from carinho.lan.