ExoticSilicon.com - writing a greylisting filter in ~1000 lines of C

Overview of the filtering API

The filtering API is documented in the smtp-filters manual page. You will probably want to read that manual in conjunction with this article, but in essence all we are doing is passing pipe-delimited text data backwards and forwards over the standard input and output file descriptors.

So we receive lines such as:

and we send back lines such as:

And that's really about as complicated as it gets.

Of course, we need some logic to keep track of different concurrent sessions and to actually do any useful filtering based on the content that we receive, but the point here is that the API communications protocol itself is just pipe-delimited text.

The filter program is started by smtpd and expected to continue running until smtpd itself terminates. If the filter terminates whilst smtpd is still running then this is considered an error condition and smtpd will exit.

As explained in the manual page, at startup the filters first receive some config lines from smtpd. In practice, most filters won't need to parse any of this information apart from waiting for the final config|ready line. At this point, the filter needs to indicate to smtpd which 'events' it wants to be notified of.

The list of required events to be reported will obviously depend on the nature of the filtering that the filter program is doing, although some events such as link-connect and link-disconnect will almost always be required in order for the filter to keep track of the currently active sessions.

Fun fact!

Timeouts

The manual page doesn't make absolutely clear whether a client that is disconnected due to a timeout will generate just a 'timeout' event or both a 'timeout' and a 'link-disconnect'. In fact, both events are generated, so unless we want our filter to take specific action in the case of a timeout, it's sufficient to just monitor 'link-disconnect' events for disconnection purposes.

After the initial configuration and setup phase, each line sent by smtpd and received by the filter represents an 'event'.
There are two types of event, 'report' events which are just informational and don't require a response from the filter, and 'filter' events which will cause smtpd to wait until a corresponding reply is received from the filter.
The format of the event lines is very consistent, even between the two types of event. From here on in this article I'll refer to the fields by their position with numbering beginning at zero, because this is the way that we'll reference them in the C code.
Field zero indicates the type of event, with a literal, 'report', or, 'filter'.
Fields one through to five are common between both types of event. The only difference here is in the documentation, which calls field four the 'event' field when talking about report events, and the 'filtering phase' field when talking about filter events. In practice, this is just two different names for the same thing, a field that indicates exactly which event we are receiving information about.
Field five is particularly important as this contains the session identifier. If smtpd is handling multiple smtp transactions at once, then the input to our filter will most likely consist of events from all of the sessions interleaved together. It's the responsibility of the filter to keep track of the state of each session and process input lines according to the appropriate one.
In the case of filter events, field six is also important as it's a token that needs to be passed back to smtpd along with the session ID. This mechanism helps to ensure that a buggy filter doesn't send erroneous data intended for one session in response to a different one. Report events don't include a token because they don't expect a response from the filter, so the token would have no purpose.

Implementation detail

Session identifiers and tokens

Strictly speaking, there is nothing in the documentation for OpenSMTPD that specifies a particular format for either the session ID, (field 5), or the opaque token, (field 6 in report events).

However, internally smtpd represents both of these values as uint64_t values, (see lka_filter.c, for example), and they are converted to and from a 16 digit hex representation when used with external filters, (see filter_protocol_query() in lka_filter.c).

It seems very unlikely that this representation will be changed, so the filter code presented in this article will assume that both session IDs and opaque tokens are fixed-length 16 byte strings. This will allow us to make some optimisations in the C code compared with processing these values as arbitrary length strings.

If you intend to write filters for widespread deployment on production systems and are concerned that this assumption might cause them to break in the future, it would be good practice to check either or both of the version numbers supplied by the API. The protocol version number is returned in field one of each event line, and the host smtpd version is returned as part of the configuration information sent by smtpd during the initial handshake. Checking these values is a much safer way to ensure complete API-level compatibility, rather than making attempts to second-guess what specific changes might happen in future releases.

Another detail to be aware of is that although the fields are pipe-delimited, in the last field, the pipe character is treated as a regular character and doesn't further delimit the line of input. Since different events have different total numbers of fields, knowing which field is the last field depends on knowing what sort of event we are processing. The upshot of this is that we can't simply apply the delimiting logic up to a particularly numbered field and then stop. In practice, though, this situation is very easy to deal with in C as we will discover shortly when we look at the code to implement the parser.

That concludes our brief look at the API protocol. For more details, refer to the smtpd-filters manual page.

Capturing sample output

If you want to capture output similar to the above from your own system, (which might be useful if you are using a different version of OpenSMTPD or have an unusual configuration), this can be done using a short shell script invoked as a primitive filter:

#!/bin/sh

echo "register|report|smtp-in|link-connect"

echo "register|filter|smtp-in|data-line"

echo "register|ready"

cat > /tmp/filter_dialogue

Obviously additional register lines can be added if you want to capture specific events, however note that any report events, (including the data-line one above), will cause the mail system to hang at that point since our shell script doesn't provide the expected replies.

If this script is placed in /usr/local/libexec/smtpd/test_filter, then a suitable reference to it can be added to /etc/mail/smtpd.conf like so:

filter test_filter proc-exec test_filter

Then assuming that your local mail program submits outbound email via a local socket:

listen on socket filter test_filter

Note that smtpd listens on a local socket by default, even if this isn't specified in the configuration file.

But if we want to pass data received via the socket to a filter then we need to explicitly specify this directive.

Curiosity

User, group and chroot options

By default, filters configured via proc-exec will run as the _smtpd user. The smtpd-filters manual page mentions that they can run as other users, but the actual configuration options necessary to do this are not stated.

Looking at the source code for smtpd, specifically for fork_filter_process() in smtpd.c, we can clearly see that code does exist to invoke filters as a custom user running with a custom group, and there is even code to support running filters in a chroot. Reading through parse.y, we find that proc-exec actually accepts the optional arguments user, group, and chroot.

None of these is documented or even mentioned in the syntax definition for proc-exec in the smtpd.conf manual page, although user and group do appear once in the examples section without further explanation.

These elusive options seemed to work as expected during testing, but obviously relying on such under-documented features for filters running on production systems is not ideal. For reference, they were added to the source code in 2018, (see smtpd.c revision 1.304, and parse.y revision 1.224).

If we wanted to run our test filter above as a dedicated user filter_user, we could use the following line in smtpd.conf:

filter test_filter proc-exec test_filter user "filter_user" group "filter_user"

Whilst the custom user and group are fairly easy to configure, getting a filter running in a chroot might cause a bit of frustration if you are not familiar with the requirements for setting up the chroot environment. If the chroot directory is not correctly populated with the required files, then smtpd will exit almost immediately with a fairly vague error message such as:

warn: lost processor: test_filter exited abnormally

Note that since fork_filter_process() uses the system() call to actually invoke the external filter program, a copy of /bin/sh will be required in $CHROOT/bin/sh in addition to any shared libraries that your filter itself uses.

To run the same filter in a chroot of /home/filter_user, we could specify:

filter test_filter proc-exec "/test_filter" user "filter_user" group "filter_user" chroot "/home/filter_user/"

Placing the filter shell script itself in /home/filter_user/test_filter, and copying /bin/cat to /home/filter_user/bin/ as well, since the script uses it. If the script is not modified to write it's output elsewhere, then the directory /home/filter_user/tmp will also need to be created.

Of course, new filters written specifically to run on OpenBSD systems might be better implemented using pledge() and unveil(), and avoiding the need to configure a chroot environment at all.

At this point, if we restart smtpd and try to send a regular email then the smtpd process will hang as noted above. This is because we've included a filter event in the list of events that our filter registers with, and when smtpd reaches this point it will be waiting for a response from the filter which will obviously never arrive.

However the output we wanted to capture will already have been written to /tmp/filter_dialogue, so we can now simply remove the lines we just added to smtpd.conf, and restart smtpd again to resume normal operation.

Input handling and buffering

Although we could use the stream I/O functions in the C standard library, the added level of abstraction that they provide doesn't really give us much benefit for the processing that we need to do in this situation so we might as well stick to the low-level file descriptor I/O calls.

Of course, when reading data from standard input in this way we need to do our own end of line parsing, since the read system call just returns arbitrary chunks of data and has no concept of 'lines'.

To deal with this, we read and store arbitrary amounts of data, (I.E. whatever comes back from the read() call), in to a fixed-size temporary buffer, (the raw buffer), noting the amount of valid data that we actually read.

We can read out exactly how much data we want from the raw buffer, looking for a newline character along the way, and store that data byte by byte in a separate line buffer. If no newline is found by the time we reach the end of the raw buffer, we simply re-fill it with another read() call, and continue transferring data to our line buffer.

Unless we encounter an error condition along the way, we will eventually find a newline character at which point our line buffer can be passed back to the calling function.

The following code implements this double buffering technique:

#define RAW_BUFFER_SIZE 65536

#define LINE_BUFFER_SIZE 4096

#define DEBUG_OUT(x) { write (STDERR_FILENO, x, sizeof(x)-1); }

int line_in(unsigned char * line_buffer, unsigned int * len, unsigned char * raw_buffer, unsigned int * raw_buffer_readpos, unsigned int * raw_buffer_writepos)

{

int bytesin;

unsigned int outpos;

outpos=0;

while (1) {

if (*raw_buffer_readpos==*raw_buffer_writepos) {

*raw_buffer_readpos=0;

bytesin=read(STDIN_FILENO, raw_buffer, RAW_BUFFER_SIZE);

* If we get end of file, or an error reading stdin, just exit.

if (bytesin==0) {

DEBUG_OUT ("Error - unexpected EOF on STDIN\n");

return(1);

}

if (bytesin==-1) {

DEBUG_OUT ("Error - got non-EOF error reading STDIN\n");

return(1);

}

*raw_buffer_writepos = bytesin;

}

* If we have already reached the end of the output buffer then exit, as this shouldn't happen during normal operation.

if (outpos==LINE_BUFFER_SIZE) {

DEBUG_OUT ("Error - buffer size exceeded\n");

return(1);

}

* Read a character from the global buffer, and put it in the output buffer. If it's 10, null terminate and return, else loop.

if ((*(line_buffer+outpos++) = *(raw_buffer+(*raw_buffer_readpos)++)) == 0x0a) {

*(line_buffer+outpos-1)=0;

*len=outpos-1;

return (0);

}

Note

The calling function is responsible for allocating memory for the two buffers as well as keeping track of the read and write pointers to the raw buffer.

In this particular application we only need to process a single input stream, but the line_in() function as written above would allow us to handle several independent streams concurrently simply by passing different sets of pointers.

Memory for the buffers is allocated using malloc_conceal() rather than regular malloc() to ensure that email content doesn't end up in core dumps, and that the buffer memory is zeroed out at program termination.

Note

We define our own DEBUG_OUT macro to write fixed-length strings to the standard error output rather than use dprintf.

None of the rest of the code requires us to include stdio.h and if we used dprintf for the debug code then it would add that dependency.

Note

The length of the line being returned is written to the supplied pointer len, but the line_in() function also null terminates line_buffer for convenience during debugging.

However the rest of the code will not rely on this null termination and will instead use the actual length returned. Although we wouldn't expect to encounter embedded 0x00 bytes in the data stream from smtpd, the filter shouldn't misbehave if it does so.

We can test the line_in() function above with the following code, which when run will just echo each new-line terminated line of input back to us rather like /bin/cat does when run interactively:

#include <stdlib.h>

#include <unistd.h>

int main()

{

unsigned char * line_buffer;

unsigned char * raw_buffer;

unsigned int readpos;

unsigned int writepos;

unsigned int len;

readpos=0;

writepos=0;

raw_buffer=malloc_conceal(RAW_BUFFER_SIZE);

line_buffer=malloc_conceal(LINE_BUFFER_SIZE);

while (line_in(line_buffer, &len, raw_buffer, &readpos, &writepos)==0) {

write (STDOUT_FILENO, line_buffer, len);

write (STDOUT_FILENO, "\n", 1);

}

Observe that we can read lines up to (LINE_BUFFER_SIZE - 1) bytes, (leaving one byte for zero termination), irrespective of the raw buffer size. We can even set the raw buffer size as low as one byte and the code still functions correctly. In practice, there isn't much point in setting RAW_BUFFER_SIZE greater than PIPE_SIZE as defined in /sys/sys/pipe.h, as this places an upper limit on the block size that will be used for such IPC.

Parsing pipelines

At this point, smtpd will start to send us data in the pipe delimited format described above. Whilst we were able to recognise the config|ready line as a direct literal string, the rest of the output requires a more thorough and methodical approach.

From here on in this document, we will use the convenient term 'pipeline' refer to a line of pipe delimited output from smtpd, (or part thereof). Don't confuse this with any other use of the term 'pipe' such as in the context of IPC. Here, we are simply referring to a line of output from smtpd such as:

Before we look at the code, let's look at the structure that we'll use to hold the deconstructed pipeline data:

struct st_pipeline {

unsigned char * value[8];

unsigned int len[8];

unsigned int totlen[8];

unsigned int lastfield;

};

Here we have an array of pointers which will point to the first character of each field, along with an array of field lengths, and a simple integer counter of how many fields were actually present and therefore how many array elements contain valid data. But what about the totlen array? This will contain the total length of the current field as well as all subsequent fields, in other words how many bytes are left until the end of the line.

The main use for the total length field in this particular filter is when we need to repeat back lines of email content without any changes. Since we also need to send the session ID and opaque token back as well, we can just start reading from the beginning of field five, (the session ID), and carry on across all of the other delimiters, automatically including the contents of fields six and seven without having to think about them as separate entities.

This is one convenient advantage of programming in a language like C where we can easily access the raw data directly as a block of memory, rather than an abstraction of it. Typically in most higher level languages you would either end up doing something like field_5 + field_6 + field_7, having previously dealt with the complexity of identifying which numbered field was the last one, (given that it may include pipe characters which are not delimiters), or alternatively doing something like field_5 + field_6 + field_7 ... field_n, where n is the last field, if you'd decided to avoid that complexity in the pipeline parsing code and just split the raw line in to fields at every pipe, with the intention of joining back the extra trailing fields that had been created afterwards.

None of that additional processing is necessary with the approach presented here.

When used in more complex filters, the total length field may also have other potential uses of a similar nature.

In the structure detailed above we have set a maximum of eight fields, and the version of our pipeline parsing code for this filter will be hard-coded to ignore any delimiters after field seven. Field seven just happens to be the last field of the data-line event, in which we want to interpret the pipe character as a literal anyway.

We can make this small optimisation here because we won't be processing any event lines which contain more than eight fields - the link-disconnect event ends at field five. Currently, as of OpenSMTPD 7.0.0, no events will create more than eleven fields anyway, although this could change in future versions of the API.

The following code parses a supplied raw pipeline as received by line_in() and fills in the supplied struct st_pipeline:

int parse_pipeline(unsigned char * buffer_line_in, unsigned int buffer_line_in_len, struct st_pipeline * pipeline) {

unsigned int i;

unsigned int pipecount;

pipecount=0;

memset(pipeline, 0, sizeof(struct st_pipeline));

if (buffer_line_in_len==0 || *buffer_line_in==0) {

return (1);

}

pipeline->value[0]=buffer_line_in;

for (i=1; i<buffer_line_in_len && pipecount < 7; i++) {

if (*(buffer_line_in+i)=='|') {

pipeline->len[pipecount]=i-(pipeline->value[pipecount]-buffer_line_in);

pipeline->totlen[pipecount]=buffer_line_in_len-(pipeline->value[pipecount]-buffer_line_in);

pipecount++;

pipeline->value[pipecount]=buffer_line_in+i+1;

}

pipeline->len[pipecount]=buffer_line_in_len-(pipeline->value[pipecount]-buffer_line_in);

pipeline->totlen[pipecount]=buffer_line_in_len-(pipeline->value[pipecount]-buffer_line_in);

pipeline->lastfield=pipecount;

return (0);

}

Coding detail

Size of a struct

Note that we take the size of struct st_pipeline for the call to memset, which will be correct and take in to account the array dimensions as they are part of the struct definition.

Common mistakes are to do something like sizeof(pipeline), which in this case will give the size of one pointer to a struct st_pipeline, (so typically 8 bytes on a 64-bit architecture).

Also, if we had an array of struct st_pipeline, then we would multiply the sizeof by the number of elements, but we don't have an array here as the only arrays are contained within the struct itself.

The parse_pipeline() function is fairly straightforward. First we zero out the entire struct st_pipeline with a call to memset, then we check for the supplied line being an empty string, which should not happen during normal operation. If we do encounter either zero-length input or input which begins with a null despite a non-zero length parameter being supplied, then we return immediately with 1 to indicate the error.
(Although the code is intended to support data containing embedded nulls, finding a null as the first byte would almost certainly indicate a logic error elsewhere such as being passed the wrong pointer value, so we treat this condition as an error.)
Since parse_pipeline() writes valid data to as many array elements as required and returns this value in pipeline->lastfield, zeroing out the entire structure is not strictly necessary if we have at least one field to process and if the calling function checks the value of pipeline->lastfield before accessing any array elements beyond zero.
However, since pipeline->lastfield is set to the index of the last valid field rather than the total number of fields, (in other words, it contains the number of valid fields minus one), zeroing out at least the first element of the arrays is important if the supplied input is empty, because simply returning the minimum value of zero for pipeline->lastfield implies that a single field with index 0 is valid. Empty input is considered as a single field with zero length, so in this case setting the length elements to zero is necessary.
By simply zeroing the whole structure each time, we also ensure that null input will return a null pointer. This should in turn cause the calling code to crash with a segfault in the case of a logic error there causing the pointer to be de-referenced, rather than have it erroneously process stale data from a previous invocation of parse_pipeline().
In any case, empty input lines are flagged as an error condition and processing should either completely ignore such input or terminate the filter.

Pedantic note:

Truly portable C code shouldn't assume that doing a memset of 0x00 bytes on a pointer will set it equal to NULL.

However in practice it is very unusual to encounter an architecture where this is not the case.

Additionally, the calling function is responsible for parsing only those fields which are indeed valid, and the intended way for it to ensure this is to check the value of pipeline->lastfield before accessing the arrays. However, by ensuring that non-existent fields have their length values set to zero the calling code can also just simply perform a check for the string it's expecting which will fail to match in the case of an empty, (zero-length), field.

(In fact, this is the method we will use in the main filter code.)

The pointer to field zero is set explicitly to the beginning of the supplied line, then we loop through the rest of the input byte-by-byte looking for pipe characters as we go. Each time we find one we calculate the length of the current field, increase the field count and set a pointer to the start of the new field.

The loop ends when we reach the end of the input, or when we get to field seven, whichever comes first. Outside the loop we fill in the length of the final field and the total field count.

At that point, we've finished parsing the line of raw input and have the data we need nicely stored in a struct st_pipeline.

Fun fact!

Trailing zero-length fields

If the supplied input pipeline ends with a pipe character then parse_pipeline() will correctly interpret this as a trailing field of zero length, and fill in a corresponding array entry. In this case, the pointer address in pipeline->value[n] will be one byte after the end of the input.

The calling function shouldn't try to access this address anyway if the field length is zero, but even if it does the fact that line_in() adds an, (otherwise un-needed), terminating null byte to the line buffer ensures that we won't read unwritten memory if we do in fact dereference the final pipeline->value[n] pointer.

Session management

With the pipeline processing code in place, the next step is to handle session management.

In more complicated filters where we need to store various pieces of metadata associated with each session, it would obviously be beneficial to define a C structure for this purpose.

However, all this current filter does is to re-write one header line, so we only need to store the session ID along with a single flag telling us whether we are still within the headers, (in which case we make the transformation when we see a line that matches our criteria), or have finished the headers and are now parsing the body text, (in which case we never match any lines, but just pass them back unaltered).

If we were going to define a structure for this, it would probably look something like the following:

struct st_session {

unsigned char sid[16];

unsigned int flag;

};

However, we can also just as easily store flat 17-byte records in memory directly and access them as raw bytes.

Doing so doesn't provide any technical advantage, in fact by using a struct and the features of the C language for handing this type of data, the compiler would almost certainly do some optimisation for us by appropriately aligning the members of the struct to improve average memory access times.

On the other hand, it is a convenient opportunity for such a programming exercise, given that the data we want to store is trivial. Seeing how it's done without recourse to a C struct is useful in case we ever want to implement something similar in assembler where we don't have high-level features such as structures.

The memory for the sessions table will be allocated early in the main() function and never free'd. We also set a counter of currently active sessions to zero:

#define MAX_SESSIONS 1024

#define SR_SIZE 17

int main()

{

unsigned char * session_table;

unsigned int sessions;

session_table=malloc(MAX_SESSIONS * SR_SIZE);

sessions=0;

return (0);

}

Data format:

In memory, each record is 17 bytes long, (SR_SIZE), and stored as follows:

16 bytes of ASCII characters for the session ID, without a terminating null.
1 byte of memory that this session can use.

The following code defines a new function session_get_add(), which accepts pointers to the two values we initialized above, as well as a pointer to the session ID to check or add, and a double pointer to a single byte of memory that the calling function can use to persistently store the required information about this session. In this case, the data is simply the one flag mentioned above.

int session_get_add(unsigned char * session_table, unsigned int * sessions, unsigned char * sid, unsigned char ** data_area)

{

unsigned int n;

for (n=0; n<*(sessions); n++) {

if (memcmp(sid, session_table+(n * SR_SIZE), 16)==0) {

*data_area=session_table+(n * SR_SIZE + 16);

return (0);

}

* No existing entry matching the session ID and token was found, so we add a new one.

if (*(sessions) == MAX_SESSIONS) {

return(2);

}

*data_area=session_table+(n * SR_SIZE + 16);

memcpy(session_table+(n * SR_SIZE), sid, 16);

(*sessions)++;

return (1);

}

The use of a double pointer is simply so that we can return the, (normal), pointer value to the calling function.

We don't zero out or clear the data byte in any way here, the calling function is responsible for setting it at the start of a new session. If we were ever to expand the data area to store more data, especially sensitive data, then this session_get_add() function should perform such initialisation.

Testing the session_get_add() function is easy enough:

#include <stdio.h>

int main()

{

unsigned char * session_table;

unsigned char * data_area;

unsigned int sessions;

session_table=malloc(MAX_SESSIONS * SR_SIZE);

sessions=0;

session_get_add(session_table, &sessions, "0123456789ABCDEF", &data_area);

printf ("%d %p\n", sessions, data_area);

session_get_add(session_table, &sessions, "0123456789ABCDEF", &data_area);

printf ("%d %p\n", sessions, data_area);

session_get_add(session_table, &sessions, "0101010101010101", &data_area);

printf ("%d %p\n", sessions, data_area);

session_get_add(session_table, &sessions, "0123456789ABCDEF", &data_area);

printf ("%d %p\n", sessions, data_area);

session_get_add(session_table, &sessions, "0123456789ABCDEF", &data_area);

printf ("%d %p\n", sessions, data_area);

session_get_add(session_table, &sessions, "0000000000000000", &data_area);

printf ("%d %p\n", sessions, data_area);

return (0);

}

The output from the test code above will be something like this:

1 0xb3368c9b010

2 0xb3368c9b021

2 0xb3368c9b010

3 0xb3368c9b032

So we can see that each unique session ID supplied to session_get_add() returns a different pointer, that the total number of active sessions increases when a new and previously unseen session ID is supplied, and that existing sessions match the pointer previously returned for them without increasing the total session count.

Implementation detail

Opaque token

Current versions of OpenSMTPD generate the opaque token once per session, (refer to lka_filter_ready() in lka_filter.c), so every event for a particular session returns the same token.

However, once again, there is nothing in the documentation that actually guarantees this behaviour. Whilst we could currently treat the session ID and opaque token as a single unit in our session management code, the correct way is to track only the session ID itself, and when we reply to a filter event to always return the token that was actually supplied that time.

To delete sessions we just need to search for the supplied session ID in the table and move any following entries back one slot:

int session_delete(unsigned char * session_table, unsigned int * sessions, unsigned char * sid)

{

unsigned int n;

for (n=0; n<*(sessions); n++) {

if (memcmp(sid, session_table+(n * SR_SIZE), 16)==0) {

if (n==((*sessions)-1)) {

(*sessions)--;

return (0);

}

memcpy(session_table+(n * SR_SIZE), session_table+((n + 1) * SR_SIZE), ((*sessions) - n - 1) * SR_SIZE);

(*sessions)--;

return (0);

}

return (1);

}

At this point we have working session management and can move on to the actual filtering code.

Main filtering code

The bulk of the main() function is a large while loop which only terminates on encountering an error condition from line_in().

We also have some more variable declarations and memory allocation for another buffer, which can go at the start of main().

unsigned char * ascii_time_buffer;

unsigned int flag_skip_line;

unsigned int ascii_time_buffer_len;

int offset;

struct tm time_tm;

time_t timestamp;

ascii_time_buffer=malloc(128);

while (line_in(line_buffer, &line_buffer_len, raw_buffer, &raw_buffer_readpos, &raw_buffer_writepos)==0) {

flag_skip_line=0;

parse_pipeline(line_buffer, line_buffer_len, &pipeline);

if (pipeline.len[4]==15 && memcmp(pipeline.value[4], "link-disconnect", 15)==0) {

session_delete(session_table, &sessions, pipeline.value[5]);

}

if (pipeline.len[4]==9 && memcmp(pipeline.value[4], "data-line", 9)==0) {

result=session_get_add(session_table, &sessions, pipeline.value[5], &data_area);

if (result==2) {

DEBUG_OUT ("Maximum number of concurrent sessions reached\n");

return (1);

}

* If this is a new session, then set the 'within headers' flag.

if (result==1) {

*data_area=1;

}

if (pipeline.totlen[7]==0) {

*data_area=0;

}

if (*data_area==1 && pipeline.totlen[7] > 5 && memcmp(pipeline.value[7], "Date:", 5) == 0 ) {

* Skip extraneous spaces and tabs.

* These will be collapsed to a single space in the output if we modify the date header.

for (i=5; i < pipeline.totlen[7] && (*(pipeline.value[7]+i)==' ' || *(pipeline.value[7]+i)==9); i++) { }

if (i < pipeline.totlen[7] && strptime(pipeline.value[7]+i, "%a,%e %b %Y %H:%M:%S %z", &time_tm)!=NULL) {

flag_skip_line=1;

offset=(time_tm.tm_gmtoff);

timestamp=timegm(&time_tm);

timestamp-=offset;

time_tm=*gmtime(&timestamp);

ascii_time_buffer_len=strftime(ascii_time_buffer, 128, "%a,%e %b %Y %H:%M:%S %z", &time_tm);

write (STDOUT_FILENO, "filter-dataline|", 16);

write (STDOUT_FILENO, pipeline.value[5], 34);

write (STDOUT_FILENO, "Date: ", 6);

write (STDOUT_FILENO, ascii_time_buffer, ascii_time_buffer_len);

write (STDOUT_FILENO, "\n", 1);

}

if (flag_skip_line==0) {

write (STDOUT_FILENO, "filter-dataline|", 16);

write (STDOUT_FILENO, pipeline.value[5], pipeline.totlen[5]);

write (STDOUT_FILENO, "\n", 1);

}

if (pipeline.len[7] == 1 && *pipeline.value[7]=='.') {

*data_area=1;

}

return (1);

We also need to increase the value of 4096 that we previously used for LINE_BUFFER_SIZE earlier during testing, as smtpd will accept smtp data lines up to 65536 bytes, (refer to SMTP_LINE_MAX in smtp_session.c). A value of 66560 allows sufficient space for a 'data-line' event with such a long final field.

#define LINE_BUFFER_SIZE 66560

Any event other than data-line or link-disconnect is ignored. Note that we check for 'data-line' and 'link-disconnect' in field four using memcmp with an explicit length value rather than strcmp, as the fields are not null-terminated.

Pedantic note:

A fully API-compliant filter should probably also check that field zero contains the corresponding literal, 'report', or, 'filter'.

Link-disconnect events just result in a call to session_delete() to remove the supplied session ID from the sessions table.
For each data-line event that is received, the code first calls session_get_add() to look up the session ID in the existing sessions table and add it if it isn't already there. In the case of a new session, the flag is then immediately set to indicate that we are processing the headers and should therefore make appropriate substitutions on any lines that look like date headers until we encounter the blank line which signals the end of the email headers and the beginning of the email body text.
This flag is reset when the code sees a data-line event with field seven set to zero length, in other words a blank line.
The code to check for a date header and actually do the timezone adjustment is only run at all if the 'within headers' flag is set.
If the current email content line begins with 'Date:', then we first skip any spaces and tabs following 'Date:', and pass the rest of the line to strptime(). If strptime() returns a valid broken down time in time_tm then we set a flag to suppress direct copying of the input to the output for this line of the email and proceed to further process the time_tm value that we have according to the algorithm described below. The result is then written to standard output along with the session ID and opaque token.
Note that we read 34 characters from field five in the following call to write(), even though field five itself should only contain the 16 characters of the session ID. This technique effectively reads field six at the same time and includes the trailing pipe character after each of those two fields, avoiding the need for us to manually insert it.
Lines in the email content which don't begin with 'Date:' are simply passed back to smtpd, and here we use the pipeline.totlen value to read everything from field five onwards in one go. So with a single call to write() we can copy out the session ID, opaque token, line of the email, and the necessary pipe delimiters.
Finally, if we see a line with the single terminating period then we reset the 'within headers' flag. This is necessary to ensure that when multiple emails are sent in a single session that the headers for each one are parsed. Without the reset of this flag, only the first email would have it's date header modified.
At this point, the code is functional and can be compiled and installed in /usr/local/libexec/smtpd for use as a filter.

Finishing touches

For completeness we could also add a call to pledge, allowing just the stdio promise:

pledge ("stdio", NULL);

Time conversion algorithm overview

The actual algorithm used to normalise the timezone might not be entirely obvious to readers unfamiliar to the various time-handling functions and representations of time used on OpenBSD, (and other BSD systems).

In this case, the following explanation should de-mystify it:

Adding or subtracting the number of hours, (and possibly minutes), that is specified in the timezone offset would be easy if it wasn't for the fact that passing midnight in either direction, (forwards or backwards), can make the day, month, and even year change as well.

As a result of this, we need to fully parse the textual date format in order to adjust it. In this case, we convert it in to a linear representation to which we can add arbitrary amounts of time, apply the timezone delta, then convert the linear representation back to text.

Since the intention here is to demonstrate the smtpd filters api rather than create a filter for production use, the example code to implement this algorithm has deliberately been kept simple at the expense of portability, (more details on this below).

Specifically, we use the following steps:

strptime() converts the textual format in to a struct tm representation, commonly known as broken down time.
we store the timezone offset from the tm_gmtoff field in offset.
timegm converts the struct tm representation to a time_t value, ignoring the tm_gmtoff field.
we then subtract the offset that we stored earlier
gmtime then converts the time_t value back to a struct time_tm, setting the tm_gmtoff field to zero.
strftime converts the struct time_tm representation back to text.

As a worked example, take the string: Thu, 1 Jan 1970 01:05:00 +0100.

The broken down time values produced by strptime() are shown in the table.

tm_sec	0	tm_min	5
tm_hour	1	tm_mday	1
tm_mon	0	tm_year	70
tm_wday	4	tm_yday	0
tm_isdst	0	tm_gmtoff	3600
tm_zone	(null)

We save the offset of 3600 seconds in, 'offset'.

The time_t value returned by timegm() is 3900, which is consistent with one hour, (60 × 60 = 3600), plus five minutes, (3600 + 300 = 3900), in other words the raw time value disregarding the timezone offset.

We perform the subtraction in step four: 3900 - 3600 = 300

tm_sec	0	tm_min	5
tm_hour	0	tm_mday	1
tm_mon	0	tm_year	70
tm_wday	4	tm_yday	0
tm_isdst	0	tm_gmtoff	0
tm_zone	GMT

In step five, gmtime() converts the value 300 in to the broken down time values shown in the second table.

Finally, strftime() produces the string: Thu, 1 Jan 1970 00:05:00 +0000, which is what we wanted.

Important!

Portability

The above timezone conversion process works on OpenBSD, but makes certain assumptions that might not be true on other systems.

In practice, the main issue is the use of strftime() to convert the modified broken down time back to a textual representation. When running on OpenBSD we can assume that this function will always use the C locale, producing output suitable for use in the email date header. Implementations on other systems might use different locales, and produce localised output which is not suitable. This issue is recognised within smtpd itself, which defines it's own function time_to_text() in to.c to avoid calling strftime(). We could easily do the same thing here if we wanted to use this filter in conjunction with the portable version of OpenSMTPD running on a system other than OpenBSD.

The tm_gmtoff field of struct tm is not mandated by POSIX, but it's rare to find a modern system which doesn't include it. Although the ctime manual page notes that this 'non-standard' field might change or be removed in the future, this advisory has been in the manual page for about 30 years so it seems fairly safe to assume that these fields won't change without widespread consensus.

Truly portable C code shouldn't even assume that subtracting an offset in seconds from a time_t value will produce the expected time value, although POSIX compliant systems do allow this assumption.

All of these portability issues could easily be fixed by writing a short custom function to do the manipulation of the date string directly instead of relying on the system library functions.

Data structures

The greylisting filter will use an almost identical version of struct st_pipeline to that used by the demonstration filter, it will just be expanded to hold a maximum of eleven fields instead of eight.

We'll also define two new C structures, one for session data, (for which the previous filter just wrote directly to a block of memory), and one for tuples.

struct st_pipeline

#define MAX_FIELDS 11

struct st_pipeline {

unsigned char * value[MAX_FIELDS];

unsigned int len[MAX_FIELDS];

unsigned int totlen[MAX_FIELDS];

unsigned int lastfield;

};

struct st_session

struct st_session {

unsigned char sid[16];

unsigned char ip[48];

unsigned char * helo;

unsigned char * env_from;

unsigned int ip_len;

unsigned int helo_len;

unsigned int env_from_len;

};

struct st_tuple

struct st_tuple {

unsigned char ip[47];

unsigned char flag_ok;

unsigned char * helo;

unsigned char * env_from;

unsigned char * env_to;

unsigned int ip_len;

unsigned int helo_len;

unsigned int env_from_len;

unsigned int env_to_len;

time_t timestamp;

};

The IP address fields in struct st_session and struct st_tuple can potentially require a maximum of 47 bytes if they need to store an IPv6 address with the full 32 hex digits and a five digit decimal port number:

[xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx]:12345

32 bytes for hex digits

7 bytes for colon separators between hex digits

2 bytes for square brackets

1 byte for the colon separator between address and port

5 bytes for the port number

Total: 47 bytes

Note that we deliberately place the flag_ok element immediately after the ip[] element within the structure. The compiler will typically align larger elements to byte boundaries that fall on some power of two, (typically eight bytes), and will also pad the end of the structure to a similarly rounded size. Since flag_ok is the only one-byte value, this means that any other ordering of the elements would be likely to create one byte of padding immediately following ip[] and increase the overall size of struct st_tuple by eight bytes.

Although it doesn't really make much sense to use this filter on connections received on a local socket, a size of 47 bytes is also sufficient to store the default socket address string, 'unix:/var/run/smtpd.sock'. Since the path to this socket can only be changed by modifying the source code to smtpd and re-compiling, the 47 byte size for ip[] is effectively sufficient to store any connecting address that might currently be encountered when running with an unmodified smtpd.

Three fields are common between st_session and st_tuple. As an smtp session progresses, the values for ip, helo, and env_from are filled in one-by-one in the corresponding session entry. Once all three of those have all been received, if a recipient address follows then a tuple is constructed from the values in the session table along with that recipient address. An entry in the struct st_tuples array will be either created if it doesn't already exist, or updated as necessary if it does already exist.

The timestamp in struct st_tuple will be set once when a particular tuple is first seen, and then not updated whilst it is greylisted. If the tuple is eventually whitelisted then this timestamp will be updated whenever it is seen again, until it expires.

The flag_ok value holds a value of 0 for greylisted, and 1 for whitelisted.

The greylisting filter will follow the same general program structure as the first demonstration filter, but we obviously need some additional functions to implement the new functionality and there will also be some changes to all but one of the existing functions.

Session management, (again!)

The session management code of the greylisting filter will differ in two main ways from the implementation that we saw earlier:

We'll be parsing the link-connect event rather than implying the start of a new session when see a new session ID.
The metadata for each session will be stored in a C structure rather than writing directly to a block of memory.

Other than that, the session handling is quite similar.

Here is the new version of session_get_add():

int session_get_add(struct st_session * session_table, unsigned int * sessions, unsigned char * sid, unsigned int * session_entry)

{

unsigned int n;

for (n=0; n<*(sessions); n++) {

if (memcmp(sid, session_table[n].sid, 16)==0) {

*session_entry=n;

return (0);

}

* No existing entry matching the session ID and token was found, so we add a new one.

if (*(sessions) == MAX_SESSIONS) {

return(2);

}

memcpy(session_table[*sessions].sid, sid, 16);

* Set pointers to NULL so that we know whether to free them later or not.

* Other values can be left wild.

session_table[*sessions].helo=NULL;

session_table[*sessions].env_from=NULL;

*session_entry=*sessions;

(*sessions)++;

return (1);

}

The first three arguments supplied have not changed, but the last argument is now a simple unsigned integer pointer which is used to return the array index of the session matching the supplied session ID to the calling function. Previously this argument was an indirect pointer to a single byte of memory.

As before, session_get_add() returns 0 on finding an existing matching entry in the session table, 1 if a new entry was added, or 2 if the sessions table is full.

Note that we should not reach MAX_SESSIONS in normal operation, if it is set higher than the maximum number of simultaneous sessions that smtpd itself would allow. Although we could add the facility to dynamically re-size the sessions table, since reaching MAX_SESSIONS here indicates either a code logic or configuration error we instead return with the value two, which the calling function should interpret as an error condition and then cause the filter to terminate.

When a new session ID is added to the table, the pointer elements are set to null. This is important because the memory that these pointers will point to will be dynamically allocated as the corresponding lines are received in the smtp transaction, and when the session ends the memory needs to be free'd. If the smtp session ended abruptly, for example after the helo but before the mail from, then the env_from pointer would not contain a value that can be free'd. To avoid this mis-management of memory, the de-allocation code will explicitly check for the pointers being set to null and not attempt to free them in that case.

Since session_get_add() now returns an array index instead of a pointer to memory, session_delete can now be changed to accept this index and remove the corresponding session from the sessions table directly rather than being supplied with a session ID and having to search the table for it.

int session_delete(struct st_session * session_table, unsigned int * sessions, unsigned int session_to_delete)

{

* Free pointers helo and env_from if they were previously malloc'ed.

if (session_table[session_to_delete].helo != NULL) {

free(session_table[session_to_delete].helo);

}

if (session_table[session_to_delete].env_from != NULL) {

free(session_table[session_to_delete].env_from);

}

if (session_to_delete==((*sessions)-1)) {

(*sessions)--;

return (0);

}

memmove(&session_table[session_to_delete], &session_table[session_to_delete+1], ((*sessions) - session_to_delete - 1) * sizeof(struct st_session));

(*sessions)--;

return (0);

}

As mentioned above, we only free pointers which were previously malloc'ed.If the session being deleted is not the last one in the array, we move all of the following entries back by one.

The actual greylisting code

The time intervals necessary for greylisting and whitelisting the tuples will be set at compile time using three defines. We'll also define a fixed maximum size of the tuple table, and a limit for how many tuple entires can be used by the same IP address:

#define GREY_MIN 600 /* Ten minutes */

#define GREY_MAX 21600 /* Six hours */

#define WHITE_MAX 864000 /* Ten days */

#define MAX_TUPLES 8192

#define MAX_TUPLES_PER_IP 16

The time values are defined in seconds, and will be used in the following ways:

GREY_MIN	The minimum amount of time required between a new tuple being seen, and a successive connection from the same tuple resulting in it being whitelisted. In other words, the minimum time a tuple needs to be successfully greylisted in order to be subsequently whitelisted.
GREY_MAX	The maximum amount of time that a tuple can be greylisted and awaiting possible whitelisting, before it is no longer considered to have been seen at all.
WHITE_MAX	The maximum amount of time that a tuple can remain whitelisted since it's last connection without being used, after which time it will be de-listed and treated as an unseen tuple on any future connection.

After an initial connection attempt by an as-yet-unseen tuple, a connection made after GREY_MIN seconds, and before GREY_MAX seconds will result in the tuple being whitelisted.

The core logic for the processing of the tuple data is in a new function tuple_add_update(). This has some similarities with session_get_add() in that we supply it with the tuple table and the current number of entries, (as well as some other parameters), and if it doesn't find an existing entry that matches then it tries to create a new one.

Unlike session_get_add(), though, the new tuple_add_update() function simply updates the tuple table and returns a value to the calling function indicating success or one of two types of failure. It doesn't return an index to the entry or any other data. Actually checking the whitelisted status of a particular IP address will be handled by a separate function that we will see shortly.

The return values from tuple_add_update() are:

0	Success. The supplied tuple data either matched an existing entry, or did not match one and was newly added to the table, (possibly replacing an existing and expired entry).
1	Failure. The supplied tuple data did not match an existing entry, but the tuple table already contains MAX_TUPLES entries and is therefore full. A new entry could not be added, but the filter can continue without it.
2	Failure. Memory could not be allocated for one of the dynamically allocated buffers. This is a critical error and the calling function should terminate the filter program.

Since tuple_add_update() is a fairly long function, we'll look at it in sections.

int tuple_add_update(struct st_tuple * tuple_table, unsigned int * tuples, struct st_session session, unsigned char * env_to, unsigned int env_to_len, int fd_l)

{

unsigned int i;

unsigned int per_ip_count;

time_t now;

now=time(NULL);

for (i=session.ip_len-1; i>0; i--) {

if (session.ip[i]==':') {

session.ip_len=i;

break;

}

Here we have the function definition, local variable declaration, and a few lines to remove the trailing port number from the supplied IP address.

The main thing to note about calling tuple_add_update() is that all of the data items for the proposed new tuple are supplied in a struct st_session, except for the envelope to value which is supplied as a separate parameter. This serves two purposes - a slight optimisation, and handling multiple recipients.

Each value ultimately comes from a different line in the filter API dialogue, (and apart from the connecting IP address, a different line in the smtp transaction). Since responses from one session might be interleaved with other sessions, and since a session might end abruptly without sending all of the expected parameters, we first build up the data for any potential new tuple in the session entry. Once it's complete, it can then be passed to tuple_add_update().

Smtpd requires the helo and mail from commands to appear in the smtp dialogue before rcpt to, so we can assume that the envelope to address will be the last data item needed to complete our tuple. We don't need to store the value in the sessions table first, but instead can call tuple_add_update() and pass the value directly from the pipeline_value[] array.

However, multiple rcpt to commands are valid for the same message. In this case we add a separate tuple for each recipient, re-using the previous values of IP address, helo, and envelope from.

Since greylisting is per IP address, repeated attempts, (within the appropriate timescale), to send to any one of the recipients that was specified in the first attempt will result in the sending host being whitelisted. Repeating the smtp transaction with the exact same list of recipients is not a requirement.

The code to strip the port number, (and it's separating colon), simply steps back from the end of the supplied IP and truncates it to the first colon that it finds. If no colon is found then it's left unchanged, although this shouldn't happen given valid input from smtpd.

Note that, as mentioned above in the section about data structures, although we only expect to encounter IP addresses in the source address field, (including both IPv4 and IPv6 addresses), it's also possible that we could be passed a local socket path if a listen on socket directive includes the filter.

Such a local socket path would look something like unix:/var/run/smtpd.sock. The port number truncating code would reduce this to the first four characters, but doing so is perfectly acceptable since only one local socket path is ever used in any particular invocation of smtpd, so we don't need to differentiate between different paths anyway. It's not really worth special-casing the port number truncating code to detect this kind of input and behave differently.

per_ip_count=0;

for (i=0; i<*(tuples); i++) {

if (tuple_table[i].ip_len == session.ip_len && memcmp(tuple_table[i].ip, session.ip, tuple_table[i].ip_len) == 0) {

if (tuple_table[i].flag_ok == 1 && now-tuple_table[i].timestamp <= WHITE_MAX) {

tuple_table[i].timestamp=now;

dprintf (fd_l, "Updated timestamp on already whitelisted tuple %d\n", i);

return (0);

}

if (tuple_table[i].flag_ok == 0 && now-tuple_table[i].timestamp < GREY_MAX) {

per_ip_count++;

}

if (per_ip_count > (MAX_TUPLES_PER_IP / 2)) {

dprintf (fd_l, "IP already has %d greylisted tuples\n", per_ip_count);

}

for (i=0; i<*(tuples); i++) {

if (tuple_table[i].ip_len == session.ip_len && tuple_table[i].helo_len == session.helo_len && tuple_table[i].env_from_len == session.env_from_len) {

if (tuple_table[i].env_to_len == env_to_len) {

if (memcmp(tuple_table[i].ip, session.ip, tuple_table[i].ip_len) == 0 &&

memcmp(tuple_table[i].helo, session.helo, tuple_table[i].helo_len) == 0 &&

memcmp(tuple_table[i].env_from, session.env_from, tuple_table[i].env_from_len) == 0 &&

memcmp(tuple_table[i].env_to, env_to, tuple_table[i].env_to_len) == 0 ) {

* Matched existing tuple.

* If already whitelisted, then update the timestamp.

* If still greylisted, don't update the timestamp, but whitelist if it's more than GREY_MIN and less then GREY_MAX behind.

if (tuple_table[i].flag_ok == 1) {

if (now-tuple_table[i].timestamp > WHITE_MAX) {

dprintf (fd_l, "Whitelisted tuple %d has exceeded WHITE_MAX, returning to greylisted status "

"with new current timestamp %lld\n", i, now);

tuple_table[i].flag_ok=0;

tuple_table[i].timestamp=now;

return (0);

}

* The following case should already have been handled by the first iteration over the tuples.

* However it's been left here in case future changes inadvertently change the program logic and

* allow this path to be followed.

tuple_table[i].timestamp=now;

dprintf (fd_l, "Updated timestamp on already whitelisted tuple %d, (unexpected code path!)\n", i);

return (0);

}

* Currently greylisted. Update to whitelisted if > GREY_MIN and < GREY_MAX.

* If and only if it's now whitelisted, then also update the timestamp.

if (now-tuple_table[i].timestamp > GREY_MIN && now-tuple_table[i].timestamp < GREY_MAX) {

tuple_table[i].flag_ok=1;

tuple_table[i].timestamp=now;

dprintf (fd_l, "Whitelisted previously greylisted tuple %d\n", i);

return (0);

}

* If greylisted and >= GREY_MAX then update the timestamp as we effectively 'start again'.

if (now-tuple_table[i].timestamp >= GREY_MAX) {

dprintf (fd_l, "Matching tuple %d had passed GREY_MAX. ", i);

if (per_ip_count >= MAX_TUPLES_PER_IP) {

dprintf (fd_l, "IP already has the configured maximum of %d greylisted tuples, ignoring\n", MAX_TUPLES_PER_IP);

return (0);

}

dprintf (fd_l, "Resetting timestamp to current time and keeping as greylisted\n");

tuple_table[i].timestamp=now;

return (0);

}

* Greylisted and hasn't yet reached GREY_MIN. Do nothing.

dprintf (fd_l, "Tuple %d is greylisted for %llu more seconds\n", i, GREY_MIN-(now-tuple_table[i].timestamp));

return (0);

}

Above is the code that implements the core of the actual greylisting logic. This is where we compare timestamps and ultimately flag entries as whitelisted.
The first loop iterates over all of the tuples currently in the tuple table, looking for a valid, (non-expired), already whitelisted entry which matches the IP address - regardless of the other parameters. If a match is found then it's timestamp is updated and the function returns without doing further processing.
This avoids the creation of a large number of excess tuple entries for any IP address that has already been identified as a genuine sender.
At the same time, we keep a count in per_ip_count of the current number of non-expired greylisted tuple entries for this IP address.
The second loop again iterates over all of the current tuples, this time comparing the supplied IP, helo, envelope from, and envelope to values looking for a match. If a match is found, we then progress to checking various combinations of already being whitelisted or not, and how long ago it was last seen.
First we check if it's already flagged as whitelisted. If so, we check if the whitelisting has passed WHITE_MAX and therefore expired, in which case we revert the entry to being greylisted. In either case the timestamp is updated to the current time.
Note that if the whitelisting hadn't expired, we shouldn't reach this part of the code because the first loop should have matched the entry in the tuple table and returned from tuple_add_update() early. However the program logic could easily be changed inadvertently and cause this code path to be followed, so we ensure that an appropriate message is written to the logfile in this case.
If the entry is not already flagged as whitelisted, then we check to see if it's between the minimum and maximum greylisting timeouts. If it is, then we whitelist it and update the timestamp.
At this point, the greylisted entry must have either expired or not yet reached the threshold for being whitelisted. In the case of having expired, we leave it as greylisted and update the timestamp effectively treating it like a previously unseen tuple, as long as this wouldn't cause the total number of active greylisted tuple entries for this IP to exceed MAX_TUPLES_PER_IP. Otherwise we do nothing apart from write an entry to the logfile.
As long the set of values supplied to tuple_add_update() matches an existing entry then one of the above actions will have run, and in each case tuple_add_update() returns zero to the calling function.

Next we have to deal with the case of adding a previously unseen tuple as a new entry.

First, we check that the current IP address hasn't already reached it's allowance of tuple table entries:

if (per_ip_count >= MAX_TUPLES_PER_IP) {

dprintf (fd_l, "IP has already reached the configured maximum of %d greylisted tuples, silently ignoring\n", MAX_TUPLES_PER_IP);

return (0);

}

Assuming that it hasn't then we continue to add the new entry.

Unless the tuple table is full, adding a new entry is trivial as we just write it to the next free slot as determined by the total number of existing tuples:

if (*(tuples) < MAX_TUPLES) {

* If there is space in the table then we just append a new entry.

tuple_table[*tuples].ip_len=session.ip_len;

memcpy(tuple_table[*tuples].ip, session.ip, session.ip_len);

tuple_table[*tuples].helo_len=session.helo_len;

tuple_table[*tuples].helo=malloc_conceal(session.helo_len);

if (tuple_table[*tuples].helo == NULL) {

return (2);

}

memcpy(tuple_table[*tuples].helo, session.helo, session.helo_len);

tuple_table[*tuples].env_from_len=session.env_from_len;

tuple_table[*tuples].env_from=malloc_conceal(session.env_from_len);

if (tuple_table[*tuples].env_from == NULL) {

return (2);

}

memcpy(tuple_table[*tuples].env_from, session.env_from, session.env_from_len);

tuple_table[*tuples].env_to_len=env_to_len;

tuple_table[*tuples].env_to=malloc_conceal(env_to_len);

if (tuple_table[*tuples].env_to == NULL) {

return (2);

}

memcpy(tuple_table[*tuples].env_to, env_to, env_to_len);

tuple_table[*tuples].timestamp=now;

tuple_table[*tuples].flag_ok=0;

dprintf (fd_l, "No matching tuple found in database, adding new tuple at slot %d\n", *tuples);

(*tuples)++;

return (0);

}

Note that if memory allocation for any of the dynamically allocated elements fails, we just immediately return a value of two to the calling function and don't explicitly free any previous allocations that we've just done. This is fine and won't cause a memory leak, because being an unrecoverable error the calling function is expected to terminate the program anyway upon receiving this return value.

Once the tuple table is full, to add a new entry we need to replace an existing one.

Greylisted entries that have passed GREY_MAX, and whitelisted entries that have passed WHITE_MAX are candidates for re-use. The code below simply looks through the tuple table for the first entry that matches either criteria.

for (i=0; i < *(tuples); i++) {

if ( (tuple_table[i].flag_ok == 0 && now-tuple_table[i].timestamp >= GREY_MAX) ||

(tuple_table[i].flag_ok == 1 && now-tuple_table[i].timestamp > WHITE_MAX) ) {

dprintf (fd_l, "Reached MAX_TUPLES when adding new entry - overwriting existing %slisted tuple at slot %d\n",

(tuple_table[i].flag_ok == 0 ? "grey" : "white"), i);

tuple_table[i].ip_len=session.ip_len;

memcpy(tuple_table[i].ip, session.ip, session.ip_len);

tuple_table[i].helo_len=session.helo_len;

free (tuple_table[i].helo);

tuple_table[i].helo=malloc_conceal(session.helo_len);

if (tuple_table[i].helo == NULL) {

return (2);

}

memcpy(tuple_table[i].helo, session.helo, session.helo_len);

tuple_table[i].env_from_len=session.env_from_len;

free (tuple_table[i].env_from);

tuple_table[i].env_from=malloc_conceal(session.env_from_len);

if (tuple_table[i].env_from == NULL) {

return (2);

}

memcpy(tuple_table[i].env_from, session.env_from, session.env_from_len);

tuple_table[i].env_to_len=env_to_len;

free (tuple_table[i].env_to);

tuple_table[i].env_to=malloc_conceal(env_to_len);

if (tuple_table[i].env_to == NULL) {

return (2);

}

memcpy(tuple_table[i].env_to, env_to, env_to_len);

tuple_table[i].timestamp=now;

tuple_table[i].flag_ok=0;

return (0);

}

dprintf (fd_l, "Error - MAX_TUPLES is too small\n");

return (1);

}

If no such expired entry is found then tuple_add_update() simply doesn't add an entry for a tuple with the data it was called with.

The filter, (and smtpd), can continue running, with the only effect being that a host which would have been greylisted and started the process of progressing towards being whitelisted, will instead just immediately be forgotten.

Whilst this might not be ideal, if we allowed the number of tuple entries to grow without limit then the filter would eventually face resource exhaustion in some other way.

Although in theory this new-entry-discarding behaviour could be used by a malicious remote host to fill the tuple table, in practice several factors limit the effectiveness of such an attack:

Hosts that are already whitelisted can continue to send mail anyway, even when the tuple table is full.
Greylisted entries expire, so a bad actor would have to repeatedly re-create them.
The remote host needs a sufficient number of IP addresses to overcome the per IP limit for tuple entries, (easy with IPv6 but likely more difficult with IPv4).
Memory usage is quite reasonable on a modern system even with 50,000 entries or more in the tuple table.

If a remote attacker has the resources to overcome the above issues, they can probably cause a denial of service quite easily without resorting to filling the tuple table.

Testing tuple_add_update()

Now that we can display the contents of the tuple table, it becomes quite easy to test the new tuple_add_update() function independently of the rest of the filter code.

The following demo_tuples() routine exercises tuple_add_update(), and allows us to observe the intended greylisting and whitelisting behaviour.

int demo_tuples(struct st_tuple * tuple_table, unsigned int * tuples)

{

unsigned int w;

struct st_session cur_session;

unsigned char * env_to;

unsigned int env_to_len;

memcpy(cur_session.ip, "10.0.0.1", 8);

cur_session.ip_len=8;

cur_session.helo="one.example";

cur_session.helo_len=11;

cur_session.env_from="operator@one.example";

cur_session.env_from_len=20;

env_to="root@example";

env_to_len=12;

tuple_add_update(tuple_table, tuples, cur_session, env_to, env_to_len, STDOUT_FILENO);

memcpy(cur_session.ip, "10.0.0.2", 8);

cur_session.helo="two.example";

cur_session.env_from="operator@two.example";

w=1;

while (1) {

tuple_add_update(tuple_table, tuples, cur_session, env_to, env_to_len, STDOUT_FILENO);

display_tuple_table(tuple_table, *tuples, STDOUT_FILENO);

dprintf (STDOUT_FILENO, "Sleeping for %d second%s\n\n", w, w > 1 ? "s" : "");

sleep(w);

w++;

}

return(0);

}

int main()

{

unsigned int tuples=0;

struct st_tuple * tuple_table;

tuple_table=malloc(MAX_TUPLES * sizeof(struct st_tuple));

demo_tuples(tuple_table, &tuples);

return (0);

}

If you want to compile the above test program, you'll need at least the following includes and definitions that we've already seen in addition to the code for display_tuple_table() and tuple_add_update():

#include <stdio.h>

#include <stdlib.h>

#include <unistd.h>

#include <time.h>

#include <string.h>

#define GREY_MIN 10

#define GREY_MAX 15

#define WHITE_MAX 12

#define MAX_TUPLES 8192

#define MAX_TUPLES_PER_IP 16

struct st_session {

unsigned char sid[16];

unsigned char ip[48];

unsigned char * helo;

unsigned char * env_from;

unsigned int ip_len;

unsigned int helo_len;

unsigned int env_from_len;

};

struct st_tuple {

unsigned char ip[47];

unsigned char flag_ok;

unsigned char * helo;

unsigned char * env_from;

unsigned char * env_to;

unsigned int ip_len;

unsigned int helo_len;

unsigned int env_from_len;

unsigned int env_to_len;

time_t timestamp;

};

Note that in this example, GREY_MIN, GREY_MAX, and WHITE_MAX have all deliberately been set to much smaller values than would be expected on a production system.

This is purely so that we can see the relevant output from the test program within a reasonable amount of time.

The test code starts by creating two tuples, and then updates just the second one at increasing time intervals.

Here is the output from running it:

No matching tuple found in database, adding new tuple at slot 0

No matching tuple found in database, adding new tuple at slot 1

Tuple: 0, IP: 10.0.0.1, helo: one.example, from: operator@one.example, to: root@example, timestamp: 1694193887, flags: GREY UNEXPIRED

Tuple: 1, IP: 10.0.0.2, helo: two.example, from: operator@two.example, to: root@example, timestamp: 1694193887, flags: GREY UNEXPIRED

Sleeping for 1 seconds

Tuple 1 is greylisted for 9 more seconds

Tuple: 0, IP: 10.0.0.1, helo: one.example, from: operator@one.example, to: root@example, timestamp: 1694193887, flags: GREY UNEXPIRED

Tuple: 1, IP: 10.0.0.2, helo: two.example, from: operator@two.example, to: root@example, timestamp: 1694193887, flags: GREY UNEXPIRED

Sleeping for 2 seconds

Tuple 1 is greylisted for 7 more seconds

Tuple: 0, IP: 10.0.0.1, helo: one.example, from: operator@one.example, to: root@example, timestamp: 1694193887, flags: GREY UNEXPIRED

Tuple: 1, IP: 10.0.0.2, helo: two.example, from: operator@two.example, to: root@example, timestamp: 1694193887, flags: GREY UNEXPIRED

Sleeping for 3 seconds

Tuple 1 is greylisted for 4 more seconds

Tuple: 0, IP: 10.0.0.1, helo: one.example, from: operator@one.example, to: root@example, timestamp: 1694193887, flags: GREY UNEXPIRED

Tuple: 1, IP: 10.0.0.2, helo: two.example, from: operator@two.example, to: root@example, timestamp: 1694193887, flags: GREY UNEXPIRED

Sleeping for 4 seconds

Tuple 1 is greylisted for 0 more seconds

Tuple: 0, IP: 10.0.0.1, helo: one.example, from: operator@one.example, to: root@example, timestamp: 1694193887, flags: GREY UNEXPIRED

Tuple: 1, IP: 10.0.0.2, helo: two.example, from: operator@two.example, to: root@example, timestamp: 1694193887, flags: GREY UNEXPIRED

Sleeping for 5 seconds

Matching tuple 1 had passed GREY_MAX. Resetting timestamp to current time and keeping as greylisted