I was quite excited to read it, as I’ve been enjoying reverse engineering quite a bit lately, so also wanted to try doing some software development directly in assembly to sharpen my skills a bit. Unfortunately, I’d have to describe my experience of reading it as disappointing. Within 10 minutes of reading I had found several errors, and sadly the trend continued through the rest of the book.

An example of one of the errors I found was in a table on page 16, listing common instructions: cupid - Query CPU identification and feature information, which should read cpuid.

I also didn’t gel with the teaching style of this book. I was hoping for and expecting a thorough guide of the fundamentals of x86 64-bit assembly language. Perhaps starting with a high level look at how functions from C or C++ are represented in assembly, what a stack frame is, what the most common calling conventions are, before drilling down into topics like floating point operations, common string manipulations, best practices, and so on.

Instead, this book shares about a page of C++ code, followed by a hand-written equivalent in assembly (with comments) and then some prose describing what the program does. Any new instructions introduced are given barely a sentence of explanation. There’s no consistent synopsis for new instructions being introduced, just what feels like hand waving.

Perhaps for some that’s a useful format, but to me it seemed to add very little value to just reading the output of gcc -s by itself, and using google to look up any instructions I was unfamiliar with.

The comments in the assembly code also doesn’t seem to have been proof read very thoroughly, containing numerous inconsistencies.

Just by flicking through the book now and inspecting random pages I can find examples easily.

On page 56: inc edx; edx = i + i. Actually inc increments a register, it does not add i to itself.

On page 76:

mov rax,r8          ; rax = 'src'
mov rdi,[rax+r11*8] ; rdi = src[i]

Why is src in quotes? rax isn’t pointing to a string containing src. Elsewhere src isn’t quoted.

Other errors I remember, but can’t be bothered to back and find were the comments using + and += interchangeably. That is, sometimes you’d see count += 1 to mean increment count, and sometimes you’d see count + 1 for the same operation. Sometimes the comments would even refer to different variable names than the code was even using.

Perhaps this sounds fussy, but I didn’t go into this book with a nitpicking attitude at all. In my casual read I just started spotting errors, and couldn’t stop spotting them. I can’t emphasise enough that this is a second edition, with experienced and respectable engineers listed as the author and technical reviewer of this book. I don’t really understand how so many errors made it in, survived review, and survived to the second edition.

Moving on from the topic of errors, I’d like to comment a little more on the way that key concepts are covered. I’m by no means an expert on assembly programming but I’d still consider calling conventions to be an absolutely critical area to cover, lest a beginner be confused by contradictory examples found across different books and the internet.

This book casually introduces them as an aside on page 31, buried under the header of Advanced Integer Arithmetic. It’s mentioned because the author is giving an example that accepts a non-trivial number of parameters and has realised the way they’re passed to the function might be non-obvious to a beginner. So on page 33 the author gives a brief summary of which parameters are passed in which registers in Visual C++ and that for other compilers and operating systems the reader should go and read their respective manuals. What?. At the very minimum the author ought to provide the most common calling convention around: the SysV x86-64 ABI, but they don’t. In 600 pages not one paragraph is spared to summarise any x86 calling conventions other than Visual C++’s.

Consider this, if someone is writing assembly language by hand what context are they going to be using it in? Most likely to write server software that is almost certain to be running on a unix-like operating system. Have you heard of any high frequency trading shops using Windows Server? I haven’t. Or perhaps they’re working on a AAA video game that’ll indeed be running on Windows 10 x64, but also very likely on a console like the PlayStation that runs on a FreeBSD derivative.

But no, this book leaves the most common environments high and dry on such a critical aspect of assembly programming. “That’s not fair,” you might say, “the author is assuming a more experienced audience.” No, they’re not. On page 21 the author felt it necessary to explain to the reader what a Visual Studio solution and project are, and how to open one by double clicking on it. I cannot reconcile the decision to explain how to use Visual Studio in an assembly book but not to cover a non Windows ABI.

Another concept that I thought was introduced and explained far too late in the text is stack frames. Stack frames are taught on page 143. Page one hundred and forty-three, under the subheading of Calling Convention, which yet again only describes Visual C++. Another thing you won’t find an explanation of in this book: __stdcall vs __cdecl vs __fastcall. Despite being extremely common calling conventions on Windows that you will encounter and have to deal with, they’re not covered. It’s all implicitly __fastcall here. I appreciate that this book is focused on 64-bit programming but giving the reader some awareness of that potential difference could save them hours of confusion in the future.

I wish I could have written something nicer about this book, but I really did feel let down by it. It doesn’t cover core concepts and best practices in a clear, methodical way. Instead it dumps sample programs on the reader with dubious code comments and hand-waving prose. I just can’t see myself recommending it to anyone. It’s not clear or thorough enough to recommend to beginners. Meanwhile I suspect experts would be better off improving their knowledge by just reading the manuals published by Intel and AMD.

If someone knows of a better book covering this topic I’d love to hear recommendations.

With so few elements, any processor with a cache and prefetcher worth its salt will yield excellent performance using just a plain dynamic array, which are trivial to implement as needed. So that’s what I’ve used.

However, on day 23 of 2021’s advent of code I finally had need of a hashmap as an optimisation. For that programming problem I found myself dealing with hundreds of thousands of elements and doing (probably) millions of lookups. Switching from a dynamic array to a hashmap improved the runtime of my solution from 10 seconds to 0.3 seconds. For that I used klib’s excellent hashmap implementation.

While I know how it works conceptually, there’s a huge difference between reading about how something works and implementing it yourself, so I wanted to finally learn how to implement my own hashmap from scratch. In this blog post, I’ll be sharing what I came up with.

Design considerations

As Wikipedia will tell you, there’s a lot of ways to build a hashmap, each with its own sets of pros and cons. I’ll be focusing on the simplest possible approach, just to get something working. I can always make more sophisticated optimisations later as needed.

Hashing

A hash map needs a hash function, but I’ve never actually written a hash function before either. I don’t know what a good hash function for this use case would actually be, since I’ve only ever heard of checksums and cryptographically secure hashes.

After asking some friends on irc, I settled on djb2, a very simple, but apparently effective, hash function by the famous DJB.

The function is very simple:

1
2
3
4
5
6
7

unsigned int djb2(const char *bytes, size_t len)
{
  unsigned int hash = 5381;
  for (size_t i = 0; i < len; ++i)
    hash = hash * 33 + bytes[i];
  return hash;
}

That’s actually simple enough that I could pull it from memory if the need should ever arise in future. Nice!

Collision resolution

So what happens if two keys hashes point to the same bucket? The two most common solutions seem to be separate chaining and open addressing. The former means each bucket holding a linked list containing all the key-value pairs that live in that bucket. The latter says to just roll over onto the next adjacent bucket until you find one that has space.

In the interests of simplicity I’m going to use open addressing. That way I won’t also have to much around with linked lists, I’ll just have a big array of buckets to oversee.

Open addressing does have an issue though. Let’s look at what happens if a conflict occurs but then the key that ‘won’ the conflict is deleted:

1. The hashmap is initially empty
2. fizz is looked up, which maps to bucket 1
   • Bucket 1 is empty, so the lookup fails
3. foo=bar is inserted, mapping to bucket 1
4. fizz=buzz is inserted, also mapping to bucket 1
   • Bucket 1 is occupied, so it goes into bucket 2 instead
5. fizz is looked up
   • Bucket 1 is occupied but has the wrong key
   • Bucket 2 is occupied and has the right key, so the lookup succeeds
6. foo is deleted, leaving bucket 1 empty
7. fizz is looked up, which maps to bucket 1
   • Bucket 1 is empty, so the lookup fails

The key for fizz is now in the wrong bucket with no indication that there was ever a conflict. We could check the next bucket just in case, but what if there were two conflicts and fizz is actually in bucket 3? We don’t want to fall back to a full search of the entire hashmap for a missing key. So what do we do?

A peek at klib’s code gives the solution. When deleting a value, instead of marking its bucket as empty, mark it as ‘deleted’ instead. When writing, treat a deleted bucket the same as an empty one. When reading, skip over deleted buckets but don’t stop searching until an empty non-deleted bucket is found.

That way, even though fizz is in the wrong bucket, the search will still move past the ‘deleted’ bucket onto the one fizz is actually stored in.

Load factor

How full should we allow the hashmap to get before increasing its size? If we let it get too full then we’ll start to have many collisions and long chains of contiguous entries that have to be searched linearly, losing that O(1) performance we’re looking for.

If we oversize it, we waste memory.

Wikipedia suggests a factor of 0.6-0.75 is good. I’m going to keep the maths simple and double the size of the hashmap any time we get beyond a load factor of 0.75, and halve the size of the hashmap any time that it drops below 0.25.

Type specialisation

I quite like klib’s approach of using macros to generate type specific versions of its hashmap as needed, similar to how C++’s templating works. I’m not going to incorporate that in my first attempt though. Instead I’ll use hardcoded key and value types for now, but typedef them so that they’re easy to change out later.

The types

With those decisions made, we’re ready to begin:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

typedef char* hm_key;
typedef char* hm_value;

enum hm_state {
  HM_EMPTY = 0,
  HM_VALID = 1,
  HM_DELETED = 2,
};

struct hashmap {
  size_t len; // number of buckets in use, for tracking load
  size_t cap; // number of buckets allocated
  enum hm_state *states; // array of bucket states
  hm_key *keys; // array of bucket keys
  hm_value *values; // array of bucket values
};

So first we typedef some custom types for easy replacement later. Then we create an enum for tracking the state of each bucket in our hashmap. Lastly we define the hashmap struct itself.

The functions

Initialisation

1
2
3
4

static void hashmap_init(struct hashmap *hm)
{
  memset(hm, 0, sizeof *hm);
}

I’m a fan of data structures that are valid when zeroed out. That allows me to instantiate them on the stack effortlessly like so:

1

struct hashmap hmap = {0};

But users of it shouldn’t need to know that, so if they malloc the hashmap it’s good to have the hashmap_init function for them to use without needing to know about the design choices I’ve made.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

static void hashmap_free(struct hashmap *hm)
{
  if (!hm)
    return;
  if (hm->cap) {
    free(hm->keys);
    free(hm->values);
    free(hm->states);
  }
  memset(hm, 0, sizeof *hm);
}

Next is the hashmap_free which again is nothing exciting, just some good housekeeping. Zeroing itself out isn’t strictly necessary but will help generate nice clean crashes if misused. A null pointer is way more obvious than a stale one in a core dump.

Hashing

1
2
3
4
5
6
7

static size_t hashmap_hash_key(hm_key key)
{
  size_t v = 5381;
  for (size_t i = 0; key[i]; ++i)
    v = v * 33 + key[i];
  return v;
}

This is exactly as I described before, except that it’s now returning a size_t. This is just done as a convenience since I know I’ll be using it as an index into my buckets, albeit with the modulo operator. It saves me some casting but I also know that it’ll be 64-bits wide most everywhere, unlike unsigned int.

Getter & Setters

1
2
3
4
5
6

#define hashmap_begin(hm) ((size_t)(0))
#define hashmap_end(hm) (((hm)->cap))
#define hashmap_states(hm, it) ((hm)->states[(it)])
#define hashmap_key(hm, it) ((hm)->keys[(it)])
#define hashmap_value(hm, it) ((hm)->values[(it)])
#define hashmap_exists(hm, it) ((it) < (hm)->cap && hashmap_states((hm), (it)) == HM_VALID)

Again inspired by klib, I decided to make these accessor functions preprocessor macros, to give the compiler the best opportunity to optimise them. These could just be static inline functions in the header arguably, but for the key and value macros there is one nice advantage:

1
2
3

struct hashmap hm; // our hashmap
size_t it; // assume it is a valid index or 'iterator' into the hashmap
hashmap_value(&hm, it) = "value"; // assign a value

That kind of assignment can be done without having to dance around pointers.

Insertion

With the boilerplate done, we can get onto the meat of the hashmap.

Insertion is a nice simple operation conceptually:

1. Hash the key, modulo by the number of buckets to get the bucket index
2. If the bucket is occupied with a different key, try the next bucket.
3. Write the key and value into the bucket, mark it as valid, and update len.

In code, this only takes a few minutes to whip up:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

static size_t hashmap_insert(struct hashmap *hm, hm_key key, bool *existed)
{
  // First see if we need to resize the hashmap
  // If that fails, abort and return an invalid iterator
  if (!hashmap_resize(hm))
    return hm->cap;

  // Hash the key, modulo by the number of buckets
  size_t it = hashmap_hash_key(key) % hm->cap;

  // Skip over full buckets until we find an available one,
  // either empty or deleted is fine. We know this can't get
  // into an infinite loop due to lack of space since we limi
  // the load factor to 0.75.
  while (hm->states[it] == HM_VALID && strcmp(key, hm->keys[it]))
    it = (it + 1) % hm->cap;

  // If we're not overwriting an existing value with the same key then
  // to increment the count of how many buckets are in use
  if (hm->states[it] != HM_VALID)
    hm->len += 1;
  // If we've been given a valid pointer, use it to report whether the
  // key already existed in the hashmap or not.
  if (existed)
    *existed = hm->states[it] == HM_VALID;
  // Lastly, mark the bucket as in use and set its key and value.
  hm->states[it] = HM_VALID;
  hm->keys[it] = key;
  hm->values[it] = value;
  // And return an iterator to the bucket
  return it;
}

One operation I didn’t mention before is the hashmap_resize function that I call. That is going to be the most difficult part of the hashmap so I’ll save it for last. We can just assume that if it returns true then the hashmap has buckets allocated and a reasonable load factor.

Removal

Compared to insertion, removal is trivial. We’re going to assume that the caller already has an iterator to the bucket they wish to remove, so we just need to check it’s a valid iterator, mark the bucket deleted if so, and then shrink the hashmap if necessary.

1
2
3
4
5
6
7
8

static void hashmap_remove(struct hashmap *hm, size_t it)
{
  if (hashmap_exists(hm, it)) {
    hm->states[it] = HM_DELETED;
    hm->len -= 1;
  }
  hashmap_resize(hm);
}

Lookup

Now for the most useful function of all: looking up a value in the hashmap by key. Conceptually it’s quite similar to what we do for insertion. We hash the key and modulo it to get the bucket index the value should live in. If it’s the right bucket, return its index. If not, but the bucket is populated or deleted, try the next bucket in case there was a collision.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

static size_t hashmap_find(const struct hashmap *hm, hm_key key)
{
  // Avoid dereferencing null pointers if we've not allocated any buffers yet
  if (hm->cap == 0)
    return hm->cap;

  // Calculate the bucket the key corresponds to
  size_t it = hashmap_hash_key(key) % hm->cap;

  // Search for a bucket with a matching key.
  // Keep going for deleted buckets, in case there was a collision
  // but then the original entry was deleted.
  while (hm->states[it] == HM_DELETED || (hm->states[it] == HM_VALID && strcmp(key, hm->keys[it])))
    it = (it + 1) % hm->cap;

  // If we found the right bucket, return the index. Otherwise return an invalid iterator
  if (hm->states[it] != HM_VALID)
    return hm->cap;
  return it;
}

Resizing

We can read from and write to the hashmap, but we’ve yet to provide that resize function we called before.

Resizing is going to be an expensive operation, since it involves allocating memory, iterating through every element in the hashmap and rehashing all their keys. So we don’t want to do it regularly. The first thing we want to determine then is whether we should resize or not. Most of the time we won’t, which will be when our load factor is between 0.25 and 0.75. If our load factor is less than 0.25 we should maintain a minimum size. I’ve picked 128 somewhat arbitrarily.

1

#define HM_MIN_CAP 128

Once we’ve decided to resize, we need to know what the new size will be. We want this to be a large enough increment to avoid having to resize regularly but without growing too large or too small too rapidly. I’ve settled on doubling and halving in size respectively, which seems to be fairly standard.

Knowing the new size we can allocate a new set of buckets. Then we simply have to iterate through all the existing buckets and for any populated ones, rehash the key modulo the new capacity, and write the key-value pairs to the new buckets.

I’ve made this slightly more complicated by defining my key, value, and states arrays separately. They could have been grouped together under a struct hashmap_bucket struct, but this is how I’ve done it. Arguably this way should have better performance since the states array is going to be accessed and iterated through frequently, so keeping it densely packed in memory will be more cache friendly. This is absolutely a premature and untested optimisation, but so be it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

static bool hashmap_resize(struct hashmap *hm)
{
  size_t oldCap = hm->cap;
  size_t newCap;

  // Calculate the new capacity depending on our current load
  // factor
  if (!hm->cap || hm->len * 4 > hm->cap * 3) {
    newCap = oldCap > 0 ? oldCap * 2 : HM_MIN_CAP;
  } else if (hm->cap > HM_MIN_CAP && hm->len * 4 < hm->cap) {
    newCap = oldCap / 2;
  } else {
    // Or if no resizing required, return success early
    return true;
  }

  // Allocate our new buckets
  hm_key *newKeys = calloc(newCap, sizeof *hm->keys);
  hm_value *newValues = calloc(newCap, sizeof *hm->values);
  enum hm_state *newStates = calloc(newCap, sizeof *hm->states);
  // If any of the allocations failed, we need to clean them up
  // and abort. free on a null pointer is a no-op, helpfully.
  if (!newStates || !newKeys || !newValues) {
    free(newStates);
    free(newKeys);
    free(newValues);
    return false;
  }

  // Now rehash all the old buckets, keeping only those
  // holding a value
  for (size_t i = 0; i < oldCap; ++i) {
    if (hm->states[i] != HM_VALID)
      continue;
    size_t it = hashmap_hash_key(hm->keys[i]) % newCap;
    while (newStates[it] == HM_VALID)
      it = (it + 1) % newCap;
    newStates[it] = HM_VALID;
    newKeys[it] = hm->keys[i];
    newValues[it] = hm->values[i];
  }

  // Clean up the old buckets and finally install our new ones
  free(hm->keys);
  free(hm->values);
  free(hm->states);
  hm->keys = newKeys;
  hm->values = newValues;
  hm->states = newStates;
  hm->cap = newCap;

  return true;
}

Putting it all together

With all of the above combined, we have a fully functioning hashmap, and that item is checked off my bucket list. Pardon the pun.

There’s many many improvements that could be made to this in terms of versatility, robustness, and performance, but in my rough testing using my advent of code solution as a benchmark, I found this performed within 10% of the much more mature klib implementation.

As a first stab at a hashmap I’m very happy with that. I hope that you found this exercise as helpful as I did. It turns out that hashmaps are very easy to build so long as you have a decent hash function to use.

So in 2022 I plan to become to learn to touch type “properly”. I currently type using eight out of ten fingers, neglecting my pinkies. I type this way at a decent speed, around 90-110 WPM, but with poor accuracy.

I technically started learning to touch type a few days ago, but new years seems like a convenient way to give myself some accountability for sticking with it and trying to cement the new muscle memory.

So far I’m able to get 60-80 WPM with the proper ten finger touch typing style but only when focusing and mostly typing lower case prose. When programming or typing text with a lot of capital letters or symbols, things get much harder.

I’ve also ordered a new keyboard, the UHK 60 v2. It looks very well made, and I’m hoping that the split layout will aid my posture and ever suffering back. Unfortunately it won’t ship until April, so I have the personal goal of typing at 100+ WPM by then using the proper style.

The only device to perhaps get more use is my phone. My phone is like my laptop in many ways. It’s portable, runs the Linux kernel, and is a critical component of my daily computing needs. However, it is built and programmed by what is probably the largest advertising company on Earth, and one whose reputation for respecting privacy has been on a continuous decline for at least 10 years.

So why do I do most of my communication on a device I’m unable to fully inspect or customise, when that device is replaced more frequently than my laptop and probably costs more? Lack of choice is the obvious answer, which is why I’ve been excitedly monitoring the PinePhone from a distance for a while now.

I want to have a phone that I can trust fully, but it needs to work as a daily driver in 2021 for someone who isn’t willing to be cut off from modern conveniences.

The hardware side of the PinePhone Pro is coming along very nicely but the software sadly isn’t ready for me to daily drive it. So I’ve been thinking a bit lately about what it would take for an open source phone to be ready for me to daily drive.

These are the requirements I’ve come up with:

1. Phone calls

On most handsets, or at least the ones I have spare, the leading Linux distributions aimed at phones do not support making phone calls. This isn’t laziness on the part of the developers, but down to the myriad of undocumented GSM chips in modern hardware and a lack of open source drivers for them.

2. Common instant messaging platforms

I love irc, and I wish it were still the standard for instant messaging, or that xmpp were still loved and supported by the walled garden builders, but unfortunately most people I message on a daily basis through my phone are using Signal or WhatsApp. I want to use an open source device but if that means dropping out of family chats, that’s a deal breaker. I’m not prepared to ostracise myself from non-FOSS communities over this.

3. Web browsing

I’m going to be missing out on a bunch of common apps. That’s a given, but fortunately most services that I use also provide a mobile web interface. So having a decent web browser, preferably with an ad-blocker is a must have workaround.

4. Email client

Like most people, mobile access to my email is critical, whether that’s for accessing theatre tickets or looking up appointment details on the go, it’s mandatory. The client needs to support IMAP and SMTP, and cache my inbox locally so that I can still pull up tickets or similar when I don’t have signal.

5. Full disk encryption

This is also a requirement on my laptop. These devices contain a lot of sensitive information, like our email inboxes or recently taken photos, and are easily stolen or lost. If that should ever happen, I want the peace of mind to know that whatever data is on the device is secure.

Stretch goals

The following items are nice to have but I wouldn’t require them before switching to such a device:

• Offline copy of OpenStreetMap
• A decent camera
• More than 24 hours battery life
• Automatic recording of all phone calls
• Good integration with password managers

PineTime

I also want to spare an honourable mention for the PineTime watch which I have been using for a couple of months now. I’ve not worn a watch for most of my life, and my only previous experience of a smart watch was a Motorola 360G I was once gifted.

I really like the PineTime, for the price it’s a very nice piece of hardware and the fully open source nature of the operating system appeals to me greatly. There’s a few things I’d like to tweak about it if I ever find the time to dig into its guts, but for now that can just be the subject of a future blog post.

I’ve never watched a competitive programmer in action before, so it was fascinating to watch his thought process and methods, since the values being optimised for are vastly different than when building software you intend to maintain for years.

The gist of the problem being solved is: You are given a series of axis-aligned bounding boxes and on/off states. Apply these boxes to a 3D grid, turning on or off all the cells within each bounding box as instructed. After applying all the bounding boxes, count how many cells are switched on.

The first twenty or so bounding boxes are easy enough, sitting within +-50 units from the origin. However, the remainder cover a much larger area, +-100,000 units from the origin. The obvious implication being that the naive solution of constructing the grid in memory isn’t feasible. 200,000 x 200,000 x 200,000 is far too many cells to iterate over in a reasonable amount of time.

My solution for this was to borrow from my hobby game-dev programming experience and manipulate the bounding boxes directly, rather than the cells that they affect.

My algorithm was roughly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

regionList = []
input = [bounding boxes + on/off state]

for isOn,box in input:
  for region in regionList:
    if region fully within box:
      remove region
    if region partially within box:
      region_a, region_b = split region along an edge of box
      remove region from regionList
      add region_a and region_b to regionList
  repeat above until no further changes made to regionList
  if isOn:
    add box to regionList
result = sum of volumes of regions in regionList

What I’m quite pleased about with this algorithm is that it does not have special logic to deal with all the different ways that two AABBs can overlap with each other. The only edge case, pardon the pun, is when two AABBs partially overlap but one is not fully contained by the other. By picking any intersection edge, we can split one of the AABBs in two. With enough passes the intersection will eventually simplify to the complete overlap case.

This is the pure maths approach and I thought it was quite neat, but when I watched Neal’s video I couldn’t figure out what his approach was until he explained it afterwards.

His approach is something I’d never heard of before, and I don’t think I’d ever have thought of it. It’s beautiful in its simplicity.

Build a grid, much like the naive solution, but not out of 1x1x1 cells. Instead, put cell boundaries at all boundaries that the input AABBs will sit on. Let me try to clarify through the medium of ascii art:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

Dense grid of 1x1x1 cells.
+-+-+-+-+-+-+-+-+-+-+-+-+
| | | | | | | | | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+
| | | | | | | | | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+
| | | | | | | | | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+
| | | | | | | | | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+
| | | | | | | | | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+

Sparse grid of differently sized cells.
+-+-+-+-+-+-+-+-+-+-+-+-+
|     |   |     |   |   |
|     |   |     |   |   |
|     |   |     |   |   |
+-+-+-+-+-+-+-+-+-+-+-+-+
|     |   |     |   |   |
|     |   |     |   |   |
+-+-+-+-+-+-+-+-+-+-+-+-+
|     |   |     |   |   |
|     |   |     |   |   |
+-+-+-+-+-+-+-+-+-+-+-+-+

When turning cells on and off by the given AABBs, both grids will give the same answer. By placing boundaries only where absolutely necessary based on the AABBs, the dimensions of the grid and memory requirements for it are vastly reduced. A 5x4x8 region in the dense grid requires a whopping 160 bits of memory to keep track of. In the sparse grid it’s reduced to a single bit, and 5x4x8 would be a very small cell compared to the ones typically generated by the advent of code input. So it’s obvious how this trick reduces the space and time requirements of the problem by orders of magnitude.

These kinds of neat hacks are exactly the sort of thing I’d love to know more of. I don’t know where Neal learned this, but I’m going to be sure to watch more of his competitive programming videos in future to see what other tricks I can pinch from his toolbox.

While solving it, it reminded me of a significant optimisation I made some years ago. Around 2014-2015, I was on a placement year away from my computer science degree. I had a 12 month placement working for the Science and Technology Facilities Council, a public sector gig which had me working on Mantid, an open source data analysis application primarily for use on neutron scattering experiments. That is, data from their particle accelerator. It was a fun first experience of professional software development and a good example of well written and maintained academic software.

One task I had was to improve the performance of various processing and filtering operations that Mantid could apply to a user’s dataset. One such operation required determining the intersection of many pairs of axis-aligned bounding boxes.

This is a very simple calculation, but my profiling informed me that it was taking the vast majority of the eight minute runtime of this particular process.

The intersection test was defined like this:

1
2
3
4
5
6

AABB findIntersection(const AABB &a, const AABB &b)
{
  // if intersection found, return an AABB of intersected area
  // else, throw an exception
}

This function was being called thousands and thousands of times on many pairs of axis-aligned bounding boxes that very rarely overlapped. The result was a deluge of non-stop exception handling.

Fixing that is simple enough fortunately, so I changed the function to look like this:

1
2
3
4
5

bool findIntersection(const AABB &a, const AABB &b, AABB &out)
{
  // if intersection, write it to out and return true
  // else, return false
}

I knew that this would improve the performance, but I didn’t appreciate just what a difference it would make.

The total runtime for the processing decreased from eight minutes down to just eight seconds. Instead of requiring a tea break, the user could carry on working in just the time it takes to have a nice yawn and stretch at their desk.

So when people say exceptions are a terrible way to implement flow control, they aren’t kidding. It doesn’t just muddy the internal flow of your software and make it harder to reason about; it can have a significant user perceivable impact to the way your software performs.

To this day, this remains the most satisfying performance fix I’ve made in my professional career.

They’re a beautifully simple way of positioning the elements in predictable locations in memory. So I’d like to take a few minutes to share what makes them so great, in my mind.

Think of a binary tree like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

               0
             /   \
           /       \
         /           \
       1               2
     /   \           /   \
    /     \         /     \
   3       4       5       6
  / \     / \     / \     / \
 7   8   9  10  11  12  13  14

Excuse the poor proportions. If you lay out the nodes of the binary tree into an array, using the indices given by the diagram above, then some neat numerical properties appear:

1. The children of a node at index i can be found at: i*2+1 and i*2+2.
2. The parent of a node at index i can be found at: (i-1)/2

With these two properties, you can trivially construct a binary tree within an array. The required size of the array can be inferred from the maximum depth the tree will have. If you consider a depth of 1 to be a tree with 3 elements then the required size of the array will always be 1+(1<<depth).

3. The size of the array for a binary tree of a given depth is 1+(1<<depth)

Breadth first traversal is as simple as a for loop over the array, while depth first traversal is the same as any other binary tree. In addition, the tree is densely packed and doesn’t waste space on pointers between nodes, making it very cache friendly.

If you want to iterate through all the nodes at a given depth in the array, the range of indices to iterate through is trivial to calculate with these next two properties:

4. The number of nodes (“width”) at a given depth is 1<<depth
5. The first index at a given depth is width-1.

1
2
3
4
5

struct node array[32];
size_t width = 1<<depth;
for (size_t i = width-1; i < 2*width-1; ++i) {
  struct node *n = array + i;
}

If you want to know the depth of node, the calculation can be derived by inverting the previous two properties.

6. The depth of a node is log2(i+1)

If you don’t want to rely on a maths library, this is easily calculable by finding the index of the left-most 1 bit of i+1.

1
2
3
4
5
6
7
8
9

size_t depth(size_t i)
{
  size_t d = 0;
  i += 1;
  while (i >>= 1) {
    d++;
  }
  return d;
}

On x86 there’s the bsr instruction that accomplishes the same thing more efficiently. Though you probably just want to use the __builtin_clz intrinsic supported by both GCC and clang.

And that’s all there really is to this data structure. The beauty of it is that these properties are all easily rediscovered with a little thought, just by considering a few examples. As a result, the only thing you need to know is the first property.

Neat.

I’ve decided to start blogging again, partly inspired by other blogs that I enjoy, and partly because there are things I’d like to share and discuss in a more public or permanent format than irc and voip with friends.

I’ve also removed several older blog posts, either because the information in them was out of date, or the tone in which they were written isn’t something I would stand behind today.

Expect to see more here in the future.

As a 3rd year Computer Science student, I’m no stranger to LaTeX and the joys of typesetting documents. Most of the time I’m lazy and I just write my reports in Markdown and compile them using the excellent pandoc. Of course, the moment I need a little more control I have to revert back to LaTeX, either embedded in the Markdown or by abandoning Markdown all together, in order to take back control of the document’s formatting. And of course, if I ever need to deviate from LaTeX’s oh-so-specific formatting, as I do from time to time, I enter the seventh circle of hell, with cryptic backdoors and hacks to get LaTeX to work anywhere close to how you want.

If you don’t believe me, find a post-graduate student and hold up a sign that reads \renewcommand. Chances are they’ll have a PTSD induced panic attack. Either that or they used Microsoft Word.

Enter HTML

In one of my more coherent LaTeX compile failure panic attacks I had an epiphany. I’d just done some work in HTML and CSS, and it’d been a great success. What if I could use HTML and CSS to write documents in for print? Well, it turns out not to have been such an original idea. Thomas Park built an interesting CSS framework for academia called PubCSS.

Pretty cool work, and the examples are really polished. But in the CS tradition of NIH syndrome, I decided I wanted something bespoke for my needs.

Now, I know what most of you might be thinking. HTML+CSS? What about page numbers, tables of contents, citations?

Well, it turns out HTML+CSS can actually manage all of those, with the exception of the table of contents. You can either built that manually and use CSS to automatically display the correct page numbers for the relevant sections, or you you can write some JavaScript to generate the table of contents automatically. Being a programmer, I took the lazy route: I automated it.

So what new magic gives us page numbers, automatic citations, and more? Well, CSS Generated Content for Paged Media. A deceptively dull title for a really cool feature.

Here’s an example of its power:

1
2
3
4
5
6
7
8

@page {
  size: a4;
  margin: 3.5cm;

  @bottom-right-corner {
    content: counter(page);
  }
}

That simple bit of CSS just let me select a paper size (you can specify exact dimensions in metric or imperial if you like too), set a margin, and then put some arbitrary content in the bottom right corner of every page. In this case, the page number. Oh, and you can control the formatting of the page number, set and reset the count at any time, or display the current chapter title, the possibilites are almost endless.

So let’s say you have a front cover, some preamble, and then the main content. Your front cover doesn’t want any page numbers, bur your preamble wants page numbers in lower case roman numerals, and your main content wants page numbers starting from 1 in our regular arabic numerals. Easy!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

<body>
<div>
  <h1>This is my front page! Isn't it cool.</h1>
  <p>I'd like to thank the academy...</p>
</div>

<div id=preamble>
  <h1>Abstract</h1>
  <p>In here I put all the content for my preamble.</p>
</div>

<div id=main>
  <h1>Introduction</h1>
  <p>This is the main content. Here I want normal page numbers.</p>
</div>
</body>

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

#main {
  page: main; /* Set current page */
}

#preamble {
  page: preamble;
}

@page main { /* For "main" pages */
  reset-counter: page; /* Restart the page count */
  @bottom-right-corner {
    content: counter(page);
  }
}

@page preamble {
  @bottom-right-corner {
    content: counter(page, lower-roman); /* lower-roman format this time */
  }
}

Page breaks

So what do we do when we want a page break? Well, there’s a useful set of css properties for influencing them:

• page-break-before
• page-break-after
• page-break-during

Which we can set to values such as always or avoid. For example, if we want all <h1>s to start on a new page:

1
2
3

h1 {
  page-break-before: always;
}

It’s really that easy.

If you want to force a new page somewhere arbitrary you can always create a class for that:

1
2
3

.page-break {
  page-break-before: always;
}

1
2
3

<p>Something at the end of one page</p>
<div class="page-break" />
<p>This is on a new page!</p>

Section numbers

Want to number your sections? That’s easy too. Here’s an example of a two-level numbering scheme.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

h1 {
  counter-increment: section;
  counter-reset: subsection;
}

h2 {
  counter-increment: subsection;
}

h1::before {
  content: counter(section) ". ";
}

h2::before {
  content: counter(section) "." counter(subsection) ". ";
}

And now our sections are numbered automatically for us.

References

This is starting to get a little tricksy, but not much more so than before. This will give us nice pretty engineering style references. They even act as intra-document links on a PDF.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

#bibliography {
  counter-reset: ref;
}

#bibliography li {
  margin: 0.5em;
  counter-increment: ref;
}

#bibliography li::marker {
  content: "[" counter(ref) "]";
}

a.ref::after {
  font-style: normal;
  content: "[" target-counter(attr(href), ref) "]";
  font-size: 0.7em;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

<div id="main">

<p>
  In some cases, foos were found to be quite dangerous<a class=ref href="#foo"></a>.
  However, in <a class=ref href="#bar">A Study on Bars</a> this was disputed.
</p>

</div>

<div id="bibliography">
  <h1>Bibliography</h1>
  <ol>
    <li id="foo">
      Foos considered harmful.
      Author Name Here.
      2010.
    </li>
    <li id="bar">
      A talk on foos considered harmful.
      Someone Else
      2012.
    </li>
  </ol>
</div>

And the result is perfect. You can use the same technique to number and refer to figures easily.

Footnotes

These turn out to be even easier. Try it.

1
2
3

.footnote {
  float: footnote;
}

1

<p>This is a sentence<span class=footnote>As sentences go though, it's pretty dull.</span>.</p>

Columns

Also easy.

1
2
3
4
5
6
7
8
9

#main {
  column-count: 3;

}

#main h1 {
  column-span: all; /* let <h1>s break through columns */
}

Images

This I saved til last because it’s what irritated me about LaTeX the most. I wanted an image in my document scaled by half and aligned to the right, with a black border and a caption beneath.

In LaTeX it’s a nightmare. In HTML it’s second nature.

1
2
3
4

<div style="width: 50%; float: right; border: 1px solid black;">
  <img src="img.png" style="width: 100%;" />
  <p>This image is awesome.</p>
</div>

Compiling

HTML is all very well, but how do you get your glorious PDF output at the end? The answer is a wonderful tool called PrinceXML. It really is that good.

For commercial use there’s a fee, but for non-commercial use you can download and use it freely. The only caveat is that the free version adds an annotation to the first page of the output PDF declaring that it was generated by Prince. The annotation is easily removed using a standard PDF viewer, so it presents no trouble to an academic. If you’re looking to generate PDFs automatically, I highly recommend you splash out on a license.

And yes, you can invoke it using GNU Make:

1
2

report.pdf: report.html style.css
	prince -v -o $@ $<

Lately, I’ve been devoting a fair bit of my time to working on a single page web application, for which I’ve been using the excellent React.js and Redux frameworks for managing my UI, and the client side state. An honourable mention goes to Material UI for the nice React components I built the interface with.

For various reasons, the server component of this application had to be built in C++, and it was required that the server could push events or data to the client. While at first I toyed with the idea of a RESTful HTTP API with either long-polling or websockets for events, I quickly abandoned this approach in favour of a pure websocket API.

Combined with the power of React+Redux, using websockets for the application led to a very smooth, and pain free development experience. The purpose of this post is to share the approach I took to handling the websocket connection with Redux, and my experiences developing with this architecture.

This post assumes some rudimentary knowledge of Facebook’s Flux architecture, and how Redux implements that, specifically the unidirectional data flow.

Websocket Middleware

In a Redux based application, the typical path of user interaction is as follows:

1. User presses button
2. Action is created
3. Action is delivered to the reducver
4. The reducer updates the store’s state
5. The UI is updated with the new state

For user interactions that have to communicate with the server, a detour is clearly needed in step 3. Fortunately, Redux provides a way to hook into actions at that step, namely middleware.

I created a custom middleware responsible for opening a websocket to the server, maintining it, and intercepting or dispatching actions as appropriate. That is, any actions that needed to be dealt with by the server were intercepted at step 3, and the appropriate message sent to the server. When a message is received from the server, an equivalent action would be dispatched to the reducer to update the state, as per the new information received from the server.

The new path of user interaction is as follows:

1. User presses button
2. Action is created
3. Action is delivered to socket middleware. If needed, the middleware re-routes the action through the websocket, and does not deliver it to the reducer. Otherwise, action is given to reducer as usual.
4. The reducer updates the store’s state
5. The UI is updated with the new state

With additional path for when a message is received from the server:

1. The socket middleware receives a message from the websocket
2. The socket middleware creates an action, and dispatches it to the reducer
3. The reducer updates the store’s state
4. The UI is updated with the new state

This design is clean, and easy to reason about. For both the user and redux, the detour and interception of remote actions is invisible. Additionally, since all redux actions are serializable, you can record and replay actions that are delivered to the reducer, or to the middleware, providing excellent debugging opportunities.

To help demonstrate this solution, I’ve adapted my middleware to provide an example websocket middleware anyone can base theirs off:

socketMiddleware.js

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

import actions from './actions'

const socketMiddleware = (function(){ 
  var socket = null;

  const onOpen = (ws,store,token) => evt => {
    //Send a handshake, or authenticate with remote end

    //Tell the store we're connected
    store.dispatch(actions.connected());
  }

  const onClose = (ws,store) => evt => {
    //Tell the store we've disconnected
    store.dispatch(actions.disconnected());
  }

  const onMessage = (ws,store) => evt => {
    //Parse the JSON message received on the websocket
    var msg = JSON.parse(evt.data);
    switch(msg.type) {
      case "CHAT_MESSAGE":
        //Dispatch an action that adds the received message to our state
        store.dispatch(actions.messageReceived(msg));
        break;
      default:
        console.log("Received unknown message type: '" + msg.type + "'");
        break;
    }
  }

  return store => next => action => {
    switch(action.type) {

      //The user wants us to connect
      case 'CONNECT':
        //Start a new connection to the server
        if(socket != null) {
          socket.close();
        }
        //Send an action that shows a "connecting..." status for now
        store.dispatch(actions.connecting());

        //Attempt to connect (we could send a 'failed' action on error)
        socket = new WebSocket(action.url);
        socket.onmessage = onMessage(socket,store);
        socket.onclose = onClose(socket,store);
        socket.onopen = onOpen(socket,store,action.token);

        break;

      //The user wants us to disconnect
      case 'DISCONNECT':
        if(socket != null) {
          socket.close();
        }
        socket = null;

        //Set our state to disconnected
        store.dispatch(actions.disconnected());
        break;

      //Send the 'SEND_MESSAGE' action down the websocket to the server
      case 'SEND_CHAT_MESSAGE':
        socket.send(JSON.stringify(action));
        break;

      //This action is irrelevant to us, pass it on to the next middleware
      default:
        return next(action);
    }
  }

})();

export default socketMiddleware

Making use of the middleware is also a breeze. More information on this can of course be found in the documentation for redux middleware.

store.js

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import { createStore, applyMiddleware } from 'redux'
import thunk from 'redux-thunk'
import reducer from './reducer'
import socketMiddleware from './socketMiddleware'

export default function configureStore(initialState) {
  return createStore(reducer, initialState,
      applyMiddleware(thunk, socketMiddleware)
  )
}

This fairly straightforward middleware handles our entire websocket with ease. It produces and consumes actions as needed, and allows us to effortlessly update the local state based on information received from the server, and to inform the server of any user actions, all without relying on messy callbacks or violating the unidirectional data flow.

I have found this approach to be very powerful, and would highly recommend it to anyone considering building a web application with websockets.

The Why

As a result of the recent public discussion regarding online privacy I decided to take a few measures to give myself some more privacy online. One such measure was the creation of a new 4096 bit PGP public key for myself. I also started running my DNS requests through tor for some more privacy, which was inspired by Drew’s excellent blog post.

Lastly, I decided it was time to stop letting Google handle all the mail for my domain. This was partly because it’s been costing me roughly the price of a pint of beer in London each month, and partly because it seemed like it would be a fun learning experience.

As a note to self, and to spare you the fun part of the learning experience I thought I’d share the details of my set up to show how straightforward it can be to get a solid mail server online.

Overview

For my mail server there are three key pieces of software at play:

• postfix - Provides an SMTP server
• dovecot - Handles mailboxes, authentication, and IMAP
• MariaDB - SQL database of email accounts

Inbound mail is given to postfix via SMTP. Postfix then checks against the SQL database whether the email address should be accepted, and performs spam filtering. If the email passes postfix’s checks it’s passed to dovecot over a unix socket via LMTP. Dovecot is then responsible for placing the mail in the correct user’s mail directory.

Email are accessed over IMAP using dovecot. Upon connection, users are authenticated by dovecot against accounts in the SQL database, and on success are connected to their mail directory.

Outbound mail is also given to postfix, however this time postfix requires authentication. Postfix sends the login details to dovecot over a unix pipe, which checks against the SQL database whether the user is permitted to send mail. If so, postfix is given the approval to do so, and relays the email to the relevant remote SMTP server.

Configuring MariaDB

Create a new database. I called mine maildb. Also create a new MariaDB user, mail with read-only access to maildb.

Within maildb you want to create the following three tables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

CREATE TABLE `aliases` (
  `mail` varchar(120) NOT NULL,
  `destination` varchar(120) NOT NULL,
  UNIQUE KEY `mail` (`mail`)
);

CREATE TABLE `domains` (
  `domain` varchar(120) NOT NULL
);

CREATE TABLE `users` (
  `email` varchar(128) NOT NULL,
  `crypt` varchar(128) NOT NULL,
  PRIMARY KEY (`email`),
  UNIQUE KEY `id` (`email`)
);

Aliases Table

The purpose of aliases is for an email address mapping step performed by postfix. Every email that is received locally has its address transformed using this lookup table.

To receive email at an address it is required to be covered by this table. I have harry@exec64.co.uk mapped to harry@exec64.co.uk, which passes my mail through without changing the destination.

However, if you’re the administrator of the server, you’ll want to receive mail for users such as postmaster@example.com, so you’d need to map that to your_address@example.com.

You can also provide a catch-all address for a domain by excluding the name portion of the email address. In my case that would be mapping @exec64.co.uk to harry@exec64.co.uk.

Domains Table

This table is even simpler to configure. It should simply contains all the domains for which your server should receive mail for. It’s used by postfix to determine whether to send mail to another server, or to pass it to dovecot.

In my case it simply contains exec64.co.uk.

Users Table

This is the most important table, which is used solely by dovecot to authenticate users. The email column should contain the users full email address, and the crypt column should contain an encrypted form of their password.

Here’s an example query to insert a new user:

1

INSERT INTO users (email,crypt) VALUES ('user@example.com',encrypt('plaintext_password', CONCAT('$5$', MD5(RAND()))) );

Configuring Dovecot

Now that the SQL database has been set up correctly, the next step is to set up Dovecot to authenticate users, manage their email directories, and provide an IMAP server for users to retrieve their mail from.

The recommended way to configure dovecot seems to be to have a main configuration file at /etc/dovecot/dovecot.conf which includes many other configuration files from /etc/dovecot/conf.d/ but I found that to be an overcomplication for my needs. Simply having one relatively short config file was perfectly adequate.

Create the virtual user

Traditionally email addresses were tied to unix users on servers, and the mail was stored in each user’s home directory. In our case, we want our email addresses to be independent of the users on the system, so instead all the mail will be “owned” by a virtual user named virtual. (Forgive me, I don’t study creative writing.)

To create the user and the mail directory run the following commands:

1
2
3
4

sudo mkdir /var/spool/mail/virtual
sudo groupadd --system virtual -g 5000
sudo useradd --system virtual -u 5000 -g 5000
sudo chown -R virtual:virtual /var/spool/mail/virtual

Now we’re ready to configure dovecot itself. It’s all fairly straight forward. Here’s a copy of my configuration file. Everything ought to be fairly self explanatory, and if it isn’t, dovecot has some pretty good documentation.

/etc/dovecot/dovecot.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

#provide acesss to imap and ltmp
protocols = imap lmtp

#don't handle mail without SSL silly
#if you don't have a cert go to LetsEncrypt.org
ssl = yes
ssl_cert = </path/to/fullchain.pem
ssl_key = </path/to/privkey.pem

listen = *, ::

base_dir = /var/run/dovecot/

instance_name = dovecot

login_greeting = Dovecot ready.

login_trusted_networks = 127.0.0.1

disable_plaintext_auth = no

auth_mechanisms = plain login

#Mail is stored in "Maildir" in each user's home directory
mail_location = maildir:~/Maildir

# authenticate users using SQL
passdb {
  driver = sql
  #this config file is provided beneath
  args = /etc/dovecot/dovecot-sql.conf.ext
}

userdb {
  driver = static
  #store each user's mail in /var/spool/mail/virtual/name/domain
  #i.e. virtual/harry/exec64.co.uk
  args = uid=virtual gid=virtual home=/var/spool/mail/virtual/%d/%n
}

namespace inbox {
  inbox = yes
}

#the mail is owned by the user/group named "virtual"
mail_uid = virtual
mail_gid = virtual

auth_mechanisms = plain login

#provide an authentication service to postfix
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }
}

protocol lmtp {
  #this option is required, or you won't be able to receive mail
  postmaster_address = postmaster@exec64.co.uk
}

#accept delivery from postfix via lmtp
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0600
    user = postfix
    group = postfix
  }
}

/etc/dovecot/dovecot-sql.conf.ext

1
2
3
4
5

driver = mysql
connect = host=localhost dbname=maildb user=mail password=mailpasswordhere
default_pass_scheme = SHA256-CRYPT #make sure this matches the encryption scheme you use

password_query = SELECT email as user, crypt as password FROM users WHERE email = '%u';

Testing dovecot

At this point, you should now have a working IMAP server that you can connect to and retrieve mail from. Of course you won’t have any mail to read, but you should still verify that you can now connect, and also check that the relevant mail directory was created in /var/spool/mail/virtual for your mail.

Configuring Postfix

And now for the final step, providing an SMTP server to send/receive mail with.

In my case, I only needed to set up one main configuration file for postfix, and a couple of other files to allow it to query the SQL database.

Please note that this configuration doesn’t provide any spam filtering.

Again, if something doesn’t make sense to you, RTFM.

/etc/postfix/main.cf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

#We don't need to set the domain here, it's configured by the virtual mailboxes
mydomain = localhost
myhostname = $mydomain
myorigin = $mydomain
mydestination = localhost
mynetworks_style = host
relay_domains = $mydestination
relayhost =
smtpd_banner = $myhostname ESMTP

#virtual mailbox settings
virtual_mailbox_base = /var/spool/mail/virtual
virtual_mailbox_maps = mysql:/etc/postfix/sql_mailbox.cf
virtual_alias_maps = mysql:/etc/postfix/sql_alias.cf
virtual_mailbox_domains = mysql:/etc/postfix/sql_domains.cf
#uid of virtual user is 5000
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000
#tell postfix to send mail for virtual inboxes to dovecot
virtual_transport = lmtp:unix:private/dovecot-lmtp

#aliases
alias_maps = hash:/etc/postfix/aliases
alias_database = hash:/etc/postfix/aliases

#tls parameters
smtpd_tls_cert_file=/path/to/fullchain.pem
smtpd_tls_key_file=/path/to/privkey.pem
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_security_level = may
smtpd_tls_security_level = may
smtp_tls_note_starttls_offer = yes
smtpd_tls_loglevel = 2
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom

#authentication settings
smtpd_sasl_auth_enable = yes
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
smtpd_recipient_limit = 16
smtpd_soft_error_limit = 3
smtpd_hard_error_limit = 12
smtpd_helo_restrictions = permit_mynetworks, warn_if_reject reject_non_fqdn_hostname, reject_invalid_hostname, permit
smtpd_sender_restrictions = permit_mynetworks, warn_if_reject reject_non_fqdn_sender, reject_unknown_sender_domain, reject_unauth_pipelining, permit
smtpd_client_restrictions = reject_rbl_client sbl.spamhaus.org, reject_rbl_client blackholes.easynet.nl
smtpd_data_restrictions = reject_unauth_pipelining
smtpd_helo_required = yes
smtpd_delay_reject = yes
disable_vrfy_command = yes

#other
compatibility_level = 2
queue_directory = /var/spool/postfix
command_directory = /usr/bin
daemon_directory = /usr/lib/postfix/bin
data_directory = /var/lib/postfix
mail_owner = postfix
inet_interfaces = all
local_recipient_maps =
sendmail_path = /usr/bin/sendmail
newaliases_path = /usr/bin/newaliases
mailq_path = /usr/bin/mailq
setgid_group = postdrop
html_directory = no
manpage_directory = /usr/share/man
sample_directory = /etc/postfix
readme_directory = /usr/share/doc/postfix
inet_protocols = ipv4
meta_directory = /etc/postfix
shlib_directory = /usr/lib/postfix
delay_warning_time = 1h
unknown_local_recipient_reject_code = 450
maximal_queue_lifetime = 7d
minimal_backoff_time = 1000s
maximal_backoff_time = 8000s
smtp_helo_timeout = 60s

We also need to tell postfix how to query the information it needs from the SQL database.

/etc/postfix/sql_alias.cf

1
2
3
4
5

user=mail
password=password
hosts=127.0.0.1
dbname=maildb
query = select destination from aliases where mail = '%s'

/etc/postfix/sql_mailbox.cf

1
2
3
4
5

user=mail
password=password
hosts=127.0.0.1
dbname=maildb
query = select 1 from users where email = '%s'

/etc/postfix/sql_domains.cf

1
2
3
4
5

user=mail
password=password
hosts=127.0.0.1
dbname=maildb
query = select 1 from domains where domain = '%'

DNS

The final step to get postfix working is to tell the world where it is. To do this you need to set a MX DNS record. The purpose of an MX record is to indicate where a domain’s mail server is hosted. While most DNS records should resolve to IP addresses, for MX records you should be returning another domain name. This is to allow resolving that domain to either an IPv4 or an IPv6 address.

MX records also have a priority field. With a single record it’s irrelevant so you can just set it to 10. For the address, give the domain name of the server that postfix is running on.

Final Thoughts

Congratulations, you should now have your own mail server.

You’ll probably want to add spam filtering, and a few other goodies. There’s a good guide to setting that up here.

Have any questions or corrections? Leave a comment below, or drop me an email.

This time around my site is generated entirely with Hugo, a neat tool for statically generating websites out of markdown and some templates.

I played around with a few themes for a while, but couldn’t find one I liked enough that I could tweak to suit my needs easily, so in the grand tradition of Not-Invented-Here syndrome I wrote my own from scratch.

Stay tuned for further updates!