After I had not received mail for a couple of days I suddenly started to receive postmaster errors from one of the servers I maintained. The message told me the mail could not be delivered. On further inspection it seemed that this was a problem because postfix could not connect to [127.0.0.1].
This meant of course that some daemon on the server was out for lunch and it was screwing up the mail deliveries. Using ‘postqueue -p’ to inspect the queues I found out that 4000 messages where packed up in the queue, filling her up to the maximum I allowed.
After running ‘postsuper -r ALL’ (which means, re-queue all messages for instant delivery) I tailed the ‘/var/log/mail.log’ file to find out that 20% of the messages got delivered and the rest wound up back in line.
One error that kept flashing by was: ‘TROUBLE in check_mail: parts_decode_ext FAILED: run_command (open pipe): Can’t fork at…’. It had something to do with the FileIO class. Now the message came from Amavis and in fact, amavisd is nothing more than a Perl framework, hooking all kinds of filters together. This means its memory hungry but won’t totally crash upon hitting the floor.
This explained why a part of the messages made it through: a few threads fitted in memory, allowing for completing the delivery, however most mail got rejected once again and returned faithfully to the waiting queue. Another inspection of the queue confirmed this: after 2 attempts 3200 messages where still there.
Then it dawned on me: I had not enabled the swap memory! After fixing this (and adding 1GB swap to the 256MB of RAM) and reloading Amavis I tried once again to requeue all mail. This time the mail got through the filters in one piece (and well within the memory boundaries). I wish all issues with servers were as simple as this ^-^.