March 11, 2004

Lump of Coal upgraded to Kryptonite: Amavisd-New

I've blogged before about the ongoing war to combat the unholy blight of spam around here, and last week I deployed a new weapon in the effort. It's called amavisd-new, and it takes the place of the previous SMTP proxy we were using (called spampd, which has served us nobly and well). The upshot is, we get some nice additional customization options and potent antivirus protection (courtesy of clamav) all under one roof. Techie details follow... The most useful customization option, and the one which took me a bit of headscratching to get right, is the idea of variable spam thresholds. As the FAQ says:
  • tag level is where X-Spam-Status and X-Spam-Level header fields start to appear (e.g. setting tag level to 0 (or even better to -999) would turn this on permanently);
  • tag2 level is where a message is considered spam as far as mail header fields are concerned: the X-Spam-Flag: YES header field appears, the X-Spam-Status gets a YES, Subject gets a ***SPAM*** if subject editing is enabled;
  • kill level is where a message is considered spam and countermeasures are taken: (reject/bounce/discard/pass), quarantine, notify, adding optional recipient address extension). It is common to set tag2 level the same as kill level, but some may prefer to set kill level even higher, perhaps combined with $final_spam_destiny=D_DISCARD;
I had to chew on that a bit, so let me post the relevant bits of our amavisd.conf and describe what it ends up doing to the spam.
$sa_tag_level_deflt  = -999.0; 
$sa_tag2_level_deflt = 5.0;
$sa_kill_level_deflt = { '' => 5, '' => 600, '.' => 20};
$final_spam_destiny  = D_DISCARD;  
So with $sa_tag_level_deflt = -999.0, we make sure that everything gets tagged. This is mostly just to make sure that SpamAssassin is working properly. Our $sa_tag2_level_dflt = 5.0 sets our threshold for what ought to be considered spam. The value for $sa_kill_level_deflt is an anonymous hash reference to key => value pairs where the key is somebody's email address (the '.' sets the default) and the value is the spam score at which the destiny will be invoked.
Since $final_spam_destiny is set to "discard", anything that gets a SA score of 20 or above goes straight into the bit bucket while crafty amavisd-new tells the sending MTA that it accepted the message, so no bounces are generated. There is a low kill level for '' because that address goes into our Request Tracker queue and tickets being auto-generated by spam causes me to curse and gnash my teeth. There's a high queue level for '' because it is a spamtrap address that feeds directly into the SpamAssassin Bayes database.
(You hear that, trawling spam-address harvesters? is a great address to send some email to! I hear they're particularly interested in performance-enhancing pharmeceuticals...)
I somewhat arbitrarily picked the 20-point default because it seems high enough that the odds of a false positive are very slim, but low enough that we can cut down on the amount of outrageously spammy spam we have to store on disk, even in your spam folder.

At any rate, this is a lengthy way of explaining how amavisd-new implements something that I had previously thought was not possible; namely, per-recipient preferences for SpamAssassin thresholds. Because it's a SMTP proxy, it knows each of a message's ultimate recipients, receives and saves a copy of the whole message, and then decides for each destination address whether to tag-and-forward or discard it.

And I haven't even talked about the Antivirus part, which is also really cool -- hourly automatic virus signature updates? same-or-better coverage and response compared to the big boy AV vendors? for FREE?! Hell yeah! Big props to the clamav and amavisd-new developers, this is one little corner of the net that's a brighter place thanks to you. Posted by eric at March 11, 2004 03:03 PM