July 23, 2007

I am allergic to spam.

One of my uncle's friends, Marc Breitsprecher, runs an internet business from his home selling ancient coins. Back in 2000, before I even started my undergrad, he approached me and asked if I could build a web site for him. Previously, he was just selling his coins on eBay.

Over the last seven years, Ancient Imports has grown beyond both of our expectations. He was able to quit his job at the postal service and work on the site full-time. We've outgrown two hosting providers, the most recent event happening a few weeks ago.

We moved from a poor shared hosting environment to a spiffy virtual private server. It's the closest thing to having full control over a physical machine as we can get right now. It's fun for me because I essentially have full control over the virtual machine, which means I'm pretty much free to do whatever I need to do to implement new functionality. The cost, however, was that I also have to maintain the security, e-mail services, and DNS that were previously dealt with by the hosting provider. The classic blessing and curse.

We were under a somewhat tight deadline to switch over (long story short, they blamed us for the problems we had with their service--that did not sit well with either of us), so I just threw up the e-mail server and configured it to make sure the mail was still delivered. That meant no spam protection. After about two weeks, the deluge of spam started to bother me enough to do something about it.

I was pleasantly surprised at how effective just a few anti-spam measures were. The first counter-measure I added was to make the server a bit more strict as to what it will accept as a properly formatted message (e-mail originating from domains that actually exist, etc.) which I assume is not immediately recommended because of the extra DNS lookups it incurs. The second was to check a DNSBL to see if the originating IP address is a known spammer (also another DNS lookup). Both of these tweaks killed a bunch of spam with the small effort of adding four lines of code to the configuration file.

The second, more involved, counter-measure I added was greylisting. This is a really nifty technique that is mostly invisible to people sending or receiving mail. The e-mail server will feign unavailability to any sender and recipient pair that it hasn't seen before. Upon receiving the temporary error message, normal e-mail servers will attempt to redeliver the message in another 10-20 minutes, at which point the server will remember the previous attempt and accept the e-mail.

Most software used by spammers to send their advertisements, however, are not so well behaved. They're more interested in sending out as much mail as possible in the shortest amount of time. This means that the spammer is unlikely to attempt a redelivery (at least using the same source e-mail address) to the same person in a reasonable amount of time. Even if they do, they greatly increase their chances of appearing in the DNSBL the next time they attempt to connect. Greylisting essentially gives you a two-for-one special.

As for numbers, the server's been running with the new counter-measures since early Sunday morning. Since then, 611 attempts were blocked by the SpamHaus DNSBL, 163 attempts were greylisted, and only 40 e-mails were actually delivered to Marc and myself (and most of those originated from the website itself and not from outside sources). Of course, the default behavior of rejecting e-mail for unknown users (and domains) is the most effective "counter-measure." In the same period, the mail server rejected 3,957 attempts at sending mail to ghosts.

The central e-mail servers employ greylisting as one of their anti-spam techniques and make liberal use of DNSBL by default. They even let you control whether or not to use those blacklists for your e-mail (just go to and click on "Incoming E-mail Controls") as well as whether or not to scan incoming e-mail with SpamAssassin. OIT Data Security and Internet Services lists their mail server statistics in the UM Tech Brief.

There was no point to this entry. Just that anti-spam techniques actually work.

Darn spam.