Alan Doherty Level Double-A conformance icon, W3C-WAI Web Content Accessibility Guidelines 1.0
Valid CSS! Valid HTML 4.01 Strict

Blacklisting and spamfiltering

I was sent here by a bounced e-mail

Read the sections below which outline why you may have recieved this message. Alternativly use the error message list to find the section relevant to your problem.

the problem

Due to the increasing volumes of spam/U.C.E. and virus laden e-mails being sent to our users, somewhat Draconian measures have had to be employed to keep our users mailboxes safe and keep e-mail from becoming useless as a communications method.

Our ever evolving solution so far

we only accept valid helo/ehlo names
if your mailserver sends a non legally allowed hostname {only a FQDN or ip-address is valid} after the HELO/EHLO greeting it will be rejected. [if your mail isn't spam and you are getting perrors due to this please contact your mailsevers admin and insist they configure your mailserver correctly or hire a qualified/experienced replacement] {non-hardcoded errors are delayed till after recipient specified as you are still allowed to e-mail postmaster and abuse with bad helo, and any users who do not want to use spam checking}
common examples are [examples preceded by * are hardcoded and cannot be exempted ever]
  • * no name given
  • * underscores in name, common ms-windows server config error utterly illegal
  • * / in name, no idea where this comes from but its illegal
  • single word name, it should be the FQDN or ip-address {thus must have at least one .}
  • leading or trailing ".", it should be a FQDN or ip-address {thus no leading or trailing "."}
our recipients address are all rfc compliant
no funny or unsupported characters in any of the address we use/allow our users to use, if the address your using has them it likely will not work anywhere
sender host white-lists
default allowing of mail from known good sources of non-forgeable {gmail, yahoo hotmail etc.} or otherwise traceable incomming e-mail {local isp's outgoing mailservers who regularly end up on public blacklists, but continue to send more signal than noise and our users regularly converse with.}
sender host public black lists

next we compare the sender to the following external blacklists, whitelisted servers bypass all checks

if you find your providers mail server to be on one of those lists you can apply to us for whitelisting, but also you should apply to the above list admin for removal too, and complain to your isp to remove the customer responsible for their listing

if this has happened to you follow the relevant link above to follow their appeals proceedure, or contact us for consultation on fixing if it is your own server that is blacklisted or request whitelisting below {unusual but sometimes we'll consider extenuating circumstances}

private dodgey country black list

Next we add a warning if the e-mail came from ip address designated as belonging to countries that have a high rate of pc infection and/or hosted spaming services {currently too many to list}, to prevent inbound viruses and spam from trojaned pc's. With known good exceptions added to the whitlist above {if and when we're informed of them by you}

so if you get an e-mail from a known good source that has been labled "X-Spam-Warning: [ip-address] is in a country/ip-block with high spam/low-zero mail volume" send me the headers via e-mail or by cutting and pasting them into this form

also if you get bounces claiming simmilar just use the form above to send us the full mail you got back, {full e-mail so we can determine it wasn't a spam only}

sender verification

Next we verify the envelope_sender {the address the e-mail is sent from} this is usually but not always the from: address you see {but always the Return-path: in the headers [where bounces should be directed to]} if it fails verification the e-mail is refused entry to the system this can cause issues for some badly setup mailing lists {those who send from a non-existant address, understandably suspicious} and for mail comming from systems that refuse bounces {which understandably makes them look suspicious} unfortunatly in both these circumstances no one recieves the warnings {as the sender didn't exist!}

if you believe that your e-mail is not arriving from such a mailinglist, or because the sending mailsystem is blocking bounces, please contact me directly so we can analize the logs identify the problem system and temporarilly whitelist them, but we will need them to repair the senders system for their sake and everyone elses

relay not allowed
obviously, we check the mail is going to: one of our users, if not we won't send it on for you! Nuff said
content analasis - attachment blocking
we block .exe .pif .bat .scr .lnk .com files by default, if you need to send them then try putting them in an archive, better yet don't use e-mail for file transfer
content analasis - non western character sets
currently blocking some chineese spam by blocking mails in that encoding, will be reviewed as soon as any user needs to recieve e-mail in chineese
content analasis - malware scan
if sending host and sender are ok thus far the e-mail is checked for viruses and refused {understandable} if virus is found. This doesn't mean our users still shouldn't still be wary of attachments and use their own local scanners too {as a well written virus can be spreading for many days before anti-virus tools can detect it, not to mention non-viral forms of malware that can be around for months and can be sent as a targeted attack}
content analasis - spam assasin
finally we get spam assasin have a look at the e-mail and it gives it a "spam score" based on how spammy it looks based on the "rules of the day" these are the best content analisis and comparisons available at the time, if the message scores over 10 it is rejected at smtp time, if between 5 and 10 it is marked as likely spam with the header "X-Spam-Flag: YES" additionally all mails get the header "X-Spam-Score: #.# +++" where the score is given as a number accurate to 0.1 of its score and as a row of 0-9 "+" symbols equaling the score as an integer. these are available for futher filtering on the recieving client side
client side filters and reporting
we strongly encourage our users to use client side filters to deal with the few remaining mails that get through. especially those in the 5-10 spam score range as these need a human to verify they arn't spam {sales news letters are an excelent example}. additionally we encourage the use of spamcop's excellent reporting service to report any and all spams that get through. {as this allows the community based filters to learn to adapt to deal with these new, and previously unrecognised variants}

These methods have cut spam to our users by 99.5%, with little effect on valid mail so far {2 valid mails refused}}

If you are seeing this message you may be the exception if so read below as we have ways round this for you.

for this reason if you arnt a spammer and your mail has been bounced for your and others sake please report the mistake by cutting and pasteing the bounce into this form {and i'll investigate} and after tracing the cause i'll either whitelist your mailserver or inform you why i cannot and how to proceed

error messages guick reference

click on the appropriate bounce messages below {the one you recieved} to see which section applies in your case

the unique and static parts of the message are linked as some of the other surronding text varies or is repeated in other error messages

Last updated Dec. 2008 Alan Doherty