Alan Doherty's Advice on how to setup/run a Mailserver well

aside: Why run your own instead of use someone elses

you can avoid the below requirements by using your connection-provider/Internet service provider { hereafter ISPs} provided relay {for outgoing email} by either directing your clients at it or preferably configuring your internal mailserver/Mail Transfer Agent {hereafter reffered to as MTA} to send via them. But this means you depend on their reputation, which may be less than perfect occasionally, as it can be tainted by you or any of their other customers sending spam or them having mal-configured machines or DNS

The upside if you run your own MTA correctly (and never spam) is a vastly better reputation than any provider or ISP can ever achieve, but this takes time as you will be beginning with no reputation good or bad.

the downside the mail-queueing re-trying and storage costs but as I assume you want to run an MTA well we will move on.

Why you should need do these things

These Requirements are not a minimum for operating a functional MTA, but are rather the minimum required for operating a trustable traceable authentic-looking reputable and unlikely to be refused service MTA

One that will comply with the strictest requirements from those receiving your e-mail {and trying to verify wheither or not your mail comes from an MTA or a bot-infested PC}

Obviously if everyone followed ALL these steps telling mail from an MTA from mail from a bot would be much simpler, unfortunatly not enough people do yet, and thus a lot of bot-spam gets delivered and a lot of badly configured 'legitimate' MTA mail gets refused, through the guesswork having to be done by recieving systems

scope

scope here is the MTA only not the content traveling via the MTA that is logically a seperate realm entirely, also remember not all MTAs receive mail and not all MTAs send mail, although admittedly most do some of both

main bullet points will be catagorised by MUST and SHOULD;
MUST: if not done /done incorrectly will count against the legitimacy of your MTA
SHOULD: will not normally detract from percieved legitimacy if not done, but if done correctly it will add to your percieved ligitimacy and positivly effect deliveries. may even mitigate against some errors in the MUST portion, such as generic looking Fully Qualified Reverse DNS Entry

items listed under SHOULD sub-catogorised as major or minor;
major: little maintenance overhead, major return, many people will notice/check
minor: little maintenance overhead, minor return, few people will notice/check

[must] Establish an Internet Organisational Identity

This means a domain name of your own, or use of a subdomain of another*

If this part isn't obvious you might not be in a position to continue reading
(*another can be friendly or parent organisation)

[must] Establish a Consistent Traceable Reputable Address

This means Static IP{s}

first and foremost you need a static ip for both incomming and outgoing smtp, for incomming the requirement is obvious, for outgoing it will become so

[must] Connect the Address with the Organisational Identity, and make it verifiably non-forged

A Non-generic looking Forward Confirmed Reverse DNS Entry for your Static IP{s}

This involves first selecting a name NB:Do NOT use _ character!!
Non-generic looking means not looking like an automatically assigned name like ip.dsl.isp-name.tld {as some others block on generic looking PTR name
if you have IP{s} dedicated to mail-only use then say name it smtp, mail, mx, mta or something-indicating-its-role .your-domain.tld
if its not dedicated to email like say a single ip used by your entire office then go with something more generic like head-office.your-domain.tld, clients.your-domain.tld, cityname.your-domain.tld etc.
next get your DNS administrator {it may be you it may be your web-host it may be your ISPs to setup this name>IP "A" record and set the IP to be your IP
then getting your ISP to setup an IP>name record for your IP {this is known as a PTR record} with the name being the same
repeat this for any/all IPs used by your MTAs (it should also be done for any other ips you use also but thats out of scope)
if you have multiple IPs on one MTA it is best to point each PTR at the one name, then point the name at ALL the IPs
{though some badly run DNS hosting providers do not allow this standard option, in this case I would suggest moving DNS hosting provider {as its not a sign of good competence with a service so central to buisness availability} or avoiding using multiple IPs on the MTA as it will cause more issues with HELO/EHLO authenticity later
You can test your ip here for many receivers this is a MUST-HAVE.

[must] Continue to use your same Organisational Identity in a verifiable way in e-mail exchanges

A helo/ehlo identity that also is resolvable via DNS to your MTA's IP{s}

Select and use an e-mail client specific role name for this identity {it can be the same as above but it is better if it is a different name within the same domain*} and set up the new name>IP "A" record for ALL the IP{s} of this MTA as above.

Most MTA software allows this identity to be directly configured, but will default to the hostname of the machine if unset, for MTA specific instructions seek out info on google ;)

[should] Ensure others know that you explicitly intend this Identity to be verifiable, So they can identify and reject forgeries from others

This means protecting your helo/ehlo Identity by any other method possible

[major] SPF record for your chosen helo/ehlo identity

this means setting up an accurate SPF record for your helo/ehlo identity name, if you have the name>IP "A" record setup as indicated above then the spf for this name simply needs to be "v=spf1 A -all", but prefferably using ip4:###.###.###.### method to directly mentioning all your IPs as it avoids others having to re-look at DNS, also if for some reason you have persisted and setup a helo/ehlo that dosen't point at all the IPs you can at least increase the IPs authorised via SPF record in this form

helo/ehlo-name.your-domain.tld. IN TXT "v=spf1 ip4:###.###.###.###[ ip4:###.###.###.###] -all" more info on spf in general

[minor] CSA record for your chosen helo/ehlo identity

this means setting up an explicit CSA record authorising this name's use in helo/ehlo and verifying the IPs it is allowed from, CSA was a brilliant and apparently under implemented and abandoned/hardly verified protocol for avoiding identity forgery, and will likely not actually effect your deliverability, but as it costs nothing other than effort its worth adding, for a full guide see here but simply put add a srv record like below for each mta

_client._smtp.helo/ehlo-name.your-domain.tld. IN SRV 1 2 1 helo/ehlo-name.your-domain.tld.

[minor] RP record for your chosen helo/ehlo identity

this means setting up an accurate RP record for your chosen helo/ehlo identity name, in this day and age the e-mail portion can be '.' {.==null} the second part being a pointer to {name of} a txt record offering name phone and/or physical address and/or URL of same. example

helo/ehlo-name.your-domain.tld. IN RP . _mtacontact.your-domain.tld. _mtacontact.your-domain.tld. IN TXT "John Smith (email administrator), phone:+35387xxxxxxx, email:postmaster, web:http://www.your-domain.tld/staff/john-smith/"

[should] {Indirectly related} explicity deny the use of other identities you control to be forged in e-mail if not intended to be used in e-mail

This means protecting your non SMTP Identities by any other method possible

[major] Null MX Entries

for any hosts receiving an "v=spf1 -all" entry you can achive similar effect {regarding forgeries using this name as right-rand side of @ in e-mail} for older servers that do-not read SPF records by adding an MX 0 . entry to them, as according to RFC's if a host has an MX it is to be used {in this case no mail} as if no mx is present they say delivery should be attempted to the IP thus the reciever has to assume the address MAY exist without this null MX record telling them directly it does not

other-name.your-domain.tld. IN MX 0 .

[major] SPF record for your envelope domains

this means setting up a spf record autorising mail from user @ your-domain to come from ALL IPs possibly allowed to be seen by recievers, and denying forgeries from all others, it can be simple it can be complex (unlike the other SPF scenarios in this document) some options elaborated here.

your-domain.tld. IN TXT "v=spf1 ip4:###.###.###.### ip4:###.###.###.### ip4:###.###.###.### -all"

[major] SPF record for your other DNS records

this means setting up a null SPF record for your other non-mail identity names, ie do not accept as a helo/ehlo or right-rand side of @ in e-mail

other-name.your-domain.tld. IN TXT "v=spf1 -all"

[minor] CSA policy record for the rest of your domain

once all your mta's have been authorised via CSA you can then set a global CSA policy which basically says allow no helo/ehlo from any non-explicitly authorised hosts in my domain.

_client._smtp.your-domain.tld. SRV 1 1 1 your-domain.tld.

[minor] RP record for the rest of your domain

this means setting up an accurate RP record(s) for your top level domain name, in this day and age the e-mail portion can be '.' {.==null} the second part being a pointer to {name of} a txt record offering name phone and/or physical address and/or URL of same. this is especially usefull if your provider dosn't make WHOIS information public

your-domain.tld. IN RP . _mtacontact.your-domain.tld. your-domain.tld. IN RP . _admincontact.your-domain.tld. your-domain.tld. IN RP . _abusecontact.your-domain.tld. _mtacontact.your-domain.tld. IN TXT "John Smith (email administrator), phone:+35387xxxxxxx, email:postmaster, web:http://www.your-domain.tld/staff/john-smith/" _admincontact.your-domain.tld. IN TXT "Tom Smith (director), phone:+35387xxxxxxx, web:http://www.your-domain.tld/staff/tom-smith/" _abusecontact.your-domain.tld. IN TXT "support dept (abuse contact), phone:+35301xxxxxxx, web:http://www.your-domain.tld/contact-abuse/"

[must] be seen to value and protect your MTA's reputation

This means denying unauthorised use, and reacting/responding to abuse/complaints this is technically unrelated to being verifiable-traceable, but what point is there being verifiable if your reputation has a negative effect on trust

Unauthorised use is a wide category and will be expanded on in another document but some things to consider doing, not all of these possible with all MTA software

Do not abuse others as part of filtering or any other system, I refer to badly setup autoresponders, accept-then-bounce-spam-filters, Challenge-Response-spam-filtering, sender-callout-verification and any other backscatter causes, I do not state that these are not possible to run in a limited way, but the abuse potential must be weighed and offset as much as humanly possible, tactics like only verify/challenge mail recieved via other verifiable reputable sources(if verifiably non-forged by SPF/DKIM for example), let the untrusted source mail sit in quarentine for manual whitelisting
control who can relay mail {incomming port 25} via you, ie mail accepted by you for onward delivery should only be accepted from external-untrusted IP space if going to a user you are an incoming MX for, and even then you should consider rejecting attempts from other MTAs with a known bad reputation {discovered via DNSBL or other information sources}
control who can relay mail {ouitgoing port 25: discouraged} via youir server, remember machines on your own IP space that may need to be able to send out mail via your MTA are not necessarily clean and may be under the control of a malicious entity {1/3 of all PCs are part of a bot-net} thus if feasable only allow unauthenticated relay from selected IPs within your own IP space if no other method supported, eg certain IPs allowed to relay because the sending software dosn't support smtp auth or submission {port 587} {like certain black-box alarm systems and automated programs{additionally try and filter these to only certain envelope-senders/recipients}
control who can submit mail {outgoing port 587+auth}, the point about bots still applies to all other authenticated outgoing relay, whether limited to your own IP space or from those possesing a valid username and password located anywhere {depending on your policy}, ie. if the machine is compromised the ID/password is too.
Thus try limit its effectiveness where feasible by limiting each autenticated ID to only logon from a limited set of networks {where possible to pre-determine}
More importantly to only sending mail with an associated list of envelope-senders
consider the attempted use of non-associated envelope-senders, or the attempted login with the correct ID/Password from a network not on that ID's list, as an instant sign of a compromised ID/password and notify the user, often by diasabling smtp-auth while still allowing pop/imap and mailing the user to ask them to contact support to get their password changed
Additionally it ensures that when compromised users ID used with the associated envelope-sender {as otherwise the submission is rejected} when used {successfully} to send spam, will result in the user being alerted to the compromise when the bounces/complaints start happening.
use ratelimiting to automatically slow outgoing suspicious traffic
use content filters +quarantining on outgoing e-mail from your users {as most clients cannot handle rejection}, so quarantined mail can be released after manual inspection
monitor rejections/bounces for signs of a user's identity being used for spamming
react quickly to disconnect/disable or otherwise stop both unauthorised use that damages reputation, and authorised users that wish to abuse your MTA in ways also damaging, remind marketing that using that stolen/purchased list of address for cold-call e-mails is illegal/abuse, (you cannot buy consent from a 3rd party)
ensure some system accepts mail addressed for (postmaster & abuse) @your-helo/ehlo-identity, either your MTA if it has just the A record, or another by adding MX records directing the mail to another MTA
ensure you receive and monitor abuse@your-helo/ehlo-identity and postmaster@your-helo/ehlo-identity even if all envelope-senders using your MTA may be unrelated to your organisation or monitored at some other office, some will still expect to be able to complain to you directly
ensure you monitor abuse@any-domain-you-are-primary-mx-for and postmaster@any-domain-you-are-primary-mx-for even if this is delivered to your support dept or some other distribution list ENSURE you are on that list if you accept and terminate mail for that domain
ensure these practices are upheld at any organisation your MTA may be providing service for ie. if you run backupmx.company.com, try and ensure the admin of primarymx.company.com is not ruining your MTA's reputation by operating badly. The same applies for downstream customers of your MTA where you provide outgoing-relay service for smaller MTAs
Finally if you are confident your house is run cleanly apply to organisations such as DNSWL.ORG for whitelisting

why the FcRDNS PTR identity should not equate to the Verifiable HELO identity, but should be within the same domain, arguments for adoption of this newer but vastly more secure and verifiable convention

* The reaason for using a different name in the HELO identity, to that found in the PTR record is initially non-obvious.
It is because, the reciever CAN already easily automatically verify that the IP's PTR identity is within the same organisation {domain} as the HELO used by the sending client.
{thus is just as veifiably trustable as using the identical name from a traceability point of view, as the "Forward Confirmed" PTR has tied the ip to the HELO domain, thus a HELO within the same domain is obviously non-forged}.

BUT when they are identical, a malicious {think infected pc} client can easily appear to be a valid MTA by just looking up its own PTR and forging this as its HELO name. {as this is the default action of most bots}

Thus having them within the same domain but different is much more secure as bots cannot easilly programatically guess/find the identity name allowed in helo from that ip, yet recieving servers can easilly tell your legit MTAs from non-legit senders using the same IPs.

Obviously this will only stop widespread forgeries from IP addresses that DO NOT Legitimatly HELO with their PTR identities, and have their PTR identities protected via SPF and CSA records instructing others that they should never be accepted in HELO identity ever from any IP

Less obviously As these 'newer standard' practices become the norm for legitimate senders it is easier for users to filter traffic from clients with suspicious markers such as no FcRDNS 'old standard' or when these exists, from clients sending from an identical {and thus easily found/forged} HELO name 'newer standard'

Last updated Mar. 2024 Alan Doherty

Please feel free to donate Pizza/Beer/Candy/IoT gadget money to me if any of the free software or advice given was worthwhile to you.