Alan Doherty Level Double-A conformance icon, W3C-WAI Web Content Accessibility Guidelines 1.0
Valid CSS! Valid HTML 4.01 Strict

Alan Doherty's things every ESP should already know about SPF (and sender-id) but most don't

or How To stop giving your users BAD advice on SPF

I'm a Network/IT Consultant based in Dublin and a regular contributor on the SPF help mailing-list.

I have seen a lot of bad SPF records in my time, and spf records seem to consistently be poorly written/understood

This is further frustrated by many ESP's both giving ludicrously bad advice to customers, and even insisting they break their correct SPF records due to bad/incorrect understanding of how it works.

Obviously if unsure after reading, please get in touch via the mailing-list or contact me direct rather than guessing, I do this to improve the 'ecosystem' as more (easily recognizable) good senders makes bad ones easier to reject

The issues apparently spread from confusion between SPF and Sender-ID, ESP's choosing to advise customers to write a single record that attempts to allow their mail to pass both (even though they are entirely incompatible by design, and sender-id has been rejected as failed/flawed by all), and the resultant record is massively sub-optimal for SPF

As sender-id is built into MS-exchange and often not disabled, if not dealt with correctly by senders, it WILL cause frequent deliver ability issues, exacerbated by Microsoft's insistence on regularly referring to sender-id as SPF as if they were compatible and interchangeable (which they are not)

SPF first

SPF can/will only check/verify the connecting ip against the list contained in the SPF records of the domain used in the HELO greeting AND separately it will check/verify the connecting ip against the list contained in the SPF records of the domain used in the envelope-sender, normally before mail proceeds to DATA transmission

No other records/domains are ever looked at by SPF

A If a customer delegates a domain to you (directs it at your MX hosts)
In this case the domain is used exclusively by you and as it likely appears in envelope-sender that domains SPF record MUST contain your IPs (usually via an include or redirect)
B IF a customer does not delegate a domain to you AND you handle bounces
If you receive the Bounces/NDRs, then your domain MUST be the envelope-sender, and as SPF by design will ONLY ever look at the envelope-sender's SPF record, the customers SPF needs no alteration/additions and MUST NOT include your IPs
C IF a customer does not delegate a domain to you AND they must receive/handle bounces themselves
If the customers domain receive the Bounces/NDRs, then their domain MUST be the envelope-sender, and as SPF by design will ONLY ever look at the envelope-sender's SPF record, the customers SPF needs to contain your IPs (usually via an include)
Separate/distinguish between user/staff and customer include-able SPF records
First a customer include should use a dedicated sub-domain, one without MX records (and potential users) it should ideally be one starting _ to indicate it is not a legal host or mail domain
_spf.yourdomain is ideal, your root domain is usually for your staff/users (and normally has MX records) and should like other customers include the record of your sending IPs
example: bigprovider.com offers ESP services to customers of all forms, but also has some staff email on office365
bigprovider.com IN TXT "v=spf1 (any other staff only systems) include:spf.protection.outlook.com include:_spf.bigprovider.com -all"
bigprovider.com IN TXT "spf2.0/pra ?all"                           //more on this later
_spf.bigprovider.com IN TXT "v=spf1 ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx -all"

customers would setup

delegated.classAcustomer.com IN TXT "v=spf1 include:_spf.bigprovider.com -all"
delegated.classAcustomer.com IN TXT "spf2.0/pra ?all"

classBcustomer.com IN TXT "v=spf1 their-own-sending-ips-no-includes-of-yours -all"
classBcustomer.com IN TXT "spf2.0/pra ?all"

classCcustomer.com IN TXT "v=spf1 their-own-sending-ips include:_spf.bigprovider.com -all"
classCcustomer.com IN TXT "spf2.0/pra ?all"
If you offer an include-able SPF it should not be costly
First it should only be for customer use thus should avoid all mentions of mx: a: and ptr: methods it should be ip4: and ip6: only
it should avoid unnecessary sub-includes by being as close to but not over the 450 minus length of sub-domain interoperability maximum length
only if it exceeds this length should it be split into multiple records
Even if your servers do not generate NDRs you should always SPF verify their HELO-ids
Older SPF documents only suggested helo checking in the case of NDRs (mails from <> null sender)
Newer advice is to perform SPF verification of helo-id's seen on all inbound connections
If servername.bigprovider.com is used in helos from ip xxx.xxx.xxx.xxx then as well as all the normal DNS records it should have an SPF
servername.bigprovider.com IN TXT "v=spf1 ip4:xxx.xxx.xxx.xxx -all"
servername.bigprovider.com IN A xxx.xxx.xxx.xxx

other general spf cleanup advice given in preceding section

Sender-id (the infected thorn in SPF's side)

All ESP's investigated thus far, entirely fail to deal with sender-id correctly or appropriately, even though it is the only possible origin for some of their bad advice given to customers on SPF

systems (millions of MS-exchange users who haven't disabled the broken 'feature') still attempting this failed/broken and retired experimental protocol, if they do not find a sender-id record, they will take any SPF record found (this is where the protocol is broken) and use it as-if it were a sender-id record

sender-id will check/verify the connecting ip against the list contained in the Sender-ID records of the domain used in the envelope-sender (fine) AND will also check/verify the connecting ip against the list contained in the Sender-ID records of the domain used in the PRA header (Resent-From: Resent-Sender: Sender: From:), if a sender-id record is not found it will look for any spf record within the same domain and use its list of ips as a sender-id record, this is fine in the envelope-sender case, but disastrous in the PRA header case, these checks also happen after message transmission complete (as headers have to be transmitted to be seen)

sender-id's use should be discouraged strongly as it has been retired/failed and is no longer an active protocol, thus doing more than limiting its damage is discouraged

It failed because too much legitimate email will have from: header via an unexpected and innumerable source-ips (all mailing-lists, all SRS-compliant forwarders, most ESP's)

for this reason class B customers above when they are made to erroneously add the ESP's SPF to their own cease seeing failures from sender-id using receivers, not because the senders SPF has been 'fixed' but because they are unnecessarily lengthening the SPF (often to breaking point) rather than fixing the sender-id

If a customer has an SPF record
you MUST advise them to mount a sender-id record to block the abuse of their correct SPF record
as long as ms-exchange is in use all users of SPF need to also have this Nullifying sender-id record
the recommend sender-id record is "spf2.0/pra ?all"
this prevents the sender-id software investigating the from: header from looking further (and finding abusing their SPF records list of IPs and failing to find your ip there) it causes a neutral response to all checks of from: for any ip, it allows envelope-sender checks to proceed (as this will yield the same result as SPF, PASS)
you can provide an optional include able sender-id record, for customers who wish to setup and use both systems
as this only extends the usefulness of the troublesome sender-id protocol it should NOT be recommended, if done it should be optional for customers to use NEVER required
But you could setup _pra.yourdomain IN TXT "spf2.0/pra ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx ?all" and instruct your customers to setup "spf2.0/pra (their ips) include:_pra.yourdomain ?all"
this causes sender-id software investigating the from: header to positive pass mail from your IPs, and neutral (non-fail) all other IPs (as PRA failures should never be fatal as they are common in normal non-forged mail flow)
example using previous bigprovider.com offering a pra include
bigprovider.com IN TXT "v=spf1 (any other staff only systems) include:spf.protection.outlook.com include:_spf.bigprovider.com -all"
bigprovider.com IN TXT "spf2.0/pra include:_pra.bigprovider.com ?all"
_spf.bigprovider.com IN TXT "v=spf1 ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx -all"
_pra.bigprovider.com IN TXT "spf2.0/pra ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx ip4:xxx.xxx.xxx.xxx/xx ?all"

customers would setup

delegated.classAcustomer.com IN TXT "v=spf1 include:_spf.bigprovider.com -all"
delegated.classAcustomer.com IN TXT "spf2.0/pra include:_pra.bigprovider.com ?all"

classBcustomer.com IN TXT "v=spf1 their-own-sending-ips-no-includes-of-yours -all"
classBcustomer.com IN TXT "spf2.0/pra their-own-sending-ips include:_pra.bigprovider.com ?all"

classCcustomer.com IN TXT "v=spf1 their-own-sending-ips include:_spf.bigprovider.com -all"
classCcustomer.com IN TXT "spf2.0/pra their-own-sending-ips include:_pra.bigprovider.com ?all"

all sender-id policies for PRA really must fail neutral in all but the most limited of scenarios, SPF on the other hand should only fail neutral during testing and them fail ~(soft fail) or -(hard fail) [often interpreted as ~(quarantine/spam-folder/tag) -(reject) but every recipient will treat there own way]

all users of SPF should have a sender-id setup because otherwise stuff breaks

the minimum "spf2.0/pra ?all" suffices to stop inappropriate misapplication of the otherwise correct SPF

as ESP's and others should be giving good advice to customers, they MUST start advising them to do same as ignoring the issue with Sender-ID's broken design is only exacerbating the issues for users who depend on good accurate advice, and SPF advocates have apparently just hoping it will go away (but as its built in to all ms-exchange products, and mis-documented as compatible, it will not anytime soon)

Last updated Sept. 2024 Alan Doherty

Please feel free to donate Pizza/Beer/Candy/IoT gadget money to me if any of the free software or advice given was worthwhile to you.

Join the Blue Ribbon Online Free Speech Campaign Stop Spam Harvesters, Join Project Honey Pot