A Brief Look at EMail, SPF, DKIM and DMARC

Having recently built my home email server and wanting to be a good MTA I decided to look a number of anti-spam mechanisms. Whilst host-based anti-virus solutions and the like offer anti-spam engines, there also exist a number of other technologies to help MTAs determine if email is legit and which can have unintended consequences. There is also a brief look at understanding the internals of an org without access it (recon).

Here I'll briefly cover four of them: SPF, DKIM and DMARC. The fourth is a a common practice that if you see email originating from you (as defined by the From: address) which did not originate from your server, you just discard the email. A good number of organisations do this.

Finally, we will have a look at a couple of examples.

The first three listed aren't always enforced. However, that doesn't infer your email is going to get through. Those anti-spam engines can and do use the results to rate the email, so a misconfigured setup can result in mail been rated as spam or discarded, not by spf/dkim/dmarc directly, but by the anti-spam mechanism.

Further, as you will see, to have a good all-round solution, you will need DNSSec fully implemented.

There is a wealth of information on this subject, including dmarc.org and Google itself. Just search for it.

SPF

At a basic level, Sender Policy Framework (SPF) works by a domain publishing a DNS TXT record in it's domain to describe who sends mail on the behalf of the domain. Typically this ends with an all action which describes how the receiving MTA should handle unmatched senders. For example:

m0noc.net   text = "v=spf1 +mx -all"

silentcircle.com  text = "v=spf1 include:spf.protection.outlook.com -all"

google.com  text = "v=spf1 include:_spf.google.com ~all"

yahoo.com   text = "v=spf1 ptr:yahoo.com ptr:yahoo.net ?all" # NB: the main rec is a redirect

A minus preceding the all infers that you should perform a a 'hard fail', a tilde infers a 'soft fail' and a question mark infers treat it as if the SPF record didn't exist (i.e. is not a positive validation; it is 'neutral').

Oftentimes, the receiving MTA does not discard a mail, even if it is a hard fail. However, an anti-spam solution may make a determination or adversely rank it. It is up to the policy that the receiver chooses to implement as to how to proceed. My server rejects hard fails but currently allows soft fails (which may get rated as spam by either my mail server or mail client).

As a result of this, and organisations rejecting email that states it came from them, but from an unknown host, mailing lists will often rewrite the From: address so you see the on behalf of statements in your email.

DKIM

At a basic level DKIM works by adding a mail header to outgoing messages. This is an assertion to protect certain header fields from being altered and to provide an assertion that it was signed by an authorized host; for example protecting the From: and Subject: fields.

The linkage to the authorized host is via a selector. This is a hook to a DNS TXT record. For example, a yahoo.com originated email may have the selector s2048, so we prepend this to domain and the subdomain _domainkey to get published bit including the public key. The following is truncated for brevity.

s2048._domainkey.yahoo.com    text = "k=rsa\; p=MIIBIjANBgkqhkiG.....”

With DKIM there is a lot of debate about what mailing lists and forwarders do with this. As previously mentioned, due to a number of reasons the From: field will get re-written. However, this will then break the DKIM signature. So, do you keep it and risk a receiving MTA or end-user anti-spam solution treating it as SPAM, or not.

DMARC

Domain-based Message Authentication, Reporting & Conformance (DMARC) sits on top of SPF and DKIM. It allows a domain to publish a policy on how to handle messages that fail SPF and DKIM tests, and also publish a reporting function.

Like SPF and DKIM the policy is published as a DNS TXT record. For example:

_dmarc.m0noc.net  text = "v=DMARC1\; p=reject\; pct=100\; rua=mailto:dmarc-feedback@m0noc.net\; aspf=r\; adkim=r\;"

_dmarc.gmail.com  text = "v=DMARC1\; p=none\; rua=mailto:mailauth-reports@google.com"

_dmarc.google.com text = "v=DMARC1\; p=reject\; rua=mailto:mailauth-reports@google.com"

_dmarc.yahoo.com  text = "v=DMARC1\; p=reject\; pct=100\; rua=mailto:dmarc_y_rua@yahoo.com\;"

The p= part is the policy: none infers to just report on it, quarantine infers mark failed messages as spam, and reject infers reject the message at the SMTP layer.

Again, it is up to the receiving MTA to determine if they honour the policy or not.

Example: Sending to (some) BT.com addresses

A recent email I sent to someone at bt.com had a delivery receipt set. The receipt will contain an attachment which is the header of the original message that was sent, and revealed a few interesting things about the path it (most likely) took. It also reveal a bit of information about how the network is set up and what's running on them. Useful information you can publicly get hold of; remember that the organisation voluntarily gave this information as part of a standard email.

Here is an edited version containing the Received: headers.

Received: from HE1PR06CA0108.eurprd06.prod.outlook.com (10.169.114.34) by

 DB5PR06MB1495.eurprd06.prod.outlook.com (10.164.40.141) with Microsoft SMTP

 Server id 15.1.557.21

Received: from VE1EUR02FT011.eop-EUR02.prod.protection.outlook.com

 (2a01:111:f400:7e06::204) by HE1PR06CA0108.outlook.office365.com

 (2a01:111:e400:7a1b::34) with Microsoft SMTP Server id 15.1.587.13 via Frontend Transport

Authentication-Results: spf=fail (sender IP is 62.239.224.235)

 smtp.mailfrom=m0noc.net; btgroupcloud.mail.onmicrosoft.com; dkim=fail

 (signature did not verify)

 header.d=m0noc.net;btgroupcloud.mail.onmicrosoft.com; dmarc=fail

 action=oreject header.from=m0noc.net;bt.com; dkim=fail (signature did not

 verify) header.d=m0noc.net;

Received-SPF: Fail (protection.outlook.com: domain of m0noc.net does not

 designate 62.239.224.235 as permitted sender)

 receiver=protection.outlook.com; client-ip=62.239.224.235;

 helo=RDW083A006ED62.bt.com;

Received: from RDW083A006ED62.bt.com (62.239.224.235) by

 VE1EUR02FT011.mail.protection.outlook.com (10.152.12.150) with Microsoft SMTP

 Server id 15.1.587.6 via Frontend Transport

Received: from EVMHT66-UKRD.domain1.systemhost.net (10.36.3.103) by

 RDW083A006ED62.bt.com (10.187.98.11) with Microsoft SMTP Server id

 14.3.181.6

Received: from smtpe1.intersmtp.com (10.187.98.11) by

 EVMHT66-UKRD.domain1.systemhost.net (10.36.3.103) with Microsoft SMTP Server id 8.3.342.0

Received: from tbhext01.m0noc.net (82.70.154.45) by smtpe1.intersmtp.com

 (62.239.224.235) with Microsoft SMTP Server id 14.3.181.6

We have to read these bottom up to track the flow.

First we note from http://www.oxfordsbsguy.com/2014/01/27/exchange-server-and-update-rollups-build-numbers-2/ (or other sources) that a Microsoft SMTP server id of 15.1.x.x is probably Exchange Server 2016; it seems that Microsoft are running a really new version of Microsoft Exchange ;-)

We also note that 14.3.181.6 is Update Rollup 5 for Exchange Server 2010 SP3 and 8.3.342.0 is probably Update Rollup 12 for Exchange Server 2007 Service Pack 3.

Now for the interconnect.

82.70.154.45 is my mail server; the origin of the email. This connected to smtpe1.intersmtp.com (ip address 62.239.224.235).

The next line up then states the message was received from smtpe1.intersmtp.com (10.187.98.11) by a different host. This tells us that the public interface for smtpe1.intersmtp.com has an IP address of 62.239.224.235 and an internal interface of 10.187.98.11 (which is also an RFC1918 address; not valid on the Internet; hence internal – or there is a bit of NAT). We can repeat this up the chain to reveal the likely path.

We can also see that smtpe1.intersmtp.com is also known internally as RDW083A006ED62.bt.com (the reverse path with the same IP's and the receiving path).

Finally, the outgoing path goes from a public address to an rfc1918 address. This suggests some DMZ with a B2B link to Microsoft (there are other possibilities).

In some cases, viewing whois records can also offer some intelligence.

I will leave it as an exercise for the reader to produce a diagram of the interconnect.

What is of particular interest is that the email goes into BT then back out again to the Microsoft cloud.

This is where the problem occurs. Earlier on, and further down in the header we see this:

Received-SPF: Pass (RDW083A006ED62.bt.com: domain of REDACTED@m0noc.net

 designates 82.70.154.45 as permitted sender) receiver=RDW083A006ED62.bt.com;

 client-ip=82.70.154.45; helo=tbhext01.m0noc.net

It shows that when it was on its initial incoming path the BT mail servers correctly validated the SPF rule. However, as it went on its way out to the cloud this test was re-done (see the earlier trace, key part of SPF highlighted in red), along with the embedded DKIM signature and the DMARC policy for my domain. Accordingly, it all failed as it was basing its results on one of the internal BT relays being authorized to generate mail for my domain and had already munged some of the other fields which broke DKIM as well, so DMARC then also failed.

Epic fail.

Raynet

So what could be worse? Perhaps ensuring that any sender with an SPF fail policy is marked as spam just because you evaluated the SPF policy against your Internet facing email relay server?

This is what the Raynet mail forwarding service appears to be doing; lets see if you concur...

The reason I was looking at this one is that none of my emails were getting through when sent to this account when it was set up to forward to my ISP.

First, the original failure was because the forwarder forwarded the email as-is, without changing the From: field. Accordingly my ISP was discarding the email since the Raynet server isn't one of my ISP's email servers, and the mail claims it originated from my ISP.

How do I know? Well I changed the forwarding to my m0noc.net address and did the same thing. A mail sent came back (in the logs at least); with two SPF fails. On marking it as SPAM by the second Raynet server, and then me as the From: field asserted it was from m0noc.net, but my SPF rules stated 'computer says no'/hard fail.

The key headers (cut for brevity) are:

Received: from mail.raynet-uk.net (www.raynet-uk.net [88.208.249.159])

Received: from amg1.gbse.gb.net (amg1.gbse.gb.net [91.234.185.121]) by mail.raynet-uk.net

Received: from tbhext01.m0noc.net ([82.70.154.45]:33182) by amg1.gbse.gb.net

  with esmtp (Exim 4.82_1-5b7a7c0-XX)

Subject: [SPAM] test

X-hMailServer-Spam: YES

X-hMailServer-Reason-1: Blocked by SPF () - (Score: 3)

X-hMailServer-Reason-Score: 3

As we can see, there are X-hMailServer headers that state that this email is spam due to an SPF failure.

Now, given that I'm not running hMailServer (a freeware mail server) and Raynet's first mail server is running Exim 4.82, we can reasonably assume that 91.234.185.121 is relaying it to 88.208.249.159, and it is this second server that is running hMailServer (we can research that online).

Secondly, as I know my SPF records are valid (others such as Google, Yahoo, etc all confirm it is OK), we can reasonably assume that the second Raynet server is performing the SPF check against the first Raynet server; which will of cause always fail where there is an SPF record for the sending domain.

Whois is also interesting with this one.

I'm sure you have seen many other examples of anomalies that are misconfiguration or logic issues rather than a suspect email.