A Brief Look at EMail, SPF, DKIM and DMARC

Having recently built my home email server and wanting to be a good MTA I decided to look a number of anti-spam mechanisms. Whilst host-based anti-virus solutions and the like offer anti-spam engines, there also exist a number of other technologies to help MTAs determine if email is legit and which can have unintended consequences. There is also a brief look at understanding the internals of an org without access it (recon).

Here I'll briefly cover four of them: SPF, DKIM and DMARC. The fourth is a a common practice that if you see email originating from you (as defined by the From: address) which did not originate from your server, you just discard the email. A good number of organisations do this.

Finally, we will have a look at a couple of examples.

The first three listed aren't always enforced. However, that doesn't infer your email is going to get through. Those anti-spam engines can and do use the results to rate the email, so a misconfigured setup can result in mail been rated as spam or discarded, not by spf/dkim/dmarc directly, but by the anti-spam mechanism.

Further, as you will see, to have a good all-round solution, you will need DNSSec fully implemented.

There is a wealth of information on this subject, including dmarc.org and Google itself. Just search for it.

SPF

At a basic level, Sender Policy Framework (SPF) works by a domain publishing a DNS TXT record in it's domain to describe who sends mail on the behalf of the domain. Typically this ends with an all action which describes how the receiving MTA should handle unmatched senders. For example:

m0noc.net   text = "v=spf1 +mx -all"

silentcircle.com  text = "v=spf1 include:spf.protection.outlook.com -all"

google.com  text = "v=spf1 include:_spf.google.com ~all"

yahoo.com   text = "v=spf1 ptr:yahoo.com ptr:yahoo.net ?all" # NB: the main rec is a redirect

A minus preceding the all infers that you should perform a a 'hard fail', a tilde infers a 'soft fail' and a question mark infers treat it as if the SPF record didn't exist (i.e. is not a positive validation; it is 'neutral').

Oftentimes, the receiving MTA does not discard a mail, even if it is a hard fail. However, an anti-spam solution may make a determination or adversely rank it. It is up to the policy that the receiver chooses to implement as to how to proceed. My server rejects hard fails but currently allows soft fails (which may get rated as spam by either my mail server or mail client).

As a result of this, and organisations rejecting email that states it came from them, but from an unknown host, mailing lists will often rewrite the From: address so you see the on behalf of statements in your email.

DKIM

At a basic level DKIM works by adding a mail header to outgoing messages. This is an assertion to protect certain header fields from being altered and to provide an assertion that it was signed by an authorized host; for example protecting the From: and Subject: fields.

The linkage to the authorized host is via a selector. This is a hook to a DNS TXT record. For example, a yahoo.com originated email may have the selector s2048, so we prepend this to domain and the subdomain _domainkey to get published bit including the public key. The following is truncated for brevity.

s2048._domainkey.yahoo.com    text = "k=rsa\; p=MIIBIjANBgkqhkiG.....”

With DKIM there is a lot of debate about what mailing lists and forwarders do with this. As previously mentioned, due to a number of reasons the From: field will get re-written. However, this will then break the DKIM signature. So, do you keep it and risk a receiving MTA or end-user anti-spam solution treating it as SPAM, or not.

DMARC

Domain-based Message Authentication, Reporting & Conformance (DMARC) sits on top of SPF and DKIM. It allows a domain to publish a policy on how to handle messages that fail SPF and DKIM tests, and also publish a reporting function.

Like SPF and DKIM the policy is published as a DNS TXT record. For example:

_dmarc.m0noc.net  text = "v=DMARC1\; p=reject\; pct=100\; rua=mailto:dmarc-feedback@m0noc.net\; aspf=r\; adkim=r\;"

_dmarc.gmail.com  text = "v=DMARC1\; p=none\; rua=mailto:mailauth-reports@google.com"

_dmarc.google.com text = "v=DMARC1\; p=reject\; rua=mailto:mailauth-reports@google.com"

_dmarc.yahoo.com  text = "v=DMARC1\; p=reject\; pct=100\; rua=mailto:dmarc_y_rua@yahoo.com\;"

The p= part is the policy: none infers to just report on it, quarantine infers mark failed messages as spam, and reject infers reject the message at the SMTP layer.

Again, it is up to the receiving MTA to determine if they honour the policy or not.

Example: Sending to (some) BT.com addresses

A recent email I sent to someone at bt.com had a delivery receipt set. The receipt will contain an attachment which is the header of the original message that was sent, and revealed a few interesting things about the path it (most likely) took. It also reveal a bit of information about how the network is set up and what's running on them. Useful information you can publicly get hold of; remember that the organisation voluntarily gave this information as part of a standard email.

Here is an edited version containing the Received: headers.

Received: from HE1PR06CA0108.eurprd06.prod.outlook.com (10.169.114.34) by

 DB5PR06MB1495.eurprd06.prod.outlook.com (10.164.40.141) with Microsoft SMTP

 Server id 15.1.557.21

Received: from VE1EUR02FT011.eop-EUR02.prod.protection.outlook.com

 (2a01:111:f400:7e06::204) by HE1PR06CA0108.outlook.office365.com

 (2a01:111:e400:7a1b::34) with Microsoft SMTP Server id 15.1.587.13 via Frontend Transport

Authentication-Results: spf=fail (sender IP is 62.239.224.235)

 smtp.mailfrom=m0noc.net; btgroupcloud.mail.onmicrosoft.com; dkim=fail

 (signature did not verify)

 header.d=m0noc.net;btgroupcloud.mail.onmicrosoft.com; dmarc=fail

 action=oreject header.from=m0noc.net;bt.com; dkim=fail (signature did not

 verify) header.d=m0noc.net;

Received-SPF: Fail (protection.outlook.com: domain of m0noc.net does not

 designate 62.239.224.235 as permitted sender)

 receiver=protection.outlook.com; client-ip=62.239.224.235;

 helo=RDW083A006ED62.bt.com;

Received: from RDW083A006ED62.bt.com (62.239.224.235) by

 VE1EUR02FT011.mail.protection.outlook.com (10.152.12.150) with Microsoft SMTP

 Server id 15.1.587.6 via Frontend Transport

Received: from EVMHT66-UKRD.domain1.systemhost.net (10.36.3.103) by

 RDW083A006ED62.bt.com (10.187.98.11) with Microsoft SMTP Server id

 14.3.181.6

Received: from smtpe1.intersmtp.com (10.187.98.11) by

 EVMHT66-UKRD.domain1.systemhost.net (10.36.3.103) with Microsoft SMTP Server id 8.3.342.0

Received: from tbhext01.m0noc.net (82.70.154.45) by smtpe1.intersmtp.com

 (62.239.224.235) with Microsoft SMTP Server id 14.3.181.6

We have to read these bottom up to track the flow.

First we note from http://www.oxfordsbsguy.com/2014/01/27/exchange-server-and-update-rollups-build-numbers-2/ (or other sources) that a Microsoft SMTP server id of 15.1.x.x is probably Exchange Server 2016; it seems that Microsoft are running a really new version of Microsoft Exchange ;-)

We also note that 14.3.181.6 is Update Rollup 5 for Exchange Server 2010 SP3 and 8.3.342.0 is probably Update Rollup 12 for Exchange Server 2007 Service Pack 3.

Now for the interconnect.

82.70.154.45 is my mail server; the origin of the email. This connected to smtpe1.intersmtp.com (ip address 62.239.224.235).

The next line up then states the message was received from smtpe1.intersmtp.com (10.187.98.11) by a different host. This tells us that the public interface for smtpe1.intersmtp.com has an IP address of 62.239.224.235 and an internal interface of 10.187.98.11 (which is also an RFC1918 address; not valid on the Internet; hence internal – or there is a bit of NAT). We can repeat this up the chain to reveal the likely path.

We can also see that smtpe1.intersmtp.com is also known internally as RDW083A006ED62.bt.com (the reverse path with the same IP's and the receiving path).

Finally, the outgoing path goes from a public address to an rfc1918 address. This suggests some DMZ with a B2B link to Microsoft (there are other possibilities).

In some cases, viewing whois records can also offer some intelligence.

I will leave it as an exercise for the reader to produce a diagram of the interconnect.

What is of particular interest is that the email goes into BT then back out again to the Microsoft cloud.

This is where the problem occurs. Earlier on, and further down in the header we see this:

Received-SPF: Pass (RDW083A006ED62.bt.com: domain of REDACTED@m0noc.net

 designates 82.70.154.45 as permitted sender) receiver=RDW083A006ED62.bt.com;

 client-ip=82.70.154.45; helo=tbhext01.m0noc.net

It shows that when it was on its initial incoming path the BT mail servers correctly validated the SPF rule. However, as it went on its way out to the cloud this test was re-done (see the earlier trace, key part of SPF highlighted in red), along with the embedded DKIM signature and the DMARC policy for my domain. Accordingly, it all failed as it was basing its results on one of the internal BT relays being authorized to generate mail for my domain and had already munged some of the other fields which broke DKIM as well, so DMARC then also failed.

Epic fail.

Raynet

So what could be worse? Perhaps ensuring that any sender with an SPF fail policy is marked as spam just because you evaluated the SPF policy against your Internet facing email relay server?

This is what the Raynet mail forwarding service appears to be doing; lets see if you concur...

The reason I was looking at this one is that none of my emails were getting through when sent to this account when it was set up to forward to my ISP.

First, the original failure was because the forwarder forwarded the email as-is, without changing the From: field. Accordingly my ISP was discarding the email since the Raynet server isn't one of my ISP's email servers, and the mail claims it originated from my ISP.

How do I know? Well I changed the forwarding to my m0noc.net address and did the same thing. A mail sent came back (in the logs at least); with two SPF fails. On marking it as SPAM by the second Raynet server, and then me as the From: field asserted it was from m0noc.net, but my SPF rules stated 'computer says no'/hard fail.

The key headers (cut for brevity) are:

Received: from mail.raynet-uk.net (www.raynet-uk.net [88.208.249.159])

Received: from amg1.gbse.gb.net (amg1.gbse.gb.net [91.234.185.121]) by mail.raynet-uk.net

Received: from tbhext01.m0noc.net ([82.70.154.45]:33182) by amg1.gbse.gb.net

  with esmtp (Exim 4.82_1-5b7a7c0-XX)

Subject: [SPAM] test

X-hMailServer-Spam: YES

X-hMailServer-Reason-1: Blocked by SPF () - (Score: 3)

X-hMailServer-Reason-Score: 3

As we can see, there are X-hMailServer headers that state that this email is spam due to an SPF failure.

Now, given that I'm not running hMailServer (a freeware mail server) and Raynet's first mail server is running Exim 4.82, we can reasonably assume that 91.234.185.121 is relaying it to 88.208.249.159, and it is this second server that is running hMailServer (we can research that online).

Secondly, as I know my SPF records are valid (others such as Google, Yahoo, etc all confirm it is OK), we can reasonably assume that the second Raynet server is performing the SPF check against the first Raynet server; which will of cause always fail where there is an SPF record for the sending domain.

Whois is also interesting with this one.

I'm sure you have seen many other examples of anomalies that are misconfiguration or logic issues rather than a suspect email.

Kernel Tracing Qmax on Solaris – Part 2

Following on from Part 1, we will further enhance our script to trace incoming connections through the kernel to the application.

Before doing that we will tweak the script to provide a CSV output. We do this using a BEGIN probe to print the header and then update the probes. We will also split the probes. Where there is a match, multiple probes are triggered in the order listed in the script. We can therefore say we are only interested in a specific port and then do the heavy lifting later on.

It is also important that you clear variables once you finish with them so as not to fill up the kernel buffers (you will loose traces if you don't; given enough probes firing).

#!/usr/sbin/dtrace -Cs

#pragma D option quiet

#include <sys/types.h>

#include <sys/socket.h>

#include <netinet/ip.h>

BEGIN

{

        printf("TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax\n");

}

fbt:ip:tcp_conn_request:entry

{

        self->tcpq = *(tcp_t**)((char*)arg0+0x28);

        self->lport = ntohs(*(uint16_t*)((char*)self->tcpq->tcp_connp+0x10a));

}

fbt:ip:tcp_conn_request:entry

/self->lport == $1/

{

        this->srcAddr = 0;

        this->srcPort = 0;

        printf("%d,%Y,%s,0x%08x,%d,%d,%d,%d,%d\n",

                timestamp, walltimestamp, "syn",

                this->srcAddr, this->srcPort,

                self->lport,

                self->tcpq->tcp_conn_req_cnt_q0,

                self->tcpq->tcp_conn_req_cnt_q,

                self->tcpq->tcp_conn_req_max

        );

}

fbt:ip:tcp_conn_request:return

/self->lport/

{

        self->tcpq = 0;

        self->lport = 0;

}

Whilst in tcp_conn_request() we don't actually have a connection from the client; the kernel is about to build one.

It is the message in the call that is actually the incoming packet to process. IP has dealt with it; now it is TCP's turn. Remember that the tcpq we have used so far is the listener's connection (the one that is in the listen state), not the eager's (new connection); so looking at that is not going to be productive.

We could probe another function or we could decode the message; tcp_accept_comm() is called by tcp_conn_request() to accept the connection providing a few things are satisfied such as Qmax not been reached. If we are only interested in connections that aren't dropped due to, e.g. Q==Qmax, then this would be a good place. However, if we also want to capture failed connections, we need to do the processing ourselves.

So, let's decode the message :D

Looking at the source for tcp_conn_request() we can see that arg1 (mp, the mblk_t) is the message. Not surprising as this is a STREAMS module. Further down we can see where the data is and the format of the data. i.e. we have a pointer to the IP header here.

ipha = (struct ip *)mp->b_rptr;

As mblk_t is a standard structure we can easily map this within our DTrace script. We can therefore change the setting of srcAddr to the following:

        this->mblk = (mblk_t *)arg1;

        this->iphdr = (struct ip*)this->mblk->b_rptr;

        this->srcAddr = this->iphdr->ip_src.s_addr;

If we then run the script and connect to the server running on 172.16.170.133 from client 172.16.170.1 we get the following:

root@sol10-u9-t4# ./tcpq_ex7.d 22

TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax

180548302609318,2016 Aug 11 14:37:02,syn,0x85aa10ac,0,22,0,0,8

As you can see, the hex address is that of the destination and not the source. Given that an IP header is a standard format (defined in rfc791) this is a bit surprising. So, what is going on?

A Slight Detour

The IP header is defined in /usr/include/netinet/ip.h. It is as expected. Whilst I won't go into all the steps to find the cause, here is the quick version.

The DTrace compiler uses the C pre-processor, but compiles it's own code for the kernel to use. DTrace should be able to import C headers to allow you to use names, etc, but clearly something has gone wrong.

One test I did is to create my own header using the constituent parts and a key tweak; one struct doesn't use fields (bitmasks) but both are the same size.

#ifndef MYIP_H

#define MYIP_H

#include <sys/types.h>

typedef struct {

        uchar_t         c1;

        uchar_t         c2,c3,c4;

        ushort_t        s1,s2,s3,s4;

        uint32_t        src,dst;

} myip_t;

typedef struct {

        uchar_t         c1_1:4,c1_2:4;

        uchar_t         c2,c3,c4;

        ushort_t        s1,s2,s3,s4;

        uint32_t        src,dst;

} myip2_t;

#endif

Then, if we run this through a simple C program that just prints the sizes all looks good:

#include <stdio.h>

#include "myip.h"

int main(int argc, char *argv[]) {

        printf("myip_t : %u\n", sizeof(myip_t));

        printf("myip2_t: %u\n", sizeof(myip2_t));

        return 0;

}

Which gives the following output:

root@sol10-u9-t4# ./a.out

myip_t : 20

myip2_t: 20

Then within the BEGIN probe of the DTrace we do the same (not forgetting to include the header):

        printf("myip_t : %d\n", sizeof(myip_t));

        printf("myip2_t: %d\n", sizeof(myip2_t));

Which gives the following output:

root@sol10-u9-t4# ./tcpq_ex8.d -I `pwd` 22

TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax

myip_t : 20

myip2_t: 24

This is the problem. A few more tests show that the DTrace compiler cannot handle fields (bitmasks) in the C structs correctly, at least in the situations I've tested (me'thinks a bug). I will leave it as an exercise for the reader to analyse this further.

Back to the Task at Hand

As the IP header is well-defined we can just define the offset without further analysis.

this->srcAddr = *(uint32_t*)((char*)(this->iphdr)+0xc);

This yields the correct IP address:

root@sol10-u9-t4# ./tcpq_ex9.d 22

TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax

181523558976884,2016 Aug 11 14:53:17,syn,0x01aa10ac,0,22,0,0,8

As the header length of the IP header and the TCP header has the bitmask after the port, we should be able to extract the port quite easily (setting srcPort to the result):

        this->ihl = 4 * ((int)*((char*)(this->iphdr)) & 0xf);

        this->tcphdr = (struct tcphdr*)((char*)(this->iphdr)+(this->ihl));

        this->srcPort = ntohs(this->tcphdr->th_sport);

This yields the following output:

root@sol10-u9-t4# ./tcpq_ex10.d 22

TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax

181903328660708,2016 Aug 11 14:59:37,syn,0x01aa10ac,40950,22,0,0,8

Which is confirmed via netstat:

root@sol10-u9-t4# netstat -an | fgrep 40950

172.16.170.133.22    172.16.170.1.40950   29312      0 49232      0 ESTABLISHED

Completion of the handshake

As you can guess, there are various locations that we can probe the kernel in order to mark events. In the case of the completion of the three-way handshake, I'm going to go for tcp_send_conn_ind() in usr/src/uts/common/inet/tcp/tcp_tpi.c. The key function is actual tcp_rput_data() in usr/src/stand/lib/tcp/tcp.c where the state is changed to EST.

Anyway, in tcp_send_conn_ind() arg0 is the listener and arg1 is a mblk_t containing a pointer to a struct T_conn_ind. However, as we have already processed the original syn packet, we also have an eager tcp_t. We can extract that, and use the offsets we know to get the address and port of the peer.

We can combine probes where there is common processing. In this particular kernel arg0 is the same.

fbt:ip:tcp_conn_request:entry,

fbt:ip:tcp_send_conn_ind:entry

{

        self->tcpq = *(tcp_t**)((char*)arg0+0x28);

        self->lport = htons(*(uint16_t*)((char*)self->tcpq->tcp_connp+0x10a));

}

We can now predicate all the other probes within the kernel thread to where the port maps what we are after. This includes a simple way to label the action. We can also change the variable context from 'this' (lexical scope) to 'self' (thread scope) to pass the parameters that are in different probes but the same thread.

First, tcp_conn_request(), which should be familiar:

fbt:ip:tcp_conn_request:entry

/self->lport == $1/

{

        self->action = "syn";

        this->mblk = (mblk_t *)arg1;

        this->iphdr = (struct ip*)this->mblk->b_rptr;

        this->ihl = 4 * ((int)*((char*)(this->iphdr)) & 0xf);

        this->tcphdr = (struct tcphdr*)((char*)(this->iphdr)+(this->ihl));

        self->srcAddr = *(uint32_t*)((char*)(this->iphdr)+0xc);

        self->srcPort = ntohs(this->tcphdr->th_sport);

}

Then tcp_send_conn_ind():

fbt:ip:tcp_send_conn_ind:entry

/self->lport == $1/

{

        self->action = "syn-aa";

        this->mblk = (mblk_t *)arg1;

        this->tconnind = (struct T_conn_ind*)(this->mblk->b_rptr);

        this->tcpp = *(tcp_t**)(((char*)this->tconnind)+(this->tconnind->OPT_offset));

        self->srcPort = ntohs(*(uint16_t*)((char *)this->tcpp->tcp_connp+0x108));

        self->srcAddr = *(uint32_t*)((char*)this->tcpp->tcp_connp+0x104);

}

The printout then is common to both, since all of it is data driven from the previous stages:

fbt:ip:tcp_conn_request:entry,

fbt:ip:tcp_send_conn_ind:entry

/self->lport == $1/

{

        printf("%d,%Y,%s,0x%08x,%d,%d,%d,%d,%d\n",

                timestamp, walltimestamp,

                self->action,

                self->srcAddr, self->srcPort,

                self->lport,

                self->tcpq->tcp_conn_req_cnt_q0,

                self->tcpq->tcp_conn_req_cnt_q,

                self->tcpq->tcp_conn_req_max

        );

}

Finally we clear the buffers to tidy up:

fbt:ip:tcp_conn_request:return,

fbt:ip:tcp_send_conn_ind:return

/self->lport/

{

        self->tcpq = 0;

        self->lport = 0;

        self->action = 0;

        self->srcAddr = 0;

        self->secPort = 0;

}

If we run this on a connection we can see the two parts. Notice the value of Q0. On entry it is one as we have an embryonic connection which is about to transition to Q.

root@sol10-u9-t4# ./tcpq_ex11.d 22

TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax

185255847646301,2016 Aug 11 15:55:29,syn,0x01aa10ac,41010,22,0,0,8

185255847845639,2016 Aug 11 15:55:29,syn-aa,0x01aa10ac,41010,22,1,0,8

If we fill up Q so it equals Qmax (using my test script from a previous blog) and retry, accepting a connection on the queue part way through to free the kernel backlog we get this, clearly showing the remote servers tcp retries.

root@sol10-u9-t4# ./tcpq_ex11.d 2000

TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax

185458485779874,2016 Aug 11 15:58:52,syn,0x01aa10ac,50066,2000,0,2,2

185459488194513,2016 Aug 11 15:58:53,syn,0x01aa10ac,50066,2000,0,2,2

185461492294103,2016 Aug 11 15:58:55,syn,0x01aa10ac,50066,2000,0,2,2

185465496397317,2016 Aug 11 15:58:59,syn,0x01aa10ac,50066,2000,0,1,2

185465496517076,2016 Aug 11 15:58:59,syn-aa,0x01aa10ac,50066,2000,1,1,2

As you can see, the max'ed out queue resulted in the connection taking 185465496517076 minus 185458485779874 = 7010737202 ns (or 7.01 seconds) to establish the connection.

In the next part we will look at extending this into the application, using both syscall tracing and userspace tracing.

As always, please can you provide feedback so I can improve the blog.