header image
 

Make sure Sender Policy Framework (SPF) is correctly configured

We had some issues at work recently with intermittent e-mail’s from a third party not arriving in our Google App’s mailboxes (we’ve now migrated to using Google App’s for Business as opposed to using Exchange exclusively).

This was a big problem for us as these e-mail’s are an important source for news stories.

I immediately suspected spam/DNS server issues due to my previous experience trouble-shooting our Ironport issue two years ago.

This time however the problem lay with the third parties DNS not ours.

Here are the steps I took to troubleshoot then fix the problem:

Symptoms:

The issue was first reported when one of our news reporters noted some e-mails were not arriving – sent in from this news sources distribution list.

Some e-mail not all were being dropped – this is an important factor as we’ll see below.

Troubleshooting steps:

1. I realised straight away the problem wasn’t related to any e-mail filters in the reporters inbox or SMTP blacklists since approx only 20% of the mails were not arriving. Nonetheless I ran the check’s necessary and found nothing causing mail to be filtering into the users bin.

2. I checked in on another colleague who’s mail domain address is different from the news reporter that detected the problem (We use multiple domain names for separate titles and business teams). That colleague indicated they were missing the exact same e-mails that the first reporter detected missing.

3. The distribution list in question is used by a number of other news organisations so I sent a mail out to another news organisation’s IT dept to ask if they’d noticed those particular e-mails missing – they hadn’t.

4. I then asked an ordinary user at the news sources site, (I’m calling her deirdre@xxxx.ie here) to send me an e-mail so I could evaluate the incoming e-mail headers for anything out of place. The e-mail she sent arrived fine but the information I found in her mail header provided the light bulb moment when taken into account with the rest of the information gathered. I found two key entries in deirdre@xxxx.ie‘s mail header:

Received-SPF: fail (google.com: domain of deirdre@xxxx.ie does not designate 137.191.225.35 as permitted sender) client-ip=137.191.225.35

Authentication-Results: mx.google.com; 

spf=hardfail (google.com: domain of deirdre@xxxx.ie does not designate 137.191.225.35 as permitted sender) smtp.mail=137.191.225.35;

It was time to brush up on SPF or Sender Policy Framework

SPF basically boils down to a DNS entry that indicates which SMTP servers are permitted to send e-mail on behalf of a mail domain. This prevents spammers from getting e-mail’s into a users inbox since spam prevention e-mail gateways like Ironport and Postini will perform a DNS verification check each time a particular SMTP server tries to deliver an e-mail purporting to be from a particular e-mail address (domain).

As an example say a spammer tried to deliver an e-mail to jobloggs@yahoo.ie, by spoofing an innocent individuals e-mail address (let’s say mr.smith@hotmail.com) and used their spamming SMTP server to try to send jobloggs@yahoo.ie a spam e-mail. The Ironport/Postini/Spam Gateway when it’s first contacted examines the mailing domain indicated in the e-mail address of the message – say “hotmail.com” and the accompanying IP address of the spamming SMTP server.

Ironport/Postini will then contact the DNS server (which should always be accessible on the internet) for hotmail.com  to check that DNS Server’s SPF record. If it see’s that the IP addresses listed in the SPF record doesn’t match the IP address of the SMTP server that just tried to deliver an e-mail purporting to be from mr.smith@hotmail.com it will drop that e-mail delivery attempt to jobloggs@yahoo.ie, thereby preventing the spam reaching jobloggs@yahoo.ie‘s inbox. According to hotmail.com‘s DNS SPF record the spamming SMTP server trying to deliver a message purporting to be from mr.smith@hotmail.com is not authorized…

Now back to our problem – So our news organisations Postini gateway was designating the IP address of our third parties SMTP server 137.191.225.35 as not being permitted to send e-mail on behalf of that user.

I had two problems with that –

1. First why did the SPF fail?

2. The SPF failed but deirdre@xxxx.ie‘s e-mail got through anyway.

It was time to probe our news sources (xxxx.ie‘s) DNS configuration. In the process of doing so I discovered it was possible to expose the SPF configuration for DNS server’s available on the internet using this tool

So I plugged in the IP Address of the 137.191.225.35 SMTP server designated not authorized in deirdre@xxxx.ie‘s e-mail header.

The Beveridge SPF Test tool then exposed the following SPF configuration for the xxxx.ie domain (some IP addresses changed for security purposes):

v=spf1 mx ip4:137.191.xxx.x1 ip4:137.191.xxx.x2 ip4:137.191.xxx.x3 -all [TTL=86400]

But I also noticed subsequent SPF test’s also returned a different SPF configuration below:

v=spf1 mx ip4:137.191.xxx.x1 ip4:137.191.xxx.x2 ip4:137.191.xxx.x3 ip4:137.191.225.35 ip4:137.191.xxx.x5 -all

So what was I looking at?

Basically one of more DNS servers for the domain xxxx.ie which our Postini gateway was performing SPF checks against had an incorrect/out of date SPF configuration. The second SPF configuration was the correct one as it listed 5 SMTP servers that were authorized to send mail on behalf of the xxxx.ie domain

The first incorrect SPF also should not have contained the  “[TTL=86400]” reference, as this doesn’t conform to SPF standards.

Cross-referencing the IP address of the 137.191.225.35 SMTP Server in the e-mail header I found that 137.191.225.35 matched a missing IP address in the incorrect SPF record. 137.191.225.35 was present in the second SPF record but not the first.

So our Postini was hitting the DNS server with the incorrect SPF configuration about 20% of the time and therefore dropping 20% of the e-mails sent because it was determining that 137.191.225.35 wasn’t authorized to send e-mail on behalf of xxxx.ie 20% of the time.

I’d figured out the issue was was due to an incomplete SPF record on one xxxx.ie DNS server but why did the e-mail from deirdre@xxxx.ie deliver though it was given an SPF failure while some e-mails from the distribution list pressxxxx@xxxx.ie were not being at all delivered?

My guess is that our Postini gateway even though it flagged deirdre@xxxx.ie as being an SPF failure – SPF checking for Google/Postini is evidently not that strict. There must be additional spam filtering criteria in place on our Postini that flags the pressxxxx@xxxx.ie e-mails based on message content, that together with the results returned from the problem DNS server flagged the pressxxxx@xxxx.ie as being spam 20% of the time. Our Postini services are provided by an outsourced company so I don’t have access to the specific spam filtering criteria to verify this.

Fix:

I contacted the IT Security department manager for the xxxx.ie, who confirmed my findings and isolated the problem DNS server. The DNS server in question was not under his dept’s direct control but was managed by another dept – I’m guessing for redundancy purposes.

He logged a change to have the SPF details applied to the problem DNS server which should eliminate our problems receiving mail from any @xxxx.ie addresses in the future.

Hopefully this has given you an insight to the complex world of spam filtering….

~ by Martin on April 26, 2013.

Security, Troubleshooting

5 Responses to “Make sure Sender Policy Framework (SPF) is correctly configured”

  1. good article,but very hard to read,make your background white and do writing in black

  2. I believe this internet site has some really fantastic information for everyone. “He is able who thinks he is able.” by Buddha.

  3. Woah! Im really digging the template/theme of this website. Its simple, yet effective. A lot of times its very difficult to get that perfect balance between superb usability and visual appeal. I must say you have done a amazing job with this. In addition, the blog loads extremely quick for me on Safari. Exceptional Blog!

  4. im having a issue with this SFP from a godaddy hosted site. People at GoDaddy yet tell me i need to contact my ISP because they have it blocked. I find that very strange cause the mails get perfect in Gmail but on hotmail they get in a spambox. I dont get what a ISP has to do with this?

    • Hi Rombout,
      I don’t check comments on my website that often as they’re mainly spam so sorry for the slow reply

      You’ll need to be more specific on your mail setup Rombout, could you send me on the following:
      The e-mail address you’re sending the e-mail from and please send a sample e-mail to mbirrane@hotmail.com so I can check your mail headers – that may give me a clue as to why hotmail is flagging it as spam.

      If I’m guessing correctly you’re having problems with e-mail sent from your mail client arriving in your hotmail recipients Inbox as spam? Yet g-mail recipients receive your e-mails fine?

      If that’s the case and the issue is as GoDaddy has described then this could be the problem:
      When hotmail does it’s SPF verification to check if the mail senders IP is authorised to send e-mail on behalf of the GoDaddy mail server, your ISP is blocking hotmail from checking the SPF record in DNS

      You may be able to get around this by checking and sending your GoDaddy hosted e-mail from G-mail itself (but you need to make sure you configure it correctly), these instructions may help you do that:
      https://uk.godaddy.com/community/forums/v3_1/forumtopicpage/board-id/Email-Office/thread-id/504/page/1
      https://www.howtogeek.com/school/gmail-guide/lesson9/

      That may be enough to authorise G-mail to send e-mail on behalf of your GoDaddy mail server

      I use Bluehost as my host so I can’t help you with the specific settings for GoDaddy to make it work (it may be tricky to get working) and I’m not even sure if sending e-mail from G-mail will bypass your problem but I’d still give it a try

      You might also want to check what kind of content you’re sending in your e-mails, if you’re sending e-mail with content in them that looks like a newsletter, or an e-mail campaign hotmail could be flagging those as spam automatically

      Hope that helps
      Martin

Leave a Reply




 
%d bloggers like this: