Skip to content

The importance of regular server status reporting for unexpected events

The following scenario highlights the importance of having regular server status reporting in place for unexpected events.

For a couple of months at irregular intervals we had users complain that some mails (not all) they’d been expecting from external users were arriving days later or were not arriving at all.

I’d been examining the headers of delayed e-mails when they did arrive and had found that external mail servers delivering e-mails to ours were taking a number of hours or days to do so. Most of the delayed e-mails “seemed” to originate from Eircom (an ISP based in Ireland) so I chalked the problem down to an issue on Eircoms side. Our Security team agreed. This was an oversight on my part however which I’ll explain later…

As I examined more closely I was able to pull more delayed e-mails from external users whose e-mail were not routed through Eircom so logically this meant the problem wasn’t specifically with Eircom.

As a test I had some of the users that reported delays e-mail my employers e-mail address whilst CCing in my personal gmail address. Eventually the issue reproduced itself.

Mails arrived in my Gmail Inbox within a couple of minutes whilst e-mail destined for my employers e-mail address arrived a number of hours later. In the end I was able to reproduce the issue with the help of 4 seperate senders.

Convincing the Security team was a problem however as everything looked fine on our end. They had a hard time accepting the issue was on our side even though I could reproduce the problem with four seperate senders. Four seperate external mail servers behaving in the same way at the same time? and the problem is not on our end?

They needed more evidence…

So the next port of call was Eircom – They were able to provide the following logs. IP addresses and e-mail addresses removed:

2010-07-13 17:22:19.744009500 starting delivery 2146375: msg 3317173 to remote [email protected]
2010-07-13 17:28:59.581941500 starting delivery 2147602: msg 3317173 to remote [email protected]
2010-07-13 17:49:00.077173500 starting delivery 2150674: msg 3317173 to remote [email protected]
2010-07-13 18:22:19.009197500 starting delivery 2158998: msg 3317173 to remote [email protected]
2010-07-13 19:08:59.078851500 starting delivery 2171856: msg 3317173 to remote [email protected]
2010-07-13 20:08:59.270208500 starting delivery 2188502: msg 3317173 to remote [email protected]
2010-07-13 21:22:20.122931500 starting delivery 2203875: msg 3317173 to remote [email protected]
2010-07-13 22:48:59.138197500 starting delivery 2224308: msg 3317173 to remote [email protected]
2010-07-14 00:28:59.064738500 starting delivery 2244997: msg 3317173 to remote [email protected]
2010-07-14 02:22:19.239298500 starting delivery 2261581: msg 3317173 to remote [email protected]
2010-07-14 04:28:59.989719500 starting delivery 2279211: msg 3317173 to remote [email protected]
2010-07-14 06:48:59.004682500 starting delivery 2299239: msg 3317173 to remote [email protected]
2010-07-14 09:22:19.021063500 starting delivery 2342483: msg 3317173 to remote [email protected]
2010-07-14 12:08:59.033052500 starting delivery 2428140: msg 3317173 to remote [email protected]
2010-07-13 17:22:19.744009500 starting delivery 2146375: msg 3317173 to remote [email protected]

2010-07-13 17:24:09.861329500 delivery 2146375: deferral: Connected_to_<IPAddress>_but_sender_was_rejected./Remote_host_said:_451_#4.1.8_Domain

_of_sender_address_<3rdparty@theircompany.ie>_does_not_resolve/

From the above you can see that Eircoms gateway had attempted to deliver the mail in question numerous times over two days but was being rejected because the senders e-mail address could not be resolved.

So I asked the Security team to investigate if any work had been carried out on our DNS infrastructure (or any failures) on the 13th that would prevent the Ironport performing a DNS lookup.

They found that one of the DNS servers which our Ironport server was actively using for sender verification was restarting intermittently causing our Ironport to drop connection attempts when the external mail server attempted delivery. 

They didn’t have status reporting in place for the DNS servers to report any unexpected events.

Moral of the story – don’t rely on e-mail headers to judge mail delivery attempts (they only indicate successful connections) and make sure you have status reporting in place.

iPhone issue – Server error Contact your server administrator Cannot get mail

Got the above error message on a users iPhone today – It was easy for me to trouble shoot as the user also uses an iPad and indicated he was also having problems downloading mail on that device.

Symptoms: When the iPhone mail app is accessed a notification appears towards the bottom for a number of seconds indicating “Downloading” but no new e-mails are downloaded and after a few more seconds the “Server error – Contact your server administrator – Cannot get mail” error message appears.

If you’ve already tried adding/removing the ActiveSync account details (and all the other recommended troubleshooting steps) and your device still cannot download e-mail – go back to the day your iPhone/iPad stopped receiving mail in Outlook (or whatever desktop client you normally use) and delete all mails from that day. Also empty your “Deleted Items” to be safe.

Your device cannot download new e-mails because one of the e-mails you’ve received has become corrupted and is now holding up other mails in the queue.

Once you’ve deleted the corrupted mail you should then be able to download the newer ones.

Kleptomania – Grab text from non-clipboard aware dialog boxes

Here’s a tool I got a lot of use of in my former life as a Software Test Engineer.

Kleptomania is able to grab text from dialog boxes which the clipboard cannot select and copy from.

So for example if an error dialog appears you can grab the text from it exactly how it appears, which normally the OS would not let you do.

It’s a really great tool for Software Testers but not 100% reliable. That said it’s a really great time saver when you need to plug in the full error message as text in your bug “Steps to Repro”…

HFSExplorer – Open Mac formatted data on your PC

Here’s a nice little app to use when your Mac breaks down and you need to read Mac formatted data on your trusty reliable and all round nice PC 😉

HFSExplorer does exactly what it says on the tin – in this case the tin is a web page from software dev Erik Larsson.

Plus it’s free – you can’t argue with that

Check it out – it works. Trust me I’m a doctor…

Finally a performance laptop with WiDi on board – WiDi 2.0 even

I’ve been in the market for a new laptop for some time now but haven’t came across anything that tickles my fancy or combines performance, USB 3.0 and Intel’s new WiDi technology which I’m already a fan of but haven’t had the opportunity to view in the flesh.
At last I think I’ve found the holy grail! Sony’s just released the new Vaio S

It starts at under 1000 dollars which isn’t cheap for a laptop these days but its cheap for Sony – when compared to the price of the old Vaio Z – which has now mysteriously disappeared from Sony’s product line up. Will we see an update to the Vaio Z coming along soon?

Yes – I am a Sony fan. I’ve had my Vaio VGN-S380P which was a great laptop and though I’m ashamed to say it I’m actually a bit proud to say also that it’s been going for 6 years now. The only upgrade I made was to increase the memory from 512mb to 1Gb. I should really have at least 2Gb’s in there at this stage – but it’s puttering along nicely for now.

Vaio S here I come – or maybe I’ll wait to see what the new Vaio Z has in store…

HTML5 Fragmentation

It seems my HTML5 predictions have started to come true. Google will no longer support H.264 video by default in it’s Chrome browser even though H.264 has seen wide implementation on the internet.

Google decided instead to focus on supporting WebM video though it doesn’t enjoy as wide implementation on the internet as H.264. This is obviously a political move on Googles part and has resulted in the term “HTML5 Fragmentation” being coined on the interwebs…

Microsoft upon hearing this announcement however rode to the rescue and released a Chrome browser extension that re-enables H.254 video in Chrome

You can get that extension here – Thanks – You’re welcome…

The argument against HTML5 – (for now)

The recent controversy surrounding Apple dropping Flash support from some of it’s product range has highlighted numerous shortcomings in the browser plugin, along the lines of performance, reliability and security.

Apple publicly criticised the plugin – recommending  HTML5 to Joe Public instead, but were they right to drop support for Flash, possibly damaging their up to now close relationship with Adobe?

In my opinion no. Here’s why:

HTML5  is still a draft standard and will continue to be so for some time to come. Due to the fact that the HTML5 feature set has not been standardized yet – web browser developers are hampered deciding which HTML5 features to support in their new web browser releases. This means that it may not be possible to view some parts of a webpage coded in HTML5 depending on which web browser and OS you have installed.

Standards also give web developers the assurance that the code they use to design a web page should work across a range of web browsers – in an ideal world that’s the way it should happen. In reality this doesn’t happen – different browsers display webpages differently. OK – but do we need Apple adding another layer of complexity to web developers jobs by forcing them to decide which browser(s) they’ll concentrate their coding and testing efforts on?

Instead of the best all round browser gaining the biggest market share the end user will probably end up suffering – they’ll be forced to choose a browser that supports the most HTML5 features. All because Apple wanted to conserve battery life on their mobile devices…

HTML5 – It should be great – but I don’t think its ready for prime-time just yet.

Flash

Logitech DiNovo Edge

I don’t know anyone that’s fond of wires trailing down the back of their PC or Mac.

And I for one have been looking to free myself from the burden of cables for a long time. Finally – the tech world seems to be making some progress.

The Logitech DiNovo Edge has been on the market for some time now but it only recently came to my attention when I started to look for a device that integrates a wireless keyboard and mouse. I also like the fact that it includes a convenient recharging dock so you don’t have to deal with rechargable batteries.

It doesn’t look half bad either:

Intel WIDI

Here’s a technology I’m excited about…

It’s Intel’s WIDI which stands for “Wireless Display”. The tech allows you to stream your PC/Laptop video to your TV wirelessly so you can do away with VGA/audio cables.

Three big manufacturers have released this kit baked into their laptops already- Dell , Sony and Toshiba.

Currently you need to hook a receiver box up to your TV in order to get the video stream onto your TV – that’s where the Netgear “Push2TV” comes in. It currently comes bundled with laptops sporting the tech. Your Intel Core 2010 laptop then creates a Personal Area Network over the Wireless N standard to push the stream to the TV.

The upsides are WIDI supports video up to 1080p. Downsides are Blu-Ray video streams are not allowed due to DRM restrictions

If this technology takes off we may see manufacturers integrating the receiver into their TV’s – Here’s hoping….

The Blog of Martin Birrane