Archive for May, 2008

The DNS records for the mailscanner2.theitguys.co.uk cluster have been checked periodically throughout the night with several different service providers with positive results.

Both mail scanning cluster queues and services have also been monitored throughout the night and all mail has been delivered successfully.

Part of our continued commitment to upgrading our core service network we will be splitting the scanning clusters between our existing BVIC data-centre and our new Telford data-centre. The upgrades will be complete by next Tuesday and will add further redundancy to our mail scanning service, which will help prevent incidents like we have suffered this week.

The Telford data-centre sits on a 10Gb link into tele-house in London, has full power fail-over generation and will be primarily running our Apple MacOSX Server Xgrid clusters for web applications and email services. More information will be posted on this subject when the upgrades are complete.

Once again please accept our apologies for any inconvenience caused.

DNS is now resolving fully, there will be the odd service provider that is not yet up to date. You may notice a flurry of messages that should have been delivered over the last few days, please check the dates and times that messages should have been delivered rather than the actual delivery time.

Any messages that are not received in the next few hours will have been bounced back to the sender.

We are continuing to monitor the situation.

We have received a few messages following our earlier post so this update attempts to address them.

Question: Certain people emailing me are still seeing bounced messages.

Answer 1: Some service providers will have old cached DNS records suggesting that mailscanner2.theitguys.co.uk is unavailable. The time-to-live on theitguys.co.uk is 2 hours, which means some service providers will not re-check the records until later today.

Answer 2: People emailing you may have sent a message on Monday and only just received a bounce. Mail servers by default are configured to retry for up to five days and then bounce a message back to the sender. They don’t always give an immediate bounce to a message.

Serious DNS issues can occur with service providers that cache DNS names and ignore the time-to-live records for the domain. Service providers that are known to do this include AOL, BT and Virgin.

We will keep you posted on the progress of this situation.

For the more technical amoungst us the correct DNS for mailscanner2 is:

$ dig mailscanner2.theitguys.co.uk mx @ns1.easynet.co.uk

; < <>> DiG 9.4.1-P1 < <>> mailscanner2.theitguys.co.uk mx @ns1.easynet.co.uk
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER< <- opcode: QUERY, status: NOERROR, id: 20069
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3

;; QUESTION SECTION:
;mailscanner2.theitguys.co.uk.	IN	MX

;; ANSWER SECTION:
mailscanner2.theitguys.co.uk. 10760 IN	MX	5 mailscanner2.theitguys.co.uk.

;; AUTHORITY SECTION:
mailscanner2.theitguys.co.uk. 2794 IN	NS	ns2.theitguys.co.uk.
mailscanner2.theitguys.co.uk. 2794 IN	NS	ns1.theitguys.co.uk.

;; ADDITIONAL SECTION:
mailscanner2.theitguys.co.uk. 2794 IN	A	78.33.33.164
ns2.theitguys.co.uk.	2794	IN	A	82.109.198.9
ns1.theitguys.co.uk.	13331	IN	A	78.33.33.162

;; Query time: 20 msec

;; SERVER: 195.40.0.250#53(195.40.0.250)
;; WHEN: Thu May 22 12:58:05 2008
;; MSG SIZE  rcvd: 146

$ dig mailscanner2.theitguys.co.uk a @ns1.easynet.co.uk

; < <>> DiG 9.4.1-P1 < <>> mailscanner2.theitguys.co.uk a @ns1.easynet.co.uk
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER< <- opcode: QUERY, status: NOERROR, id: 47152
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;mailscanner2.theitguys.co.uk.	IN	A

;; ANSWER SECTION:
mailscanner2.theitguys.co.uk. 2634 IN	A	78.33.33.164

;; AUTHORITY SECTION:
mailscanner2.theitguys.co.uk. 2634 IN	NS	ns2.theitguys.co.uk.
mailscanner2.theitguys.co.uk. 2634 IN	NS	ns1.theitguys.co.uk.

;; ADDITIONAL SECTION:
ns2.theitguys.co.uk.	2634	IN	A	82.109.198.9
ns1.theitguys.co.uk.	13171	IN	A	78.33.33.162

;; Query time: 24 msec
;; SERVER: 195.40.0.250#53(195.40.0.250)
;; WHEN: Thu May 22 13:00:45 2008
;; MSG SIZE  rcvd: 130

This week we have had several reported issues regarding the resolution of DNS and email being bounced. The issues have finally been isolated to one of our primary DNS servers and the records it holds. The server has had it records reloaded from a backup as corruption had occurred, which was difficult to trace due to the randomness of the reported problems and obtaining decent bounce messages.

For information this affected emails being delivered to our secondary mailscanner2 cluster service from some service providers. The exhaustive tests we have performed in the last 20 minutes are showing positive results. We will continue to monitor the situation and inform you of any further situations.

We apologise for any inconvenience caused.