Archive for category Service Announcements

This Sunday the 7th September 2008 our local electricity supplier will be upgrading the local power relay to our building. Regrettably this means that some of our data-center services will not be available during the hours of 8am and 2pm.

What impact does this have?

• All mail will be directed to our second data-centre in Telford and queued for delivery (if you use our MailScanner cleaning service). If you have an office server that handles mail from your site your mail will continue to be delivered normally. If your email account is hosted remotely or we operate a Zimbra server on your behalf these services will be available after 2pm.

• VoIP telephone services will operate normally, with the exclusion of hosted VoIP.

• Application hosting will operate normally.

• 80% of website hosting will suffer, premium web stores will still be available.

• Inbound Fax services will not be available.

Engineers will be available on-site following the downtime to re-commission and monitor the services.

If you have any questions regarding this news post please do not hesitate to contact me.

A news post just left our system that should have gone out on the 16th June 2008, please ignore this message as it was related to some downtime last month.

Today at 12:46pm our data centre was affected by the UK wide failing leased line circuits at BT exchanges. BT were alerted to the issue by Easynet and BT had restored connections by 3:45pm. We are waiting for confirmation from BT that everything has been restored successfully and that this issue is now fully resolved. 

We apologise for any inconvenience caused. 

The DNS records for the mailscanner2.theitguys.co.uk cluster have been checked periodically throughout the night with several different service providers with positive results.

Both mail scanning cluster queues and services have also been monitored throughout the night and all mail has been delivered successfully.

Part of our continued commitment to upgrading our core service network we will be splitting the scanning clusters between our existing BVIC data-centre and our new Telford data-centre. The upgrades will be complete by next Tuesday and will add further redundancy to our mail scanning service, which will help prevent incidents like we have suffered this week.

The Telford data-centre sits on a 10Gb link into tele-house in London, has full power fail-over generation and will be primarily running our Apple MacOSX Server Xgrid clusters for web applications and email services. More information will be posted on this subject when the upgrades are complete.

Once again please accept our apologies for any inconvenience caused.

DNS is now resolving fully, there will be the odd service provider that is not yet up to date. You may notice a flurry of messages that should have been delivered over the last few days, please check the dates and times that messages should have been delivered rather than the actual delivery time.

Any messages that are not received in the next few hours will have been bounced back to the sender.

We are continuing to monitor the situation.

We have received a few messages following our earlier post so this update attempts to address them.

Question: Certain people emailing me are still seeing bounced messages.

Answer 1: Some service providers will have old cached DNS records suggesting that mailscanner2.theitguys.co.uk is unavailable. The time-to-live on theitguys.co.uk is 2 hours, which means some service providers will not re-check the records until later today.

Answer 2: People emailing you may have sent a message on Monday and only just received a bounce. Mail servers by default are configured to retry for up to five days and then bounce a message back to the sender. They don’t always give an immediate bounce to a message.

Serious DNS issues can occur with service providers that cache DNS names and ignore the time-to-live records for the domain. Service providers that are known to do this include AOL, BT and Virgin.

We will keep you posted on the progress of this situation.

For the more technical amoungst us the correct DNS for mailscanner2 is:

$ dig mailscanner2.theitguys.co.uk mx @ns1.easynet.co.uk

; < <>> DiG 9.4.1-P1 < <>> mailscanner2.theitguys.co.uk mx @ns1.easynet.co.uk
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER< <- opcode: QUERY, status: NOERROR, id: 20069
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3

;; QUESTION SECTION:
;mailscanner2.theitguys.co.uk.	IN	MX

;; ANSWER SECTION:
mailscanner2.theitguys.co.uk. 10760 IN	MX	5 mailscanner2.theitguys.co.uk.

;; AUTHORITY SECTION:
mailscanner2.theitguys.co.uk. 2794 IN	NS	ns2.theitguys.co.uk.
mailscanner2.theitguys.co.uk. 2794 IN	NS	ns1.theitguys.co.uk.

;; ADDITIONAL SECTION:
mailscanner2.theitguys.co.uk. 2794 IN	A	78.33.33.164
ns2.theitguys.co.uk.	2794	IN	A	82.109.198.9
ns1.theitguys.co.uk.	13331	IN	A	78.33.33.162

;; Query time: 20 msec

;; SERVER: 195.40.0.250#53(195.40.0.250)
;; WHEN: Thu May 22 12:58:05 2008
;; MSG SIZE  rcvd: 146

$ dig mailscanner2.theitguys.co.uk a @ns1.easynet.co.uk

; < <>> DiG 9.4.1-P1 < <>> mailscanner2.theitguys.co.uk a @ns1.easynet.co.uk
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER< <- opcode: QUERY, status: NOERROR, id: 47152
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;mailscanner2.theitguys.co.uk.	IN	A

;; ANSWER SECTION:
mailscanner2.theitguys.co.uk. 2634 IN	A	78.33.33.164

;; AUTHORITY SECTION:
mailscanner2.theitguys.co.uk. 2634 IN	NS	ns2.theitguys.co.uk.
mailscanner2.theitguys.co.uk. 2634 IN	NS	ns1.theitguys.co.uk.

;; ADDITIONAL SECTION:
ns2.theitguys.co.uk.	2634	IN	A	82.109.198.9
ns1.theitguys.co.uk.	13171	IN	A	78.33.33.162

;; Query time: 24 msec
;; SERVER: 195.40.0.250#53(195.40.0.250)
;; WHEN: Thu May 22 13:00:45 2008
;; MSG SIZE  rcvd: 130

This week we have had several reported issues regarding the resolution of DNS and email being bounced. The issues have finally been isolated to one of our primary DNS servers and the records it holds. The server has had it records reloaded from a backup as corruption had occurred, which was difficult to trace due to the randomness of the reported problems and obtaining decent bounce messages.

For information this affected emails being delivered to our secondary mailscanner2 cluster service from some service providers. The exhaustive tests we have performed in the last 20 minutes are showing positive results. We will continue to monitor the situation and inform you of any further situations.

We apologise for any inconvenience caused.

During the night we were exposed to a power outage, which tripped our main power lines. Some equipment survived, however the power to routers and switches failed. Normal power came back on at 8:02am this morning and engineers started testing services - all services were confirmed working at 9:05am.

Customers concerned about email will start to receive a trickle as the MailScanners are now available. Email is redundant by design therefore any messages queued waiting to be delivered to accounts will now filter through automatically. Anybody trying to send a message through to you may have received the following bounce:

  ----- Transcript of session follows -----

your@email.address ... Deferred: saferelay.intercellsolutions.net.: No route to host

Warning: message still undelivered after 4 hours

Will keep trying until message is 5 days old

If you have any concerns please do not hesitate to contact support on 0845 004 6789.