We recently migrated from an ADSL line to a 10mb BT leased line. We kept the ADSL as a backup. However, I forgot to change the default gateway of our internal DNS servers to the new line (it was on my ‘do it later, after the dust settles’ lists). This wouldn’t normally cause a problem, although it will slow down people’s internet experience a little as they are getting DNS resolutions from an ADSL line rather than a leased line.
However, when the ADSL line went down today, no-one could use the internet. I couldn’t understand what the problem was, as I knew the internet was working (I could ping an IP address, I just couldn’t resolve a domain name).
The first thing I did was check the DNS servers, and this is when I remembered that they were pointing to the ADSL line rather than the leased line. So I changed their default gateways (and rebooted them for good measure). I was convinced this would do the trick, but it didn’t.
At this stage, the pressure starts to mount. A company without internet is a very unproductive company. And I know very little about DNS. I tested the firewalls (both for the ADSL line and the leased line) and they were able to resolve domain names. The ADSL line actually came back online, but this still didn’t fix the problem.
I was out of ideas, so I phoned a friend. Straight away he asked if port 53 was open on the firewall. This is used for DNS. Of course! I quickly saw that it was open on the ADSL line, but not on the leased line. I opened it up, and bingo, the internet returned, and my users were happy again.
The ADSL line actually went down because an air-con froze and flooded the server room floor, and water must have got into the BT socket. I couldn’t make the connection between an air-con failing and a port being open on the firewall. Would I ever have? It’s times like this when I appreciate the benefits of having someone to call on, who can maybe look at things from a more logical perspective. Because this problem started with a wet floor, I wasn’t able to think logically enough about the real cause of the outage.