Bogons Ate My Web Site

We were recently contacted by one of our customers wanting to know why their site was dropping in their search engine rankings. The site's presence in Yahoo! was slowing eroding - every day a few more indexed pages would disappear. The site had not undergone any content changes, but it had recently changed ISPs. What could be causing the problem? It was only through the teamwork of networking staff at both Yahoo! and RackSpace that the issue was identified, isolated, and corrected. Curiously, neither was to blame, but both were instrumental in solving the problem.

There were a number of unusual aspects to the case, including the fact that only rankings on Yahoo seemed to be affected. More interestingly, Yahoo's spiders were reaching the site according to the logs, but when we checked with Yahoo they were apparently reporting an error.

So we began a detailed analysis of what was going on. The analysis was made more difficult by the fact that running a traceroute from Yahoo to the website showed that the ping made it into RackSpace's network before disappearing. RackSpace, doing the same trace in reverse, noted that the ping disappeared once Yahoo's network was reached, thus prompting a scenario where each initially blamed the other based on the evidence presented.

We had no reason to believe that either party was misleading us, so we began a careful troubleshooting checklist. The site itself was clean and fast, used no risky tactics and was very spiderable. The robots.txt and other robot behavior control codes freely allowed spiders into the site, and there were no errors in the DNS or other related server technologies. Additionally, the site had been spidered by various search engines in the past, and was ranking well in several categories. We knew it wasn't the site, and began to check the network.

First, we checked the IP address of the website. There was no indication of it being blacklisted for spam purposes.

Next, we checked for the website being blocked by Yahoo or the Yahoo spiders being blocked by RackSpace. Both parties assured us that this was not the case. Yahoo was able to confirm that the spiders were unable to access the site, however.

The only odd factor in the new hosting location was that the IP block was only recently allocated - previously it was part of a reserved block not in use anywhere. It was this oddity that allowed us to identify the culprit: a monster lurking in the internet known as a bogon.

After a few phone calls and some additional tests, we verified that a bogon had indeed eaten our website and Yahoo's spiders, and could very well be lurking out there on the internet waiting for others, as well. Once we confirmed this to Yahoo they we able to deal with the matter and the client is now enjoying his previous rankings and traffic.

So, you are probably asking, what is a bogon?

Let's back up a step and outline an issue that networks deal with all the time - spammers and hackers. Naturally these people do not wish to be traced or identified, so they often send fake return IP addresses to make it difficult to track them. It's like sending a nasty letter to someone and using a fake return address.

There are billions of potential IP addresses in the world, including a lot of IP addresses that are not being used. They are either being held in reserve for future needs, or are allocated for testing or other purposes. It's a fair assumption that anything coming to you with an address you know is fake is probably up to no good.

Network administrators have discovered that if they block all of these known unused addresses from moving through their networks, they can block upwards of 60% of spam and hackers. This makes blocking known unused addresses a very easy way to make your network run better and make your users happy.

These unused IP addresses are collectively known as bogons.

Smart network administrators block bogons at the router or firewall level, never letting them get into the system in the first place. But what happens when a previously unallocated IP address is released into public use?

In a perfect world, all network administrators everywhere would either manually or automatically update their bogon filters to allow the new IP's to flow through their network. But it's not a perfect world, and RackSpace reports that a large number of administrators either do not update their bogon filters automatically or are running systems that need to be manually updated and have not done so yet.

This means that if you are given an IP address that was a bogon until recently, you may find your website blocked from various ISP's and networks.

The issue with our client was not that Yahoo was blocking the IP, but Yahoo's ISP was. Since the main router for RackSpace was a known safe IP address, what would happen was that the trace from Yahoo would leave Yahoo, go through their ISP, enter RackSpace's network and then get directed to the new IP, all the while reporting back at each step that it had successfully arrived.

Of course, the report back from the new IP was being blocked by Yahoo's ISP, which made it look like the trace ping entered RackSpace but never arrived at the new IP, since the return packet was being blocked. This explains why Yahoo's spiders were showing up in the logs, but were not reporting their visit back to Yahoo - they were visiting and reporting, but the reports were being intercepted by the bogon filter and discarded.

The lucky thing is that in this case there was only one intervening network between Yahoo and RackSpace, allowing us to narrow in quickly. Imagine the potential confusion if there had been several ISP's. Worse, what if the spiders took a different path to the website sometimes? You would get an intermittent block to the spiders and visitors, which would be very hard to track down unless you knew exactly what you were looking for.

Yahoo's ISP has since fixed this problem, but there are a great many ISP's and networks out there that have not. There may be websites that are not accessible to some search engines and visitors, with the website owner not having any clue as to the reason why.

So, how do you check to see if you are affected by a bogon filter? First, check your IP address. If it's a recently allocated (within the last year) IP then there is a possibility that it may be affected by bogon filters put in place by various networks. Another symptom is that a traceroute will show a response at each step of the way until the new IP is reached, and it is consistently blocked by the same network node.

At this time, we recommend that you avoid newly released IP addresses unless you are certain that they are not being affected by bogon filters. If you are a network administrator who uses bogon filters, we strongly recommend you use one of the many freely available automatic bogon updating scripts and services.

Bogon filters can be a powerful ally in fighting hackers and spammers, but it's important to realize that a website or mail server can be inadvertently blocked by out-of-date bogon lists as a false positive. Without the help and support of both Yahoo and RackSpace, this would have been a very difficult issue to troubleshoot.

Useful Links:

Ian McAnerin, founder of McAnerin Networks Inc, is a moderator for the High Rankings and the Search Engine Watch forums, and has been vetted by the standards watchdog SEOConsultants.com and SEOPros directories. He is a founding member of the Search Marketing Association of North America, with published articles in many international newsletters and web site news reports. He has a special interest in SEO legal issues due to his legal background.

Mike Churchill is the Tech Guru at KeyRelevance.com. He has been an Internet Junkie since 1992, cutting his Internet teeth on FTP, Archie, and Mosaic 1.0. He is the former Chief Technical Officer and co-founder of NetMechanic.com and is an avid developer of Internet applications.

© Ian McAnerin and Mike Churchill 2005

Editor: Mike Grehan. Search engine marketing consultant, speaker and author. http://www.search-engine-book.co.uk

Associate Editor: Christine Churchill. KeyRelevance.com

e-marketing-news is published selectively on a when it's
ready basis. ©2005 Net Writer Publishing.

At no cost you may use the content of this newsletter on
your own site, providing you display it in its entirety
(no cutting) with due credits and place a link to:

< http://www.e-marketing-news.co.uk >

• Intro from Christine Churchill
• Interview with Apostolos Gerasoulis and Jim Lanzone
• Bogons Ate My Web Site
• Craig Garber Copywriting Tips