I decided to check the traffic statistics for this blog today (I usually do daily..) and was surprised to see my traffic fall over 50%. Wondering if I’ve pissed off somebody and got a bad review.. or something was wrong with the way I designed my site.
The answer was neither of that. The real problem was:
Googlebot not finding my robots.txt file
I logged into my Google Webmaster Tools area and found over 62 errors – mostly containing the “Robots.txt Unreachable” error. The help icon beside the errors stated:
Google encountered an error when trying to access this URL. We may have encountered a DNS error or timeout, for instance. Your server may have been down or busy when we tried to access the page.
My guess is that my server had some downtime when Googlebot was checking this website out. Have you had any trouble lately with getting your site crawled?
UPDATE (8:00PM) – I checked again to see that the errors were down to 17.. mostly consisting of me restricting robots.txt to different file types. I think the problem was fixed as quickly as it appeared.
7 Comments
How important would you say having a site map is, in google webmaster tools? Because I’d done one before but there was an error, but then I didn’t think it was worth figuring out what was wrong. Does that help alot to have it? Thanks for the help.
Jonathan Muller
http://www.mastersofseo.com
How To Get Traffic
mastersofseo’s last blog post..How to Rank in Search Engines
Hi Master Of SEO..
I’d say having a sitemap is pretty important. It allows google to find your site better, without having the googlebot to find it. See this plugin:
http://www.arnebrachhold.de/redir/sitemap-home/
For generating a sitemap that instantly updates whenever you post something.
You’re right Brad. That google sitemap plugin is a must-have tool for website to get google crawl the site structure better.
Icon’s last blog post..$2200+ Worth Of Prizes Given Away from BradBlogging
I had not heard of a problem like this, and will now have a better idea if it occurs. I did read some material about editing or improving the usage of one’s robot.txt file, and the writer of that information said that it led to better results by Google.
Armen Shirvanian’s last blog post..Quality Builds On Itself
I generate a sitemap, and I believe it does make a diference. Make sure you set it to update daily.
Joan’s last blog post..A Robust Free SEO Keyword Research Tool
Howdy! I found you through Nice2All’s blog review.
I’ve had some trouble with Google Analytics lately because I changed my theme and placing the code in my new file proved very frustrating.
I know. They tell you. They want you to put it before the end body text section.
And I did. And like a petulant child it changed by body text and then it took four of five days to see that it was installed. And even so it’s not tracking anything. GRR!
So I ended up putting the old code from the old template somewhere else above the end body text section and it stopped messing with my body text mojo and finally verified that it was working. Only there have been no visits for the last week.
Which cannot be true because I am receiving comments and Sitemeter is tracking visits, so I’m not extremely happy with Google right now. It is possible that using the old code from the old theme — even though my account number was correct — was a problem.
So I used the new code just today, thanks to finding this post, and I’ll see if anything starts tracking in the next couple of days.
Other than that, things are going great with Google Analytics.
Welcome to the blog cardiogirl!
I’d highly recommend the third party tracking software called, “Statcounter”… There has been no downtime for me and it works like a charm!
One piece of code, and it records everything that loads it. It is simply amazing what people can do with code.
I use both on my blog however, just incase one doesn’t work for a while!