info [ AT ] redcardinal [ DOT ] ie
+John Mueller second question in a day
+John Mueller second question in a day. I work with a company who are getting reports in Webamaster Tools of connectivity issues for about ~20% of requests Google makes. They've checked their servers, and the host has also checked things and can find no connectivity issues. I can also crawl the entire site without issue. It could be temporal, but we are trying to figure out if it could be a Google issue. Is there anything you can help us with? No worries if not.
Google+: View post on Google+
This post was first made on the Richard Hearne Google+ profile.
The times I checked these there were real issues involved. Sometimes not as much as Webmaster Tools would make you believe though. For example, if the robots.txt is unreachable briefly, then we'll stop crawling anything until we can reach it again. Also, sometimes with big sites, we crawl so much that even if 10% fail, we still have a pretty good coverage of the new & updated content. Happy to check if you give me a URL though.
Comment by John Mueller — October 2, 2014 @ 12:42 pm
Hi John were seeing 20% fail only over past 2 weeks. Multiple emails coming from wmt telling us also. Domain is motorcheck.ie if there's anything you can see from your site it would be great.
Comment by Richard Hearne — October 2, 2014 @ 12:54 pm
Get the server logs. They will reveal if there is an issue with the server itself. Could be as simple as needing a reboot..
Comment by Joseph Mas — October 2, 2014 @ 2:04 pm
In nearly every case I have handled it was either the mail server being bombed or SQL server bring the entire http server down for a short periods that were undetectable if browsing the site. But for a bot during a crawl, a hickup can kill the entire crawl. Hope that helps.
Comment by Joseph Mas — October 2, 2014 @ 2:07 pm
Server was rebooted 2 days ago but still getting issue reports from GWT. Network guys also checked and found nothing. Thanks for your suggestions +Joseph Mas. Hopefully John can check if anything obvious from their side.
Comment by Richard Hearne — October 2, 2014 @ 2:48 pm
+Richard Hearne, hope you find the issue. Please share your findings if you discover something other than the server
Comment by Joseph Mas — October 2, 2014 @ 3:09 pm
Something on your server or upstream of it seems to be blocking our 66.249.78.189. Those are the requests that are timing out. The other IP addresses pretty much work normally. Depending on which IP address is used for which part of the crawl, that's the bumps you're seeing.
Comment by John Mueller — October 2, 2014 @ 9:46 pm
Thanks John for your help. We'll try to figure out where that IP is being blocked and why.
Comment by Richard Hearne — October 3, 2014 @ 4:17 am
FYI +John Mueller it turned out that that IP had been added to an RTBH list by the hosting provider.
Should be fine now. Thanks for your help with this – it's very much appreciated.
It might be a nice future feature for GWT to message more info around this as it's not something that could ever be diagnosed in the logs.
Again thanks John.
Comment by Richard Hearne — October 3, 2014 @ 9:59 am
Great to hear, Richard! Yes, that would have made this more useful here. Luckily we don't see a lot of these cases, but maybe there's a way to aggregate that information without cluttering the UI for those who don't need it. I'll pass that on.
Comment by John Mueller — October 3, 2014 @ 1:36 pm