A couple of days ago my site absolutely fell out of the SERPs. I really couldn’t tell what was causing Google to receive 404 errors that Webmaster Console was reporting.

Further Digging

This was really beginning to hurt me so I decided to grab my raw access logs and look to see what was going on:

66.249.65.97 – - [23/Jun/2007:02:59:52 -0400] “GET / HTTP/1.1″ 302 5 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

66.249.65.97 – - [23/Jun/2007:02:59:55 -0400] “GET /wp-mobile.php HTTP/1.1″ 404 20530 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

What’s happening there is Googlebot is requesting my homepage, getting a 302 redirect for /wp-mobile.php, and then a 404 not found for that file. In my stupidity I didn’t copy across the file in question as per the installation instructions (although I’m not sure why the plugin doesn’t simply redirect to the plugin folder?).

So you can see how Google was getting those 404 errors. But my stupidity aside, there is a very nasty flaw in Ruadhan O’Donoghue’s plugin: mobile content is served to search engine robots.

If you serve excerpts for each post on your homepage then you really want the Search Engine bots to see that content. Granted, my own cock-up added to my issues by serving 404′s to the bots, but I think the plugin will need some modification to ensure that regular web-crwalers aren’t getting the minimal content that mobile devices get. For actual post pages this isn’t really an issue, but for the homepage this plugin could really affect your rankings – I for one need to ensure that my homepage is served correctly to the bots.

Here’s a few requests from my log:

74.6.69.105 – - [23/Jun/2007:03:05:04 -0400] “GET /statistics/02-03-2007/social-media-marketing/ HTTP/1.0″ 302 0 “-” “Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)”
74.6.69.105 – - [23/Jun/2007:03:05:24 -0400] “GET /wp-mobile.php HTTP/1.0″ 404 20492 “-” “Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)”

72.30.216.101 – - [23/Jun/2007:03:10:34 -0400] “GET /contact HTTP/1.0″ 302 0 “-” “Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)”
72.30.216.101 – - [23/Jun/2007:03:10:35 -0400] “GET /wp-mobile.php HTTP/1.0″ 404 20495 “-” “Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)”

66.249.65.97 – - [23/Jun/2007:02:59:52 -0400] “GET / HTTP/1.1″ 302 5 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
66.249.65.97 – - [23/Jun/2007:02:59:55 -0400] “GET /wp-mobile.php HTTP/1.1″ 404 20530 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

It appears that MSNbot isn’t affected by this, but both Googlebot and Yahoo!Slurp are served up the mobile equivalent of your blog.

The plugin is taking over the parsing for all those user agents as they accept mobile content. But the fly in the ointment is that the content served up by Ruadhan’s plugin is extremely paired down: the homepage simply includes links to your last 10 posts. I’d say this could spell the kiss of death for your search engine rankings (even if you manage to copy the files across *doh*).

I’ve left a comment on the plugin page over on the .mobi blog, and trackbacked to Michele’s post where I first saw this plugin. Hopefully Ruadhan can come up with a workaround for this issue, as I’m quite sure a mobile plugin will be very useful given that mobile devices are going to appear more and more in your logs going forward.