Although last year Google warned that it was not a particularly good idea, they’ve changed their tune and now have developed a way for their bots to fill in certain form and form fields and then crawl the subsequent results so they can be indexed.

At this point, Google is only crawling forms for “a small number of particularly useful sites.” But this promises to open up a boatload of deep web data that has previously been hidden from the search engine bots.

Forms that requires usernames or passwords, or have nofollow, robot.txt, or noindex directives, will be ignored by the bots. Pagerank will not be lost in other pages with the indexing of additional pages subsequent to the filling out of forms.

Last year, Google warned that sites that do not put robots.txt directives in their search results pages would run into some trouble. For example, if you plugged “watches” into Overstock.com, and then searched for “watches” on Google, the search results for Google would show a page that reflected the search results for “watches” on Overstock. Google didn’t like that, but apparently, they are changing their tune.

There are more implications for this coming, including a possible flood of bot-submitted forms that mess up your statistics. One commenter on Search Engine Land suggested inserting a hidden version of your form that is available for the bots. If that form comes in completed, you can be sure it’s not an actual customer and then there’s no need to reply to it.

Thanks to Search Engine Land for the story.

This entry was posted on Sunday, April 13th, 2008 at 4:09 pmand is filed under Search Engine Marketing. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.