Redirecting google bot
 

Hi I'm working on doing some SEO work for my website. However the front page of said site requires that the end user inputs their age to proceed onto the rest of the site. As Google's crawler can't do that I need to know how I can redirect it past that front page.

In theory I suspect it should be easy since anyone can already do that if they know the appropriate URL, No cookies used. But none of the guides I've found on robots.txt seem to touch upon the subject.

I would include the url i want crawled in my site map.

I would also go into webmaster tools and tell it to spider that exact url.

Google should be able to figure stuff out from there.

I think.

.

Umm what's the syntax for that?

I would suggest detecting that it's Googlebot/Bingbot (look for 'googlebot' or 'bingbot' in the User Agent string) and then defaulting the year/month/date fields with valid values. They should then be able to get to the next page. (They do submit forms if they know values to enter).

BUT if you've got that aggressive of an age check do you really want Google/Bing to index pages beyond that point? It means that traffic will bypass the age check, no?

If you're not familiar with sitemaps google "Google Sitemaps" and you'll find information to get you started. All the pages you want indexed should be in the sitemap.

Do it in javascript, the regular googlebot doesn't run it.

Although it looks like bingbot does so...hmmmm.