Redirecting google bot - GoFuckYourself.com

Hamilton Steele · 01-28-2021, 12:20 PM

Hi I'm working on doing some SEO work for my website. However the front page of said site requires that the end user inputs their age to proceed onto the rest of the site. As Google's crawler can't do that I need to know how I can redirect it past that front page.

In theory I suspect it should be easy since anyone can already do that if they know the appropriate URL, No cookies used. But none of the guides I've found on robots.txt seem to touch upon the subject.

sarettah · 01-28-2021, 12:24 PM

I would include the url i want crawled in my site map.

I would also go into webmaster tools and tell it to spider that exact url.

Google should be able to figure stuff out from there.

I think.

.

Hamilton Steele · 01-28-2021, 12:32 PM

Quote:

Originally Posted by sarettah

I would include the url i want crawled in my site map.

.

Umm what's the syntax for that?

S3X_Jay · 01-28-2021, 12:40 PM

I would suggest detecting that it's Googlebot/Bingbot (look for 'googlebot' or 'bingbot' in the User Agent string) and then defaulting the year/month/date fields with valid values. They should then be able to get to the next page. (They do submit forms if they know values to enter).

BUT if you've got that aggressive of an age check do you really want Google/Bing to index pages beyond that point? It means that traffic will bypass the age check, no?

If you're not familiar with sitemaps google "Google Sitemaps" and you'll find information to get you started. All the pages you want indexed should be in the sitemap.

redwhiteandblue · 01-28-2021, 01:05 PM

Do it in javascript, the regular googlebot doesn't run it.

Although it looks like bingbot does so...hmmmm.

01-28-2021, 12:20 PM	#1
Hamilton Steele Registered User Industry Role: Join Date: Jan 2021 Posts: 85	Redirecting google bot Hi I'm working on doing some SEO work for my website. However the front page of said site requires that the end user inputs their age to proceed onto the rest of the site. As Google's crawler can't do that I need to know how I can redirect it past that front page. In theory I suspect it should be easy since anyone can already do that if they know the appropriate URL, No cookies used. But none of the guides I've found on robots.txt seem to touch upon the subject. __________________ (Pornstar, Producer, Writer and Pimp) http://le-beverley.com/ (Under Construction) Email : HamiltonSteele[at]Protonmail.com - Skype - @Hamilton Steele69 / live: cid.afa877d846dcfad5

01-28-2021, 12:40 PM	#4
S3X_Jay Registered User Industry Role: Join Date: Dec 2020 Location: NYC (Harlem) Posts: 38	I would suggest detecting that it's Googlebot/Bingbot (look for 'googlebot' or 'bingbot' in the User Agent string) and then defaulting the year/month/date fields with valid values. They should then be able to get to the next page. (They do submit forms if they know values to enter). BUT if you've got that aggressive of an age check do you really want Google/Bing to index pages beyond that point? It means that traffic will bypass the age check, no? If you're not familiar with sitemaps google "Google Sitemaps" and you'll find information to get you started. All the pages you want indexed should be in the sitemap. __________________ For the past 12+ years I've focused on building & running gay affiliate sites.

01-28-2021, 01:05 PM	#5
redwhiteandblue Bollocks Industry Role: Join Date: Jun 2007 Location: Bollocks Posts: 2,792	Do it in javascript, the regular googlebot doesn't run it. Although it looks like bingbot does so...hmmmm. __________________ Interserver unmanaged AMD Ryzen servers from $73.00

01-28-2021, 12:24 PM	#2
sarettah see you later, I'm gone Industry Role: Join Date: Oct 2002 Posts: 14,072	I would include the url i want crawled in my site map. I would also go into webmaster tools and tell it to spider that exact url. Google should be able to figure stuff out from there. I think. .