How Do You Stop Fucking Baido Indexing Sites?.. - GoFuckYourself.com

EddyTheDog · 03-22-2014, 10:13 AM

It ignores robots.txt - Anyone know what IPs it uses?....

Magnetron · 03-22-2014, 10:31 AM

What do you have against Scott Baio?

WDF · 03-22-2014, 10:35 AM

Don't you like all that CN traffic?

freecartoonporn · 03-22-2014, 10:35 AM

here it is .

Stop Baidu crawler

brassmonkey · 03-22-2014, 10:42 AM

fucking racist!!

bean-aid · 03-22-2014, 12:35 PM

Just have host blacklist china traffic.

ErectMedia · 03-22-2014, 08:51 PM

robots.txt is like asking your neighbor to keep his dog off your lawn, htaccess is like installing an electric fence

rowan · 03-22-2014, 09:39 PM

Quote:

Originally Posted by beaner

Just have host blacklist china traffic.

Make sure they have clue before they do that, and don't end up blacklisting all of Asia Pacific.

My own solution to the problem is to firewall any IP that presents a Baidu user-agent.

fuzebox · 03-22-2014, 09:55 PM

I would never turn down free traffic. If you don't want china leeching your resources, redirect it somewhere useful

rowan · 03-23-2014, 04:23 AM

Quote:

Originally Posted by fuzebox

I would never turn down free traffic. If you don't want china leeching your resources, redirect it somewhere useful

Dunno about the OP but I'm assuming that he's in the same boat as I am - the issue is that the Baidu web spider trawls over the whole site, but never actually sends any (or very little) human traffic.

If your site has a decent number of pages and/or it is dynamically generated then Baidu really is just wasting resources.

Phoenix · 03-23-2014, 05:41 AM

i might be interested in taking all chinese traffic.

FlowerKid · 03-23-2014, 06:12 AM

If baidu spider teaffic is relevant for your server performance, maybe it's time to upgrade.

hdbuilder · 03-23-2014, 08:44 PM

Just put this in your htaccess file in the root of each domain:

# Block bad spiders
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Sosospider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Sogou
RewriteRule ^.* - [F,L]

You can add as many as you want , make sure the lines ends with [NC,OR] and with nothing for the last one

Using it for years and been tested ...

EddyTheDog · 03-24-2014, 12:39 AM

Quote:

Originally Posted by hdbuilder

Just put this in your htaccess file in the root of each domain:

# Block bad spiders
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Sosospider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Sogou
RewriteRule ^.* - [F,L]

You can add as many as you want , make sure the lines ends with [NC,OR] and with nothing for the last one

Using it for years and been tested ...

Thanks - That looks like it will do the trick...

The main reason is that it inflates my traffic to sponsors so much its hard to see what the real conversions are - If you are as into stats as me it is a real pain in the ass...

I am moving towards using GeoIp scripts and sorting traffic that way but it takes time.....

03-22-2014, 10:13 AM	#1
EddyTheDog Just Doing My Own Thing Industry Role: Join Date: Jan 2011 Location: London, Spain, New Zealand, GFY - Not Croydon... Posts: 24,800	How Do You Stop Fucking Baido Indexing Sites?.. It ignores robots.txt - Anyone know what IPs it uses?....

03-22-2014, 10:31 AM	#2
Magnetron Lord High Groundhog Industry Role: Join Date: Jun 2013 Location: Bumblefuck, USA Posts: 1,841	What do you have against Scott Baio? __________________ . Approach the mirrored reflection saying his name three times : "Butcher .... Butcher .... Butcher ....." and wait to see if this Bogeyman urban legend manifests in the background, looming over shoulder While your neighbors were busy killing off everyone in the neighborhood with your own butcher knife in hand concealed behind your back you stood for ever before the window saying nothing

03-22-2014, 10:35 AM	#3
WDF Confirmed User Industry Role: Join Date: Jan 2013 Location: Nashville,TN. Music City U.S.A. Posts: 2,248	Don't you like all that CN traffic? __________________ Please HELP

03-22-2014, 10:35 AM	#4
freecartoonporn Confirmed User Industry Role: Join Date: Jan 2012 Location: NC Posts: 7,683	here it is . Stop Baidu crawler __________________ SSD Cloud Server, VPS Server, Simple Cloud Hosting \| DigitalOcean

03-22-2014, 10:42 AM	#5
brassmonkey Pay It Forward Industry Role: Join Date: Sep 2005 Location: Yo Mama House Posts: 76,429	fucking racist!! __________________ TRUMP 2024 KEKAW!!! - Support The Laken Riley Act!!! END DACA - SUPPORT AZ HCR 2060 52R - email: brassballz-at-techie.com

03-22-2014, 12:35 PM	#6
bean-aid So Fucking Banned Industry Role: Join Date: Jun 2011 Location: the land of woke sleuths Posts: 16,493	Just have host blacklist china traffic.

03-22-2014, 08:51 PM	#7
ErectMedia Confirmed Chicago Pimp Industry Role: Join Date: Aug 2004 Location: Chicago Posts: 7,100	robots.txt is like asking your neighbor to keep his dog off your lawn, htaccess is like installing an electric fence __________________ Domain Name Registration Web3 Domains

03-22-2014, 09:55 PM	#9
fuzebox making it rain Industry Role: Join Date: Oct 2003 Location: seattle Posts: 21,817	I would never turn down free traffic. If you don't want china leeching your resources, redirect it somewhere useful

03-23-2014, 05:41 AM	#11
Phoenix BACON BACON BACON Industry Role: Join Date: Nov 2002 Location: Poems everybody, the laddie fancies himself a poet Posts: 35,457	i might be interested in taking all chinese traffic. __________________ Skype Phoenixskype1 Telegram PhoenixBrad https://quantads.io

03-23-2014, 06:12 AM	#12
FlowerKid Confirmed User Join Date: Sep 2005 Location: Zurich Posts: 1,045	If baidu spider teaffic is relevant for your server performance, maybe it's time to upgrade. __________________

03-23-2014, 08:44 PM	#13
hdbuilder Confirmed User Industry Role: Join Date: Jun 2012 Location: Canada Posts: 1,338	Just put this in your htaccess file in the root of each domain: # Block bad spiders RewriteEngine On RewriteCond %{HTTP_USER_AGENT} Sosospider [NC,OR] RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC,OR] RewriteCond %{HTTP_USER_AGENT} Sogou RewriteRule ^.* - [F,L] You can add as many as you want , make sure the lines ends with [NC,OR] and with nothing for the last one Using it for years and been tested ...