GoFuckYourself.com - Adult Webmaster Forum - Business Best software that finds and filter bots ?

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - Business Best software that finds and filter bots ? (https://gfy.com/showthread.php?t=1167712)

Best software that finds and filter bots ?

What are the best software available to find and filter bot traffic ?
Thanks a lot

bump for software that clean traffic

Here is a video describing my current state
https://www.youtube.com/watch?v=m_uWS6K-VF8

Quote:

Originally Posted by Aka_Bluey (Post 20489630)

Are you looking for a trade script?

Trade expert
Trade Expert - Free Traffic Trade Script - Trading Script - Traffic Trade - Trader Script

No I am looking more for something that detects fraudulent traffic off the bat. A script that can be used by anyone to detect patterns of bots based on an algorithm of logic. The same logic used by analysts when analyzing the traffic but automated.

What type of bots are you trying to prevent? Search engines, spiders, scrapers? Checking the host agent is the easiest but it is easy to fake. You can put some JS in your page that makes an outgoing request that tells you they are not a bot. That might block some. But a lot of bots now just use real browsers so that won't work. The next best approach is some kind of heuristics based approach that involves machine learning. That is really hard to determine and implement. It is also very specific to your site and will need a lot of fine tuning. Another simple answer is to just rate limit the number of requests per IP.

Quote:

Originally Posted by brandonstills (Post 20489684)

The next best approach is some kind of heuristics based approach that involves machine learning. That is really hard to determine and implement. It is also very specific to your site and will need a lot of fine tuning. Another simple answer is to just rate limit the number of requests per IP.

Brandonstills thanks for your smart answer.
A heuristic approach is exactly what I had in mind.

An algorithm to which you can feed a site's average KPIs to be used as the control group. Ideally set them as a range MIN MAX.

Example (not exhaustive) Takes into account #of requests by IP (vs Human Request)
AND User Agent
AND BR% range
AND Page/Sess
AND TOS
AND USER FLOW (1st/ 2nd/ nth interaction compared to the MIN MAX users habits from the same traffic type)
Etc.

It is a good project. The one that makes it can sell it to many, many webmasters/ traffic brokers..

Quote:

Originally Posted by PornoPlopedia (Post 20489698)

That could work. The difference between a bot and a human would be that a bot would probably go through things faster. It might also go through the navigation slightly differently.

Are you trying to protect against people scraping from the site or just saving bandwidth? Even with IP rate limiting though, if they really wanted to they could just spin up 1000 different instances on Amazon Web Services and it would come from 1000 different IPs.

It all depends on how badly they want to crawl your site.

What's the nature of what you are protecting and what kind of threat are you trying to block? I might be able to give you a better answer if there is a more concrete example.

I had some friends who said the liked Bad Behaviour but I've never tried it myself;
Bad Behavior / Bad Behaviour | The Web's premier link spam killer.

Quote:

Originally Posted by brandonstills (Post 20489765)

What's the nature of what you are protecting and what kind of threat are you trying to block? I might be able to give you a better answer if there is a more concrete example.

It can have many functionalities
Qualifying Traffic by pretty much anyone when buying skimmed (and some new plugs) Monitoring the quality of that same traffic over time. Bcs you may receive a fair quality at first but then they can start sending fake clicks later.
Basically, need to be proactive instead of removing fake traffic from the stats system.
Thanks

Quote:

Originally Posted by Paz (Post 20489789)

I had some friends who said the liked Bad Behaviour but I've never tried it myself;
Bad Behavior / Bad Behaviour | The Web's premier link spam killer.

Thanks, I will look at this later.

Is CAPTCHA an option? I don't mean as the first line of defence (which is annoying for your users); rather, you challenge them only once your heuristic system suspects they may be a bot. There's a small chance of a human going down this road - say someone with cookies disabled, presenting no referer, coming via a proxy etc - but it's not really an issue since once they solve the CAPTCHA they can continue anyway (perhaps a successful CAPTCHA could also give them some positive heuristic score.) The more important thing is the accesses that do *not* solve the CAPTCHA, but just keep repeatedly hitting your challenge page without submitting a correct answer (or anything)... this further reinforces that it's some sort of automated agent accessing your site, not a human using a browser interactively.

I use this on a site which gets scraped to hell and back. I used to log headers and manually find patterns or signatures to block (most were pretty obvious) but an automated possible bot + CAPTCHA confirm system is so much easier. :thumbsup

Quote:

Originally Posted by rowan (Post 20490186)

Thanks for the tip ! Captcha is not an option tho in this case .

Quote:

Originally Posted by PornoPlopedia (Post 20490092)

You might want to look into Complex Event Processing (CEP) solutions. I might set up some monitoring using something like Riemann - A network monitoring system.

Quote:

Originally Posted by PornoPlopedia (Post 20490232)

Thanks for the tip ! Captcha is not an option tho in this case .

Block Bots | Bot Detection Distil Networks

Quote:

Originally Posted by PornoPlopedia (Post 20488491)

What are the best software available to find and filter bot traffic ?
Thanks a lot

.htaccess can stop a lot of the low life bots,scrapers, etc. You get immediate results, though this took me a long time to test via trial and error. Here's a snippet from one of mine. A lot is cut out, this is only a snippet, immediately resolved some of my biggest issues. I allow more than the US on my sites, again, this is just a snippet and I've given just a few lines from each section :thumbsup

Code:

Options -indexes

ServerSignature Off

Options +FollowSymlinks

GeoIPEnable On

RewriteEngine On



SetEnvIf GEOIP_COUNTRY_CODE US AllowCountry

SetEnvIf GEOIP_COUNTRY_CODE CN AllowCountry

Allow from env=AllowCountry



# FORWARD CHINA TO A FITTING YOUTUBE VIDEO

<IfModule mod_rewrite.c>

RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^(CN)$

RewriteRule ^(.*)$ https://www.youtube.com/watch?v=SLMJpHihykI$1 [L] 

</IfModule>



# NO PROXIES FORWARDERS BLANK REFS ETC

RewriteCond %{HTTP:VIA}  !^$ [OR] 

RewriteCond %{HTTP:FORWARDED}  !^$ [OR] 

RewriteCond %{HTTP:USERAGENT_VIA}  !^$ [OR] 

RewriteCond %{HTTP:X_FORWARDED_FOR}  !^$ [OR] 

RewriteCond %{HTTP:PROXY_CONNECTION}  !^$ [OR] 

RewriteCond %{HTTP:XPROXY_CONNECTION}  !^$ [OR] 

RewriteCond %{HTTP:HTTP_PC_REMOTE_ADDR} !^$ [OR] 

RewriteCond %{HTTP:HTTP_CLIENT_IP}  !^$ 

# ISSUE 403 / SERVE ERRORDOCUMENT

RewriteRule ^(.*)$ - [F]



RewriteCond %{REQUEST_METHOD} POST

RewriteCond %{HTTP_REFERER} !.*YOURWEBSITE.COM* [OR]

RewriteCond %{HTTP_USER_AGENT} ^$

RewriteRule (.*) http://%{REMOTE_ADDR}/$ [R=301,L]



# STARTS WITH WEB

RewriteCond %{HTTP_USER_AGENT} ^web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack) [NC]

 

# ANYWHERE IN UA -- GREEDY REGEX

RewriteCond %{HTTP_USER_AGENT} ^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$ [NC]

 

RewriteCond %{HTTP_USER_AGENT} ^.*(BlogScope|Butterfly|DCPbot|discoverybot|domain|Ezooms|ImageSearcherFree).*$ [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^.*(ips-agent|linkdex|MJ12|Netcraft|NextGenSearchBot|SISTRIX|Sogou|soso|TweetmemeBot|Unwind|Yandex).*$ [NC]



RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]

RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]

RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]

RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]

RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]

RewriteCond %{HTTP_USER_AGENT} ^Navroad 

# ISSUE 403 / SERVE ERRORDOCUMENT

RewriteRule .* - [F,L]



# IF THE UA STARTS WITH THESE

RewriteCond %{HTTP_USER_AGENT} ^(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon) [NC,OR]

# ISSUE 403 / SERVE ERRORDOCUMENT

RewriteRule .* - [F,L]



SetEnvIfNoCase User-Agent ^$ bad_bot

SetEnvIfNoCase User-Agent "^Download\ Demon" bad_bot

SetEnvIfNoCase User-Agent "^Download\ Devil" bad_bot

SetEnvIfNoCase User-Agent "^Download\ Wonder" bad_bot

SetEnvIfNoCase User-Agent "^dragonfly" bad_bot

SetEnvIfNoCase User-Agent "^Drip" bad_bot



# Vulnerability Scanners

SetEnvIfNoCase User-Agent "Acunetix" bad_bot

SetEnvIfNoCase User-Agent "FHscan" bad_bot

 

# Aggressive Chinese Search Engine

SetEnvIfNoCase User-Agent "Baiduspider" bad_bot

 

# Aggressive Russian Search Engine

SetEnvIfNoCase User-Agent "Yandex" bad_bot



<Limit GET POST HEAD>

Order Allow,Deny

Allow from all

Deny from env=bad_bot

</Limit>

Quote:

Originally Posted by Bladewire (Post 20491192)

Thanks a lot Bladewire ! Appreciated
Just sent a ticket to our developer to ad to our .htaccess

Quote:

Originally Posted by RazorSharpe (Post 20491135)

Block Bots | Bot Detection Distil Networks

Thanks RazorSharpe.
They seem to provide the service. I may go for the free trial.
If / when I do, I will keep you updated with the results

Quote:

Originally Posted by brandonstills (Post 20491046)

You might want to look into Complex Event Processing (CEP) solutions. I might set up some monitoring using something like Riemann - A network monitoring system.

Awesome, thanks brandonstills

Will check those out over the week end and see if we can implement this

Pornoplopedia! I love how you're spanking those girls ;)

block bots at server level,
using firewall rules and rewrite rules.,
by checking user agents , and some other data.