Originally Posted by rowan

Is CAPTCHA an option? I don't mean as the first line of defence (which is annoying for your users); rather, you challenge them only once your heuristic system suspects they may be a bot. There's a small chance of a human going down this road - say someone with cookies disabled, presenting no referer, coming via a proxy etc - but it's not really an issue since once they solve the CAPTCHA they can continue anyway (perhaps a successful CAPTCHA could also give them some positive heuristic score.) The more important thing is the accesses that do *not* solve the CAPTCHA, but just keep repeatedly hitting your challenge page without submitting a correct answer (or anything)... this further reinforces that it's some sort of automated agent accessing your site, not a human using a browser interactively.

I use this on a site which gets scraped to hell and back. I used to log headers and manually find patterns or signatures to block (most were pretty obvious) but an automated possible bot + CAPTCHA confirm system is so much easier.