Quote:
Originally Posted by Klen
I had once also agreggator tube server taken down by semrush bot. All those crawling bots are really badly configured.
|
I've had problems with Semrush too. They don't respect a Disallow directive when there's a redirect.
So if url1 redirects to disallowed url2, they'll still load url2... even though robots.txt asks them not to.
They helpfully suggested that I should just disallow url1 as well as url2.