Quote:
Originally Posted by wankawonk
Like how ridiculous is it that now when I build and operate sites I literally have to make sure there's not too many crawlable pages, or else google will molest my servers to death. And there's literally no way to tell them to stop, other than to forcibly block them.
|
A 429 Too Many Requests error code could help, although you would need to be careful not to overdo it.
According to this page, Google sees 429 as being the same as 503 Service Unavailable.
https://www.seroundtable.com/google-...ode-20410.html