Warning with this technique (Google)
 

I'm still an inexperienced webmaster...

I was having good placements on Google for my new site... He was even making good sales!

Suddenly something happened. My website disappeared from search! Today I found out what caused it... My site was totally copied by the proxy service Google! See the link below:

https://www.google.com/#q=%22appspot...1280& bih=663

I believe this was done intentionally by another webmaster and I believe I have solved this problem using my htaccess!

But be warned for other amateurs like me! :thumbsup

Had same happening to me,totally copied site to three different domains.Which is why i have now in my settings protection against all kind of downloaders.

I put it on my htaccess:

RewriteCond %{HTTP_USER_AGENT} AppEngine [NC]
RewriteRule .* - [F]

Let me know if there is better solution...

Wait..what? Can someone explain what this is?

Search for article like this:
http://www.htaccess-guide.com/blocki...-and-bad-bots/
Had one with huge list of those downloaders,but dont know what was exact url.

Sorry if my English is not helpful...

It's fine. Thanks for pointing that out and sorry about your site.

Thanks for the warning :thumbsup What does this code do?

damn sorry to hear that

It blocks access to any client which presents a user agent with the string "AppEngine" in it. But there are a whole load of others you need to block too, and that won't even stop all scrapers. I'd love to know the UA updowner.com is using and block that fucker.

check out google.com/safebrowsing/diagnostic?site=updowner.com/
interesting analysis

The mod rewrite you suggested offer some protection. When I had similar problem a few years ago, here are other things I did in addition to the mod_rewrite:

1. Get strong links to your site so that your site is at least stronger than the copycat.
2. Rewrite some of the contents of your site so that its not a duplicate any more but a different version of the one that was copied.
3. Update you site regularly so that the copycat cannot keep up.
4. use a more aggressive exclusion robots.txt file, allowing ONLY the big 3 crawlers.

Any half decent scraper probably uses a common UA, and possibly a random selection of common UAs. I've seen loads of multiple pages from the same IP use a different UA each time.