Web page downloader/scraper...or whatever it's called... - GoFuckYourself.com

FlexxAeon · 07-23-2009, 04:10 PM

Seems like there should be some type of application that can do this I just am not sure what the proper term for it would be (never used one). I need to:

- Input several URL's (to FHG's) to batch
- Program will download the pages and the elements it links to (need to at least get the thumbs and the videos)

Anyone know an app that will accomplish this? Or what an app like this would be called? Preferably a freebie as I don't see myself using it very often after this task gets completed

CYF · 07-23-2009, 04:49 PM

Curl
wget

both are free and will download or mirror a whole website.

cLin · 07-23-2009, 05:24 PM

On the flip side, how do we stop site ripping? =\ I was thinking javascript but that screws the people who don't use it (which I really shouldn't care about but whatever).

raymor · 07-24-2009, 10:06 AM

Quote:

Originally Posted by cLin

On the flip side, how do we stop site ripping? =\ I was thinking javascript but that screws the people who don't use it (which I really shouldn't care about but whatever).

Strongbox has a couple of anti-ripping features - one that's automatic and one
you can activate. Throttlebox is all about preventing ripping.

07-23-2009, 04:10 PM	#1
FlexxAeon Confirmed User Industry Role: Join Date: May 2003 Posts: 3,765	Web page downloader/scraper...or whatever it's called... Seems like there should be some type of application that can do this I just am not sure what the proper term for it would be (never used one). I need to: - Input several URL's (to FHG's) to batch - Program will download the pages and the elements it links to (need to at least get the thumbs and the videos) Anyone know an app that will accomplish this? Or what an app like this would be called? Preferably a freebie as I don't see myself using it very often after this task gets completed __________________ flexx [dot] aeon [at] gmail

07-23-2009, 04:49 PM	#2
CYF Coupon Guru Industry Role: Join Date: Mar 2009 Location: Minneapolis Posts: 10,973	Curl wget both are free and will download or mirror a whole website. __________________ Webmaster Coupons Coupons and discounts for hosting, domains, SSL Certs, and more! AmeriNOC Coupons \| Certified Hosting Coupons \| Hosting Coupons \| Domain Name Coupons

07-23-2009, 05:24 PM	#3
cLin Registered User Industry Role: Join Date: May 2009 Location: Orange County, CA Posts: 94	On the flip side, how do we stop site ripping? =\ I was thinking javascript but that screws the people who don't use it (which I really shouldn't care about but whatever). __________________ Chris The Ex Girlfriend Pics