How to map / get idea of a large site? - GoFuckYourself.com

AdultSites · 02-13-2015, 03:46 AM

There is a website that I would need to map, get directory structure, and it is around 1,000,000 pages big. I have no access to the server, or cms. Not all the pages show in Google, and other search engines, and even if they would, it would not be too much help.

I probably need a sitemap, but from what I remember, it is kind of hard to make a sitemap for a big page like that. What would be the best way to work on that / get it done? Something free would be good too. I need website structure, and all urls.

Thanks.

clickity click · 02-13-2015, 12:18 PM

Just use something like httrack which will make a directory on your PC with all the files. If you don't need images you can disable scanning for them.
Most sites have a sitemap as well. Usually something like site.com/sitemap.xml

Might be worth checking first.

Harmon · 02-13-2015, 12:22 PM

So basically, and let me get this straight... you are 2 posts deep into this forum and you are trying to copy somebody elses business?

You'll fit right in.

The Porn Nerd · 02-13-2015, 12:27 PM

Quote:

Originally Posted by Harmon

So basically, and let me get this straight... you are 2 posts deep into this forum and you are trying to copy somebody elses business?

You'll fit right in.

sandman! · 02-13-2015, 01:11 PM

Quote:

Originally Posted by Harmon

So basically, and let me get this straight... you are 2 posts deep into this forum and you are trying to copy somebody elses business?

You'll fit right in.

Bladewire · 02-13-2015, 03:41 PM

Quote:

Originally Posted by Harmon

So basically, and let me get this straight... you are 2 posts deep into this forum and you are trying to copy somebody elses business?

You'll fit right in.

Quote:

Originally Posted by The Porn Nerd

Quote:

Originally Posted by sandman!

AdultKing · 02-13-2015, 03:49 PM

Quote:

Originally Posted by Harmon

So basically, and let me get this straight... you are 2 posts deep into this forum and you are trying to copy somebody elses business?

You'll fit right in.

This forum only attracts assholes now.

Barry-xlovecam · 02-13-2015, 05:48 PM

learn to hack :P

Quote:

ssh user@target
user@target password:

omitted

scp user@target:

omitted

ssh again

user@target:~$/path to document route/$ rm remote-tree.csv
user@target:~$ cd~
user@target:~$ nano .bash_history
You want root so you can clean your connections from the system logs
su {or sudo}
??? look it up l4m3r :P
nano pico vi whatever ...
delete your tracks ^x save
^d

And don't use your mother's IP

epitome · 02-13-2015, 05:53 PM

How the fuck was the name AdultSites not registered on GFY until 2015?

Seems like such an obvious choice.

freecartoonporn · 02-13-2015, 11:34 PM

are you scraping Wikipedia ?

post the url so i can give some input,

AdultSites · 02-14-2015, 04:28 AM

Quote:

Originally Posted by clickity click

Just use something like httrack which will make a directory on your PC with all the files. If you don't need images you can disable scanning for them.
Most sites have a sitemap as well. Usually something like site.com/sitemap.xml

Might be worth checking first.

This program does not seem to be perfect, but I may be doing it wrong too. I will look at the settings, and see what I can make of it. It looks like Teleport Pro, and other download managing programs. They dont seem to be 100% good. They should just work, but there seem to be a lot of bs with that. In the end I end up checking on these programs a lot, instead of getting everything done in the background. Again, I may be doing it wrong, as I did not pay too much attention to this, I was just testing.

clickity click · 02-14-2015, 05:55 AM

what exactly are you trying to do. What it the site. You do know you can't just copy a site and have it running unless it's HTML only.

Roald · 02-14-2015, 06:55 AM

Quote:

Originally Posted by AdultKing

This forum only attracts assholes now.

this forum always attracted assholes. nothing new really.

02-13-2015, 03:46 AM	#1
AdultSites Confirmed User Industry Role: Join Date: Feb 2015 Posts: 203	How to map / get idea of a large site? There is a website that I would need to map, get directory structure, and it is around 1,000,000 pages big. I have no access to the server, or cms. Not all the pages show in Google, and other search engines, and even if they would, it would not be too much help. I probably need a sitemap, but from what I remember, it is kind of hard to make a sitemap for a big page like that. What would be the best way to work on that / get it done? Something free would be good too. I need website structure, and all urls. Thanks.

02-13-2015, 11:34 PM	#10
freecartoonporn Confirmed User Industry Role: Join Date: Jan 2012 Location: NC Posts: 7,683	are you scraping Wikipedia ? post the url so i can give some input, __________________ SSD Cloud Server, VPS Server, Simple Cloud Hosting \| DigitalOcean

02-13-2015, 12:18 PM	#2
clickity click So Fecking Bummed Industry Role: Join Date: Aug 2014 Posts: 3,682	Just use something like httrack which will make a directory on your PC with all the files. If you don't need images you can disable scanning for them. Most sites have a sitemap as well. Usually something like site.com/sitemap.xml Might be worth checking first.

02-13-2015, 12:22 PM	#3
Harmon ( ͡ʘ╭͜ʖ╮͡ʘ) Industry Role: Join Date: Mar 2004 Posts: 19,989	So basically, and let me get this straight... you are 2 posts deep into this forum and you are trying to copy somebody elses business? You'll fit right in.

02-13-2015, 05:53 PM	#9
epitome So Fucking Lame Industry Role: Join Date: Jun 2009 Location: St. Petersburg, FL Posts: 12,158	How the fuck was the name AdultSites not registered on GFY until 2015? Seems like such an obvious choice.

02-14-2015, 05:55 AM	#12
clickity click So Fecking Bummed Industry Role: Join Date: Aug 2014 Posts: 3,682	what exactly are you trying to do. What it the site. You do know you can't just copy a site and have it running unless it's HTML only.