Best way to copy from server to server? - GoFuckYourself.com

camperjohn64 · 02-17-2011, 10:02 AM

I have a video converter on www.*.com. After converting, it sends the file to img.*.com and want your ideas on how to send the file across. They are on different servers.

Currently, on www.*.com, I use something like this:

Code:

        fsockopen('img1.domain.com', 80);    
        fputs($fp,"POST /image_upload.php HTTP/1.1\r\n");
        etc etc etc...(also sends MD5 and password)

On the recieving end (img1.*.com), I have image_upload.php that takes the POST'ed file and saves it disk...

Code:

        if (MD5 security check) {
        $fp = fopen(IMAGE_PATH . '/'. $filename,"w");
        fwrite($fp,$uploaded);
        fclose($fp);

This has been working fine between several servers, for a couple of years of service, but it's failing on 100MB + video files.

Does anyone think there is a better method of doing this?

JM

Jakez · 02-17-2011, 10:06 AM

Try cURL?

Klen · 02-17-2011, 10:13 AM

Try PHPcli?

CYF · 02-17-2011, 05:07 PM

scp possibly?

rowan · 02-17-2011, 05:21 PM

Looks like you're loading the entire temp upload file into memory in order to save a copy? PHP is probably hitting the configured maximum memory limit for a script. Why don't you just use php's copy() function?

If it's still failing then try increasing the value of upload_max_filesize in php.ini

blackmonsters · 02-17-2011, 05:26 PM

Quote:

Originally Posted by rowan

Looks like you're loading the entire temp upload file into memory in order to save a copy? PHP is probably hitting the configured maximum memory limit for a script. Why don't you just use php's copy() function?

If it's still failing then try increasing the value of upload_max_filesize in php.ini

rowan · 02-17-2011, 05:42 PM

Pre-empting an "I need to load the file into memory in order to md5 it" reply... use md5_file()

Dirty D · 02-17-2011, 05:56 PM

rsync or perhaps wget

camperjohn64 · 02-17-2011, 06:23 PM

Quote:

Originally Posted by Dirty D

rsync or perhaps wget

- It is a push, not a pull, so I can't use wget. The sending server must initiate the call.

Quote:

Originally Posted by rowan

Pre-empting an "I need to load the file into memory in order to md5 it" reply... use md5_file()

- The md5 is just a security measure - it's not against the file itself. Since the destination server is listening for incoming files from multiple servers, I wanted to make sure a hacker could never stumble upon the URL call and send files to the server without some sort of checksum / security.

Quote:

Originally Posted by rowan

Looks like you're loading the entire temp upload file into memory in order to save a copy? PHP is probably hitting the configured maximum memory limit for a script. Why don't you just use php's copy() function?

If it's still failing then try increasing the value of upload_max_filesize in php.ini

- Duh!! Of course. copy is exactly what I neeed (rather than load/save). Thanks! And since it's a copy of a tmp file, I could actually just use move!

Amazing how a different pair of eyes saw that immediately!

CYF · 02-17-2011, 06:36 PM

Quote:

Originally Posted by camperjohn64

- It is a push, not a pull, so I can't use wget. The sending server must initiate the call.

- The md5 is just a security measure - it's not against the file itself. Since the destination server is listening for incoming files from multiple servers, I wanted to make sure a hacker could never stumble upon the URL call and send files to the server without some sort of checksum / security.

check out scp

woj · 02-17-2011, 07:34 PM

send a url to the 2nd server...
then from the 2nd server initiate a GET request with curl to grab the file from the first server...

POSTing large files is usually problematic, there are more than dozen things than can go wrong... GET is 10x more robust...

or just use scp or rsync....

Cyandin · 02-17-2011, 10:12 PM

Quote:

Originally Posted by rowan

Looks like you're loading the entire temp upload file into memory in order to save a copy? PHP is probably hitting the configured maximum memory limit for a script. Why don't you just use php's copy() function?

If it's still failing then try increasing the value of upload_max_filesize in php.ini

</thread>

facialfreak · 02-18-2011, 02:19 AM

you can use the method you are using ... although maybe not the most efficient way, it does the trick ...

just run it from within the SCREEN command (linux/unix) so that even if your connection burps or hits a bump in the road, the process will keep going ...

http://magazine.redhat.com/2007/09/2...to-gnu-screen/

k0nr4d · 02-18-2011, 03:37 AM

Quote:

Originally Posted by camperjohn64

- It is a push, not a pull, so I can't use wget. The sending server must initiate the call.

Do it the other way then:
file_get_contents('http://www.otherserver.com/get.php?filename=whatever');

get.php:
shell_exec("wget http://www.firstserver.com/files/".$_GET[filename]);

Thats it in a nutshell, but add sanitization on the inputs, check the server ip on get.php etc...

PowerCum · 02-18-2011, 03:44 AM

use rsync. It will make the entire md5 process for you and ensure that the content on both servers is always the same.

02-17-2011, 10:02 AM	#1
camperjohn64 Confirmed User Industry Role: Join Date: Feb 2005 Location: Los Angeles Posts: 1,531	Best way to copy from server to server? I have a video converter on www..com. After converting, it sends the file to img..com and want your ideas on how to send the file across. They are on different servers. Currently, on www..com, I use something like this: Code: fsockopen('img1.domain.com', 80); fputs($fp,"POST /image_upload.php HTTP/1.1\r\n"); etc etc etc...(also sends MD5 and password) On the recieving end (img1..com), I have image_upload.php that takes the POST'ed file and saves it disk... Code: if (MD5 security check) { $fp = fopen(IMAGE_PATH . '/'. $filename,"w"); fwrite($fp,$uploaded); fclose($fp); This has been working fine between several servers, for a couple of years of service, but it's failing on 100MB + video files. Does anyone think there is a better method of doing this? JM __________________ www.gimmiegirlproductions.com Last edited by camperjohn64; 02-17-2011 at 10:03 AM..

02-17-2011, 10:06 AM	#2
Jakez Confirmed User Industry Role: Join Date: Jan 2004 Location: oddfuturewolfgangkillthemall!!!!!!! Posts: 5,656	Try cURL? __________________ [email protected] - jakezdumb - 573689400 Killuminati

02-17-2011, 05:07 PM	#4
CYF Coupon Guru Industry Role: Join Date: Mar 2009 Location: Minneapolis Posts: 10,973	scp possibly? __________________ Webmaster Coupons Coupons and discounts for hosting, domains, SSL Certs, and more! AmeriNOC Coupons \| Certified Hosting Coupons \| Hosting Coupons \| Domain Name Coupons

02-17-2011, 05:56 PM	#8
Dirty D Confirmed User Join Date: May 2002 Location: Paying Webmasters Millions Since 1999 Posts: 4,044	rsync or perhaps wget __________________ Dirty D - ICQ #1326843 - $1 Million Dollars of Bonus Money - 8,000+ FHG! Glory Hole Girlz - Crack Whore Confessions - Tampa Bukkake - Slut Wife Training - Fuck a Fan Electricity Play - Porn Video Drive - Theater Sluts - Skunk Riley - Ukraine Amateurs - Strapon Sessions

02-17-2011, 07:34 PM	#11
woj <&(©¿©)&> Industry Role: Join Date: Jul 2002 Location: Chicago Posts: 47,882	send a url to the 2nd server... then from the 2nd server initiate a GET request with curl to grab the file from the first server... POSTing large files is usually problematic, there are more than dozen things than can go wrong... GET is 10x more robust... or just use scp or rsync.... __________________ Custom Software Development, email: woj#at#wojfun#.#com to discuss details or skype: wojl2000 or gchat: wojfun or telegram: wojl2000 Affiliate program tools: Hosted Galleries Manager Banner Manager Video Manager Wordpress Affiliate Plugin Pic/Movie of the Day Fansign Generator Zip Manager

02-17-2011, 10:13 AM	#3
Klen Industry Role: Join Date: Aug 2006 Location: Little Vienna Posts: 32,235	Try PHPcli?

02-17-2011, 05:21 PM	#5
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Looks like you're loading the entire temp upload file into memory in order to save a copy? PHP is probably hitting the configured maximum memory limit for a script. Why don't you just use php's copy() function? If it's still failing then try increasing the value of upload_max_filesize in php.ini

02-17-2011, 05:42 PM	#7
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Pre-empting an "I need to load the file into memory in order to md5 it" reply... use md5_file()

02-18-2011, 02:19 AM	#13
facialfreak Confirmed User Join Date: Feb 2005 Location: Montreal Posts: 3,018	you can use the method you are using ... although maybe not the most efficient way, it does the trick ... just run it from within the SCREEN command (linux/unix) so that even if your connection burps or hits a bump in the road, the process will keep going ... http://magazine.redhat.com/2007/09/2...to-gnu-screen/ __________________ Managed Shared Hosting starting at $4.99/mo Managed VPS starting at $29.99/mo

02-18-2011, 03:44 AM	#15
PowerCum CjOverkill Industry Role: Join Date: Apr 2003 Location: Woldwide Posts: 1,328	use rsync. It will make the entire md5 process for you and ensure that the content on both servers is always the same. __________________ CjOverkill Traffic Trading Script Free, secure and fast traffic trading script. Get your copy now