server experts: varnish crashing and taking my websites down with it - GoFuckYourself.com

Socks · 08-08-2011, 12:40 AM

Every once in a while I'll have a huge HD spike that takes my websites offline, a "swap death spiral". Varnish seems to be the culprit, as I can manually get the websites back up by restarting memcached, varnish and apache. When varnish restarts it says "shutting down: failed!" so I assume it's the problem.

Anyways short of figuring out what the actual problem is, I'd like to just find a workaround so that I don't have to worry about hours of downtime between crash and me realizing things are down.

I guess I could put something in cron that used lynx to hit my websites and if it didn't get a response it would go ahead and restart the three processes?

Someone mind typing out what I'd put in cron for me to do that, or other ideas perhaps?

darksoul · 08-08-2011, 01:29 AM

http://www.nagios.org/

V_RocKs · 08-08-2011, 01:35 AM

First off... How much traffic is causing this to happen?

Is the server running suPHP? This can cause it to tax the fuck out of itself and should only be used on virtual boxes.

Babaganoosh · 08-08-2011, 06:10 AM

Don't do the cron job. You'd just be putting a bandage on the real problem. Consult your server logs and see what they're saying around the time things go to shit.

If you can catch it happening, run top to find what's eating all of the memory.

https://www.varnish-cache.org/trac/wiki/Performance might give you some help too.

barcodes · 08-08-2011, 06:32 AM

You could try something like this also if that Nagios solution doesn't work for you for whatever reason.
http://www.catswhocode.com/blog/how-...rver-using-php

Another solution would be a paid service like this, http://aremysitesup.com.

rowan · 08-08-2011, 07:23 AM

What OS?

Is it possible you've allocated a little too much memory to one or more things, so once the server starts doing a bit more work and taking more memory for other tasks it starts swapping, causing further degradation as the queue of yet-to-be-serviced client connections starts growing?

Babaganoosh · 08-08-2011, 12:05 PM

Quote:

Originally Posted by rowan

What OS?

Is it possible you've allocated a little too much memory to one or more things, so once the server starts doing a bit more work and taking more memory for other tasks it starts swapping, causing further degradation as the queue of yet-to-be-serviced client connections starts growing?

Ya think?

critical · 08-08-2011, 01:32 PM

Your host isn't helping you out with that?

08-08-2011, 12:40 AM	#1
Socks Confirmed User Industry Role: Join Date: May 2002 Location: Toronto Posts: 8,475	server experts: varnish crashing and taking my websites down with it Every once in a while I'll have a huge HD spike that takes my websites offline, a "swap death spiral". Varnish seems to be the culprit, as I can manually get the websites back up by restarting memcached, varnish and apache. When varnish restarts it says "shutting down: failed!" so I assume it's the problem. Anyways short of figuring out what the actual problem is, I'd like to just find a workaround so that I don't have to worry about hours of downtime between crash and me realizing things are down. I guess I could put something in cron that used lynx to hit my websites and if it didn't get a response it would go ahead and restart the three processes? Someone mind typing out what I'd put in cron for me to do that, or other ideas perhaps?

08-08-2011, 01:29 AM	#2
darksoul Confirmed User Join Date: Apr 2002 Location: /root/ Posts: 4,997	http://www.nagios.org/ __________________ 1337 5y54\|)m1n: 157717888 BM-2cUBw4B2fgiYAfjkE7JvWaJMiUXD96n9tN Cambooth

08-08-2011, 06:32 AM	#5
barcodes Confirmed User Industry Role: Join Date: Mar 2011 Location: Florida Posts: 2,040	You could try something like this also if that Nagios solution doesn't work for you for whatever reason. http://www.catswhocode.com/blog/how-...rver-using-php Another solution would be a paid service like this, http://aremysitesup.com. Last edited by barcodes; 08-08-2011 at 06:39 AM..

08-08-2011, 01:35 AM	#3
V_RocKs Damn Right I Kiss Ass! Industry Role: Join Date: Dec 2003 Location: Cowtown, USA Posts: 32,409	First off... How much traffic is causing this to happen? Is the server running suPHP? This can cause it to tax the fuck out of itself and should only be used on virtual boxes.

08-08-2011, 06:10 AM	#4
Babaganoosh ♥♥♥ Likes Hugs ♥♥♥ Industry Role: Join Date: Nov 2001 Location: /home Posts: 15,841	Don't do the cron job. You'd just be putting a bandage on the real problem. Consult your server logs and see what they're saying around the time things go to shit. If you can catch it happening, run top to find what's eating all of the memory. https://www.varnish-cache.org/trac/wiki/Performance might give you some help too.

08-08-2011, 07:23 AM	#6
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	What OS? Is it possible you've allocated a little too much memory to one or more things, so once the server starts doing a bit more work and taking more memory for other tasks it starts swapping, causing further degradation as the queue of yet-to-be-serviced client connections starts growing?

08-08-2011, 01:32 PM	#8
critical Confirmed User Join Date: Aug 2009 Posts: 478	Your host isn't helping you out with that?