server experts: varnish crashing and taking my websites down with it
 

Every once in a while I'll have a huge HD spike that takes my websites offline, a "swap death spiral". Varnish seems to be the culprit, as I can manually get the websites back up by restarting memcached, varnish and apache. When varnish restarts it says "shutting down: failed!" so I assume it's the problem.

Anyways short of figuring out what the actual problem is, I'd like to just find a workaround so that I don't have to worry about hours of downtime between crash and me realizing things are down.

I guess I could put something in cron that used lynx to hit my websites and if it didn't get a response it would go ahead and restart the three processes?

Someone mind typing out what I'd put in cron for me to do that, or other ideas perhaps?

First off... How much traffic is causing this to happen?

Is the server running suPHP? This can cause it to tax the fuck out of itself and should only be used on virtual boxes.

Don't do the cron job. You'd just be putting a bandage on the real problem. Consult your server logs and see what they're saying around the time things go to shit.

If you can catch it happening, run top to find what's eating all of the memory.

https://www.varnish-cache.org/trac/wiki/Performance might give you some help too.

You could try something like this also if that Nagios solution doesn't work for you for whatever reason.
http://www.catswhocode.com/blog/how-...rver-using-php

Another solution would be a paid service like this, http://aremysitesup.com.

What OS?

Is it possible you've allocated a little too much memory to one or more things, so once the server starts doing a bit more work and taking more memory for other tasks it starts swapping, causing further degradation as the queue of yet-to-be-serviced client connections starts growing?

Ya think? :1orglaugh

Your host isn't helping you out with that?