The scenario goes more or less like this. You have your Virtuozzo powered VPS. You have your RAM limit. You have your mongrels behind nginx plus mysql taking on average 60-70% of your available RAM (quite sensible limit). Now imagine the server gets hammered (not necessairly your VPS). Load goes over 10 (or even over 50). Mongrels stop responding. Queue builds up (inside mongrels), mongrels consume more and more of memory. Load still over 10. Memory limit reached. One of the mongrels is killed by Virtuozzo (too bad if it was the only one). Load still high. At some point other mongrels stop accepting new requests (queue limit reached?) and when load goes down they are unable to process the built up queue. Effect: your website is returning a 500 error code and you have to manually restart the mongrels (they are hanged up). I’ve been seeing similar behaviour too often lately…
Possible solutions: xen (does not kill your children), passenger (spawns new children whenever needed), haproxy (prolongs the life of your children), god (brings dead children back to life).
I’m going with passenger for now, thinking about moving to xen in the future.
