Heroku memory quota killing machine

When I was migrating a suite of Exvo apps to Heroku‘s Cedar stack I’ve noticed a very strange behaviour. For two apps during first “run” the main process spun out of control, trying to consume way too much memory. After doing a manual restart everything went back to normal and the problem was basically gone (we haven’t seen it reoccuring).

Here is the log of such an out of control worker:

2012-04-02T11:52:32+00:00 heroku[worker.1]: Process running mem=550M(107.5%)
2012-04-02T11:52:32+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-04-02T11:52:52+00:00 heroku[worker.1]: Process running mem=811M(158.5%)
2012-04-02T11:52:52+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-04-02T11:53:12+00:00 heroku[worker.1]: Process running mem=858M(167.7%)
2012-04-02T11:53:12+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-04-02T11:53:32+00:00 heroku[worker.1]: Process running mem=1099M(214.8%)
2012-04-02T11:53:32+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-04-02T11:53:52+00:00 heroku[worker.1]: Process running mem=1539M(300.6%)
2012-04-02T11:53:52+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-04-02T11:54:12+00:00 heroku[worker.1]: Process running mem=1580M(308.7%)
2012-04-02T11:54:12+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-04-02T11:54:32+00:00 heroku[worker.1]: Process running mem=1709M(333.8%)
2012-04-02T11:54:32+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-04-02T11:54:53+00:00 heroku[worker.1]: Process running mem=2115M(413.1%)
2012-04-02T11:54:53+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-04-02T11:55:13+00:00 heroku[worker.1]: Process running mem=2301M(449.6%)
2012-04-02T11:55:13+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-04-02T11:55:33+00:00 heroku[worker.1]: Process running mem=2302M(449.6%)
2012-04-02T11:55:33+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2012-04-02T11:55:53+00:00 heroku[worker.1]: Process running mem=2747M(536.6%)
2012-04-02T11:55:53+00:00 heroku[worker.1]: Error R15 (Memory quota vastly exceeded)
2012-04-02T11:55:53+00:00 heroku[worker.1]: Stopping process with SIGKILL
2012-04-02T11:55:54+00:00 heroku[worker.1]: Process exited with status 137

This is just a worker, but I’ve observed the same for web processes as well. Be aware that as soon as your application starts serving R14/R15 errors it stops processing any new requests (they simply time out).

Previously: Introduction to Heroku H12 timeouts.