Postgres process fork error on Heroku dev

Question

I'm currently having an issue with my Heroku RoR app which results in it not being able to communicate with the Postgres database. Upon trying to load my app, I get the generic "something went wrong" issue.

Running heroku logs returns the following:

2013-03-28T03:46:01+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2013-03-28T03:46:01+00:00 heroku[web.1]: State changed from starting to up
2013-03-28T03:46:01+00:00 app[web.1]: [2013-03-28 03:46:01] ERROR SignalException: SIGTERM
2013-03-28T03:46:01+00:00 app[web.1]:   /usr/local/lib/ruby/1.9.1/webrick/server.rb:90:in `select'
2013-03-28T03:46:05+00:00 heroku[web.1]: Error R12 (Exit timeout) -> At least one process failed to exit within 10 seconds of SIGTERM
2013-03-28T03:46:05+00:00 heroku[web.1]: Stopping remaining processes with SIGKILL
2013-03-28T03:46:07+00:00 heroku[web.1]: Process exited with status 137
2013-03-28T03:48:19+00:00 app[web.1]: Started GET "/projects" for 0.0.0.0 at 2013-03-28 03:48:19 +0000
2013-03-28T03:48:19+00:00 app[web.1]: PG::Error (could not fork new process for connection: Cannot allocate memory
2013-03-28T03:48:19+00:00 app[web.1]: 
2013-03-28T03:48:19+00:00 app[web.1]: could not fork new process for connection: Cannot allocate memory
2013-03-28T03:48:19+00:00 app[web.1]: 
2013-03-28T03:48:19+00:00 app[web.1]: ):
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/postgresql_adapter.rb:1208:in `initialize'
2013-03-28T03:48:19+00:00 app[web.1]: could not fork new process for connection: Cannot allocate memory
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/postgresql_adapter.rb:1208:in `new'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/postgresql_adapter.rb:326:in `initialize'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/postgresql_adapter.rb:28:in `new'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/postgresql_adapter.rb:1208:in `connect'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/postgresql_adapter.rb:28:in `postgresql_connection'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:315:in `new_connection'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:325:in `checkout_new_connection'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:242:in `loop'
2013-03-28T03:48:19+00:00 app[web.1]:   /usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:239:in `checkout'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:102:in `block in connection'
2013-03-28T03:48:19+00:00 app[web.1]:   /usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:101:in `connection'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:247:in `block (2 levels) in checkout'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:242:in `block in checkout'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_specification.rb:171:in `retrieve_connection'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_specification.rb:145:in `connection'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:410:in `retrieve_connection'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/query_cache.rb:67:in `rescue in call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/query_cache.rb:61:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.11/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.11/lib/active_support/callbacks.rb:405:in `_run__3713961688065884330__call__3295828753336647570__callbacks'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.11/lib/active_support/callbacks.rb:405:in `__run_callback'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.11/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.11/lib/active_support/callbacks.rb:81:in `run_callbacks'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.11/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.11/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.11/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/railties-3.2.11/lib/rails/rack/logger.rb:32:in `call_app'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.11/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/railties-3.2.11/lib/rails/rack/logger.rb:16:in `block in call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.11/lib/active_support/tagged_logging.rb:22:in `tagged'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/railties-3.2.11/lib/rails/rack/logger.rb:16:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.11/lib/action_dispatch/middleware/request_id.rb:22:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.11/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/lock.rb:15:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.11/lib/action_dispatch/middleware/static.rb:62:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/railties-3.2.11/lib/rails/application.rb:223:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/railties-3.2.11/lib/rails/engine.rb:479:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/content_length.rb:14:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   /usr/local/lib/ruby/1.9.1/webrick/httpserver.rb:70:in `run'
2013-03-28T03:48:19+00:00 app[web.1]:   /usr/local/lib/ruby/1.9.1/webrick/httpserver.rb:111:in `service'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/railties-3.2.11/lib/rails/rack/log_tailer.rb:17:in `call'
2013-03-28T03:48:19+00:00 app[web.1]:   vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/handler/webrick.rb:59:in `service'
2013-03-28T03:48:19+00:00 app[web.1]:   /usr/local/lib/ruby/1.9.1/webrick/server.rb:183:in `block in start_thread'

The key part being at the beginning:

2013-03-28T03:48:19+00:00 app[web.1]: PG::Error (could not fork new process for connection: Cannot allocate memory
2013-03-28T03:48:19+00:00 app[web.1]: 
2013-03-28T03:48:19+00:00 app[web.1]: could not fork new process for connection: Cannot allocate memory
2013-03-28T03:48:19+00:00 app[web.1]: 
2013-03-28T03:48:19+00:00 app[web.1]: ):

I found this question, and I tried to limit the amount of returned items, but that doesn't appear to be the issue. I only have two items in my database, both of which have the following model:

class Project < ActiveRecord::Base
  attr_accessible :description, :image, :name, :url
end

Which are text/string fields.

Answer 1

I noticed you posted this roughly 15 mins ago. I have started seeing this error in the last half hour on heroku too (connecting to the dev/free postgres db). I wonder if it's a problem at their end? Their status page doesn't give any hints.

Answer 2

Just so you know, the key part of this error is:

could not fork new process for connection: Cannot allocate memory

This suggests that the host, container, or ulimit context has no more available memory. Creating a new process requires allocating memory for the stack and various other basic data structures. It isn't much memory, so the inability to fork a process suggests that you're totally out of RAM.

On a normal host you would log in (if you could), examine the running processes, kill problematic processes, set memory limits on problem services, etc. This isn't really an option on Heroku, but you can:

• Reduce the number of concurrent connections you are using; and
• Reduce work_mem and maintenance_work_mem for each connection

If you're exceeding the RAM available in your container this will help. If the whole host is out of memory it won't do you any good.

I did some emergency support for a customer who had a similar problem after they upgraded to a bigger Heroku plan. We never got a solid answer as to what happened, but the customer was eventually - after several tries and a couple of days - to create a follower (replica) of their database and then fail over to it. This resolved the issues, suggesting that the problems may have been to do with the container that was hosting that database instance. If you can successfully create a follower, try this. You may have to reduce your connection counts and work_mem etc to be able to do this.

In general, please consider that PostgreSQL will perform best with relatively low numbers of actively working connections and a good connection pool. You'll typically get better throughput with 20 or 30 pooled connections than you will with 200 or 300 and no pooler. Reduce your connection counts until the problem goes away and you'll probably see performance increase too.

(Disclosure: I work for 2ndQuadrant, another PostgreSQL support and services company).