First request after DB restart bombs

[al4]

Should automatically retry a few times.

[2016-04-13 13:07:24 +0000] [22609] [DEBUG] POST /releases
[2016-04-13 13:07:24 +0000] [22609] [ERROR] Error handling request /releases
Traceback (most recent call last):
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 130, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 171, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/orlo/route_api.py", line 51, in post_releases
    release = create_release(request)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/orlo/util.py", line 49, in create_release
    platforms = append_or_create_platforms(request_platforms)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/orlo/util.py", line 24, in append_or_create_platforms
    platform = query.one()
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2473, in one
    ret = list(self)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
    return self._execute_and_instances(context)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/share/python/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (psycopg2.OperationalError) terminating connection due to administrator command
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
 [SQL: 'SELECT platform.id AS platform_id, platform.name AS platform_name \nFROM platform \nWHERE platform.name = %(name_1)s'] [parameters: {'name_1': u'gumtree'}]

[al4]

http://docs.sqlalchemy.org/en/rel_0_9/core/pooling.html#disconnect-handling-pessimistic

[al4]

Seems to now fail on all requests after DB restart:

2017-07-25 16:16:49,791 [orlo.app] INFO base:log_post_data:22 - GET https://orlo.gt.ecg.so/info/packages/versions
[2017-07-25 16:16:49 +0000] [29084] [ERROR] Error handling request /info/packages/versions
Traceback (most recent call last):
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/orlo/routes/info.py", line 136, in info_package_versions
    result = q.all()
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2703, in all
    return list(self)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2855, in __iter__
    return self._execute_and_instances(context)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1121, in _execute_context
    None, None)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1114, in _execute_context
    conn = self._revalidate_connection()
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 424, in _revalidate_connection
    "Can't reconnect until invalid "
StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT package.name AS package_name, package.version AS package_version \nFROM package JOIN (SELECT package.name AS name, max(package.stime) AS max_stime \nFROM package \nWHERE package.status = %(status_1)s GROUP BY package.name) AS anon_1 ON anon_1.max_stime = package.stime AND anon_1.name = package.name GROUP BY package.name, package.version'] [parameters: [{}]]
[2017-07-25 16:17:32 +0000] [29083] [INFO] Worker exiting (pid: 29083)
[2017-07-25 16:17:32 +0000] [29091] [INFO] Worker exiting (pid: 29091)
[2017-07-25 16:17:32 +0000] [29081] [INFO] Worker exiting (pid: 29081)
[2017-07-25 16:17:32 +0000] [29084] [INFO] Worker exiting (pid: 29084)
ERROR [sqlalchemy.pool.QueuePool] Exception during reset or similar
Traceback (most recent call last):
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 687, in _finalize_fairy
    fairy._reset(pool)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 827, in _reset
    self._reset_agent.rollback()
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1621, in rollback
    self._do_rollback()
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1659, in _do_rollback
    self.connection._rollback_impl()
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 703, in _rollback_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 701, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
  File "/opt/venvs/orlo/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 440, in do_rollback
    dbapi_connection.rollback()
OperationalError: (psycopg2.OperationalError) terminating connection due to administrator command
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.