SQLAlchemy errors preventing installation of tools

Hello. I am running a galaxy in my institute. Lately my galaxy has a lot of hiccups. Installing tools doesn’t work. These will keep cycling on the “new” status. Internally I see the error below.
When I restart Galaxy the issue is fixed, only for the issue to reoccur the next day.

What I can grok from the error is that it has something to do with a connection that explicitly needs to call a rollback. This is not something that can be fixed by an end-user.

Does this issue have any known cause? I work with PostgreSQL as a backend database, so nothing too fancy. The Galaxy version is 21.09.

Thanks!

Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: galaxy.tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager DEBUG 2021-11-26 16:03:06,145 [pN:main.web.1,p:372188,w:1,m:0,tN:uWSGIWorker1Core2] Building repository dependency relationships...
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: galaxy.tool_util.toolbox.base DEBUG 2021-11-26 16:03:06,145 [pN:main.web.1,p:372188,w:1,m:0,tN:uWSGIWorker1Core2] Appending to tool panel section: ARGalaxy Pipelines
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: galaxy.web.framework.decorators ERROR 2021-11-26 16:03:06,156 [pN:main.web.1,p:372188,w:1,m:0,tN:uWSGIWorker1Core2] Uncaught exception in exposed API method:
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: Traceback (most recent call last):
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/server/lib/galaxy/web/framework/decorators.py", line 312, in decorator
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     rval = func(self, trans, *args, **kwargs)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/server/lib/galaxy/webapps/galaxy/api/tool_shed_repositories.py", line 161, in install_repository_revision
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     payload)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/server/lib/galaxy/tool_shed/galaxy_install/install_manager.py", line 714, in install
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     install_options
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/server/lib/galaxy/tool_shed/galaxy_install/install_manager.py", line 808, in __initiate_and_install_repositories
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     return self.install_repositories(tsr_ids, decoded_kwd, reinstalling=False, install_options=install_options)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/server/lib/galaxy/tool_shed/galaxy_install/install_manager.py", line 827, in install_repositories
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     tool_panel_section_keys=tool_panel_section_keys)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/server/lib/galaxy/tool_shed/galaxy_install/install_manager.py", line 1102, in order_components_for_installation
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     repo_info_dicts)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/server/lib/galaxy/tool_shed/util/repository_util.py", line 265, in get_prior_import_or_install_required_dict
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     repository, repository_dependencies = get_repository_and_repository_dependencies_from_repo_info_dict(app, repo_info_dict)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/server/lib/galaxy/tool_shed/util/repository_util.py", line 299, in get_repository_and_repository_dependencies_from_repo_info_dict
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     repository = get_repository_for_dependency_relationship(app, tool_shed, repository_name, repository_owner, changeset_revision)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/server/lib/galaxy/tool_shed/util/repository_util.py", line 380, in get_repository_for_dependency_relationship
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     app.tool_shed_repository_cache.rebuild()
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/server/lib/galaxy/tools/cache.py", line 304, in rebuild
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     defer(ToolShedRepository.metadata), joinedload('tool_dependencies')
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2709, in all
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     return self._iter().all()
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2847, in _iter
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     execution_options={"_sa_orm_load_options": self.load_options},
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     result = conn._execute_20(statement, params or {}, execution_options)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_20
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     return meth(self, args_10style, kwargs_10style, execution_options)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 324, in _execute_on_connection
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     self, multiparams, params, execution_options
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1461, in _execute_clauseelement
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     cache_hit=cache_hit,
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1668, in _execute_context
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     conn = self._revalidate_connection()
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 560, in _revalidate_connection
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     self._invalid_transaction()
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:   File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 540, in _invalid_transaction
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]:     code="8s2b",
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: http://sqlalche.me/e/14/8s2b)

Hi @rhpvorderman,

I don’t have an answer yet, but I’ll look into it. Can you give me an example of a tool that causes this? (or does this happen with any tool?)

Hi @ic4f ,
Thank you for your response. I found the error. The database connection was unstable due to a configuration issue. The postgresql log had entries complaining about broken or aborted connections. The cause was found and now the issue has been resolved.
Thanks again for looking into this!

Best regards,
Ruben

Great! Happy to hear that’s resolved!

Sergey