Hello. I am running a galaxy in my institute. Lately my galaxy has a lot of hiccups. Installing tools doesn’t work. These will keep cycling on the “new” status. Internally I see the error below.
When I restart Galaxy the issue is fixed, only for the issue to reoccur the next day.
What I can grok from the error is that it has something to do with a connection that explicitly needs to call a rollback. This is not something that can be fixed by an end-user.
Does this issue have any known cause? I work with PostgreSQL as a backend database, so nothing too fancy. The Galaxy version is 21.09.
Thanks!
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: galaxy.tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager DEBUG 2021-11-26 16:03:06,145 [pN:main.web.1,p:372188,w:1,m:0,tN:uWSGIWorker1Core2] Building repository dependency relationships...
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: galaxy.tool_util.toolbox.base DEBUG 2021-11-26 16:03:06,145 [pN:main.web.1,p:372188,w:1,m:0,tN:uWSGIWorker1Core2] Appending to tool panel section: ARGalaxy Pipelines
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: galaxy.web.framework.decorators ERROR 2021-11-26 16:03:06,156 [pN:main.web.1,p:372188,w:1,m:0,tN:uWSGIWorker1Core2] Uncaught exception in exposed API method:
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: Traceback (most recent call last):
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/server/lib/galaxy/web/framework/decorators.py", line 312, in decorator
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: rval = func(self, trans, *args, **kwargs)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/server/lib/galaxy/webapps/galaxy/api/tool_shed_repositories.py", line 161, in install_repository_revision
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: payload)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/server/lib/galaxy/tool_shed/galaxy_install/install_manager.py", line 714, in install
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: install_options
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/server/lib/galaxy/tool_shed/galaxy_install/install_manager.py", line 808, in __initiate_and_install_repositories
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: return self.install_repositories(tsr_ids, decoded_kwd, reinstalling=False, install_options=install_options)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/server/lib/galaxy/tool_shed/galaxy_install/install_manager.py", line 827, in install_repositories
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: tool_panel_section_keys=tool_panel_section_keys)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/server/lib/galaxy/tool_shed/galaxy_install/install_manager.py", line 1102, in order_components_for_installation
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: repo_info_dicts)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/server/lib/galaxy/tool_shed/util/repository_util.py", line 265, in get_prior_import_or_install_required_dict
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: repository, repository_dependencies = get_repository_and_repository_dependencies_from_repo_info_dict(app, repo_info_dict)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/server/lib/galaxy/tool_shed/util/repository_util.py", line 299, in get_repository_and_repository_dependencies_from_repo_info_dict
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: repository = get_repository_for_dependency_relationship(app, tool_shed, repository_name, repository_owner, changeset_revision)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/server/lib/galaxy/tool_shed/util/repository_util.py", line 380, in get_repository_for_dependency_relationship
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: app.tool_shed_repository_cache.rebuild()
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/server/lib/galaxy/tools/cache.py", line 304, in rebuild
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: defer(ToolShedRepository.metadata), joinedload('tool_dependencies')
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2709, in all
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: return self._iter().all()
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2847, in _iter
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: execution_options={"_sa_orm_load_options": self.load_options},
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: result = conn._execute_20(statement, params or {}, execution_options)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_20
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: return meth(self, args_10style, kwargs_10style, execution_options)
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 324, in _execute_on_connection
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: self, multiparams, params, execution_options
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1461, in _execute_clauseelement
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: cache_hit=cache_hit,
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1668, in _execute_context
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: conn = self._revalidate_connection()
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 560, in _revalidate_connection
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: self._invalid_transaction()
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: File "/galaxy/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 540, in _invalid_transaction
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: code="8s2b",
Nov 26 16:03:06 my.galaxy.tld uwsgi[372188]: sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: http://sqlalche.me/e/14/8s2b)