server error when opening certain shared workflows

1

I am using Galaxy 20.09 (last pull January) and found a strange error when trying to open some shared workflows (trace below). The workflows themselves work fine, both execution and loading the editor from the “workflow” menu. But from the “shared workflow” menu you get 500 if you click on it. I cleared the template cache to no avail. This effect only some workflows, other shared workflows on the same instance can be opened. New basic workflow can be opened as well. If I download the problematic workflow and import it the error will occur again. If I import it to another instance of Galaxy, it will also cause server 500 error. One of the failing workflow can be seen exported here: 500.ga · GitHub

trace:

86.49.250.134 - - [10/Mar/2021:15:14:35 +0000] "GET /u/marten/w/mzmlconvert HTTP/1.1" 500 - "https://umsa.cerit-sc.cz/workflows/list_published" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36"
Traceback (most recent call last):
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/web/framework/middleware/error.py", line 153, in __call__
    app_iter = self.application(environ, sr_checker)
  File "/mnt/volume/shared/galaxy/py3_venv/lib/python3.6/site-packages/paste/recursive.py", line 85, in __call__
    return self.application(environ, start_response)
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/web/framework/middleware/statsd.py", line 33, in __call__
    req = self.application(environ, start_response)
  File "/mnt/volume/shared/galaxy/py3_venv/lib/python3.6/site-packages/paste/httpexceptions.py", line 640, in __call__
    return self.application(environ, start_response)
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/web/framework/base.py", line 141, in __call__
    return self.handle_request(environ, start_response)
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/web/framework/base.py", line 220, in handle_request
    body = method(trans, **kwargs)
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/webapps/galaxy/controllers/workflow.py", line 230, in display_by_username_and_slug
    return self._display(trans, stored_workflow)
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/webapps/galaxy/controllers/workflow.py", line 272, in _display
    user_is_owner=user_is_owner)
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/webapps/base/webapp.py", line 972, in fill_template_mako
    return template.render(**data)
  File "/mnt/volume/shared/galaxy/py3_venv/lib/python3.6/site-packages/mako/template.py", line 476, in render
    return runtime._render(self, self.callable_, args, data)
  File "/mnt/volume/shared/galaxy/py3_venv/lib/python3.6/site-packages/mako/runtime.py", line 883, in _render
    **_kwargs_for_callable(callable_, data)
  File "/mnt/volume/shared/galaxy/py3_venv/lib/python3.6/site-packages/mako/runtime.py", line 920, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/mnt/volume/shared/galaxy/py3_venv/lib/python3.6/site-packages/mako/runtime.py", line 947, in _exec_template
    callable_(context, *args, **kwargs)
  File "/mnt/volume/shared/galaxy/var/cache/template_cache/base/base_panels.mako.py", line 124, in render_body
  File "/mnt/volume/shared/galaxy/var/cache/template_cache/display_base.mako.py", line 234, in render_center_panel
  File "/mnt/volume/shared/galaxy/var/cache/template_cache/display_base.mako.py", line 269, in render_render_content
  File "/mnt/volume/shared/galaxy/var/cache/template_cache/workflow/display.mako.py", line 256, in render_render_item
  File "/mnt/volume/shared/galaxy/var/cache/template_cache/workflow/display.mako.py", line 227, in do_inputs
  File "/mnt/volume/shared/galaxy/var/cache/template_cache/workflow/display.mako.py", line 134, in render_do_inputs
KeyError: 'input_files'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/web/framework/middleware/batch.py", line 80, in __call__
    return self.application(environ, start_response)
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/web/framework/middleware/request_id.py", line 15, in __call__
    return self.app(environ, start_response)
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/web/framework/middleware/xforwardedhost.py", line 23, in __call__
    return self.app(environ, start_response)
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/web/framework/middleware/translogger.py", line 70, in __call__
    return self.application(environ, replacement_start_response)
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/web/framework/middleware/error.py", line 163, in __call__
    exc_info)
  File "/mnt/volume/shared/galaxy/server/lib/galaxy/web/framework/middleware/translogger.py", line 69, in replacement_start_response
    return start_response(status, headers, exc_info)
SystemError: <built-in function uwsgi_spit> returned a result with an error set

another workflow has the same trace except the middle part is:

  File "/mnt/volume/shared/galaxy/var/cache/template_cache/workflow/display.mako.py", line 134, in render_do_inputs
KeyError: 'input_smiles'

but there is no input_smiles value when looking at the .ga file

2

I found a suspicious thing - the workflow runs the tool galaxyp/thermo_raw_file_converter/thermo_raw_file_converter/1.3.2+galaxy0 however the downloaded file has version 1.2.3+galaxy0 instead. Moreover I am unable to change the version of this tool, no matter what I select in the workflow editor it will always download as 1.2.3.

This would explain the input_files mismatch, since that was input name of the previous tool version.

3

So if I manually change the version of the tool in the .ga file to 1.3.2 and then import - everything works well. Moreover the original (broken) .ga file contains the following, which includes the newest changeset corresponding for the 1.3.2 version…

            "tool_shed_repository": {
                "changeset_revision": "92ac8e086317",
                "name": "thermo_raw_file_converter",
                "owner": "galaxyp",
                "tool_shed": "toolshed.g2.bx.psu.edu"
            },
4

Sounds like a variation of [20.09] Incorrect tool version saved in workflow · Issue #10914 · galaxyproject/galaxy · GitHub?

5

Looks like it, I will try tomorrow, thank you!

6

This was the right direction. To add to it you also need to delete all problematic nodes from the workflow and re-add them. :confused: