Workflow submission failed

Hello!

I am trying to install galaxy and our workflow to another server. On the old server this workflow works like a charm. However when i exported this workflow, imported it to the new server and ran it, it said:

Workflow submission failed

The server could not complete the request. Please contact the Galaxy Team if this error persists.

{
    "new_history_name": "h1902_he4722",
    "history_id": null,
    "resource_params": {},
    "replacement_params": {},
    "parameters": {
        "0": {
            "sample": "h1902_he4722",
            "left": {
                "values": [
                    {
                        "hid": 1,
                        "id": "f2db41e1fa331b3e",
                        "keep": false,
..
..

And long enumeration of parameters of all 55 steps, but nothing about actual error. And no task was queued.

By means of deleting some steps i found that the root of all evil is step “AddOrReplaceReadGroup tumor transcriptome” which is described as follows:

    "16": {
        "annotation": "",
        "content_id": "toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_AddOrReplaceReadGroups/2.18.2.1",
        "errors": null,
        "id": 16,
        "input_connections": {
            "inputFile": {
                "id": 13,
                "output_name": "outFile"
            }
        },
        "inputs": [
            {
                "description": "runtime parameter for tool AddOrReplaceReadGroups",
                "name": "read_group_id_conditional"
            }
        ],
        "label": "AddOrReplaceReadGroup  tumor transcriptome",
        "name": "AddOrReplaceReadGroups",
        "outputs": [
            {
                "name": "outFile",
                "type": "bam"
            }
        ],
        "position": {
            "bottom": 678.1381988525391,
            "height": 163.7804718017578,
            "left": -710.7962646484375,
            "right": -576.7962493896484,
            "top": 514.3577270507812,
            "width": 134.00001525878906,
            "x": -710.7962646484375,
            "y": 514.3577270507812
        },
        "post_job_actions": {},
        "tool_id": "toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_AddOrReplaceReadGroups/2.18.2.1",
        "tool_shed_repository": {
            "changeset_revision": "9ffcddf6f9c0",
            "name": "picard",
            "owner": "devteam",
            "tool_shed": "toolshed.g2.bx.psu.edu"
        },
        "tool_state": "{\"CN\": \"\", \"DS\": \"\", \"DT\": \"\", \"PI\": \"\", \"PL\": \"ILLUMINA\", \"PU\": \"run\", \"inputFile\": {\"__class__\": \"ConnectedValue\"}, \"read_group_id_conditional\": {\"do_auto_name\": \"false\", \"__current_case__\": 1, \"ID\": {\"__class__\": \"RuntimeValue\"}}, \"read_group_lb_conditional\": {\"do_auto_name\": \"false\", \"__current_case__\": 1, \"LB\": \"LIB1\"}, \"read_group_sm_conditional\": {\"do_auto_name\": \"false\", \"__current_case__\": 1, \"SM\": \"\"}, \"validation_stringency\": \"LENIENT\", \"__page__\": null, \"__rerun_remap_job_id__\": null}",
        "tool_version": null,
        "type": "tool",
        "uuid": "507b3f9d-a7a3-45bc-9251-161d66e647b4",
        "workflow_outputs": [
            {
                "label": "AddOrReplaceReadGroups on input dataset(s): BAM with replaced/modified readgroups",
                "output_name": "outFile",
                "uuid": "e9fe451f-226d-4f8c-8527-38b5804d15bb"
            }
        ]
    },

I think it can not be caused by absence of picard tools (which had been installed via admin panel but galaxy can not run them anyway, but this is another question) cos other picard tool invocations lead to successful queuing of such task and “picard: command not found” error in the stderr output.

On the other hand, i changed absolutely nothing when exporting and importing the workflow, and both galaxies are 20.09, so it should not be version mismatch.

Can you please tell me what is going on? And how can i defeat this error?

Thanks in advance.

I installed picard tools but the error is still there. Moreover, i can execute (successfully!) this particular step and the workflow without this step, but i can not execute the whole workflow!

Hi @wormball, I think you were unlucky enough to update to a commit that was broken on release 21.01. If you update to the latest commit of release 21.01 you should be able to replace this step. Importantly, do not copy the step in the workflow editor, but add picard_AddOrReplaceReadGroups from the left hand side and set the parameters as necessary.

Thanks, but i am using release 20.09 on both servers!

Has the workflow ever seen a 21.01 server ? If you can download the workflow and put it somewhere I can fix it for you, or you can do this yourself by removing

        "inputs": [
            {
                "description": "runtime parameter for tool AddOrReplaceReadGroups",
                "name": "read_group_id_conditional"
            }
        ],

and the reference to read_group_id_conditional in this step’s tool state entry. (Or using the instruction I posted earlier)

I found that if i specify read group ID and SM, it works, and if i do not specify, it do not works, in both cases regardless of suggested editions. However when i first encountered this error, i specified such fields and still got the error.

I got this error in the old galaxy too. Galaxy’s picard 2.18.2 gave me errors, so i uninstalled it at the “manage dependencies” tab and installed 2.20.5 outside the galaxy. And after that when i run picard as “run again” in some history, it runs perfectly, but when i invoke the whole workflow, i get aforementioned error.

It magically worked again (on the old server). But i am not sure why and not sure it will work tomorrow.

When i upload my files and invoke the workflow, it says me in some file selection controls:

the previously selected dataset has been deleted

despite i have not deleted anything in the current history. And says “Workflow submission failed” if i press “run”. However if i “copy” the whole history and invoke the workflow on the “copied” history, it works fine.

Also i made automated workflow like advised here Workflow automation? - #6 by jennaj , and it began to schedule, but only 7 of 60 steps were scheduled, and then it said

Invocation scheduling failed - Galaxy administrator may have additional details in logs.

But when i looked at the “View Error Logs” in the admin panel, i saw “No errors available.”, so previous “Workflow submission failed” errors also left no trace in the logs.

Then i looked at galaxy.log and found many letters there. Here are all error entries for march-april:

galaxy.web.framework.decorators ERROR 2021-03-18 18:16:54,941 [p:484294,w:1,m:0] [uWSGIWorker1Core0] Uncaught exception in exposed API method:
galaxy.jobs DEBUG 2021-03-18 19:05:44,300 [p:484294,w:1,m:0] [LocalRunner.work_thread-2] (1042) setting dataset 1154 state to ERROR
galaxy.jobs DEBUG 2021-03-18 19:05:44,327 [p:484294,w:1,m:0] [LocalRunner.work_thread-0] (1040) setting dataset 1152 state to ERROR
galaxy.jobs DEBUG 2021-03-18 19:07:10,039 [p:484294,w:1,m:0] [LocalRunner.work_thread-0] (1048) setting dataset 1160 state to ERROR
galaxy.jobs.runners ERROR 2021-03-18 19:19:51,740 [p:484294,w:1,m:0] [LocalRunner.work_thread-2] (1049/) Job wrapper finish method failed
galaxy.jobs DEBUG 2021-03-19 20:46:51,586 [p:484294,w:1,m:0] [LocalRunner.work_thread-1] (1105) setting dataset 1224 state to ERROR
galaxy.jobs DEBUG 2021-03-19 23:21:16,758 [p:484294,w:1,m:0] [LocalRunner.work_thread-0] (1111) setting dataset 1234 state to ERROR
galaxy.jobs DEBUG 2021-03-19 23:21:16,766 [p:484294,w:1,m:0] [LocalRunner.work_thread-1] (1057) setting dataset 1172 state to ERROR
galaxy.jobs DEBUG 2021-03-23 14:23:12,842 [p:484294,w:1,m:0] [LocalRunner.work_thread-3] (1132) setting dataset 1257 state to ERROR
galaxy.jobs ERROR 2021-03-23 14:32:17,280 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1043
galaxy.jobs ERROR 2021-03-23 14:32:17,381 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1045
galaxy.jobs ERROR 2021-03-23 14:32:22,933 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1069
galaxy.jobs ERROR 2021-03-23 14:32:23,030 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1120
galaxy.jobs ERROR 2021-03-23 14:32:24,704 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1063
galaxy.jobs ERROR 2021-03-23 14:32:24,850 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1113
galaxy.jobs ERROR 2021-03-23 14:32:25,019 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1115
galaxy.jobs ERROR 2021-03-23 14:32:26,295 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1078
galaxy.jobs ERROR 2021-03-23 14:32:26,630 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1122
galaxy.jobs ERROR 2021-03-23 14:32:27,064 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1123
galaxy.jobs ERROR 2021-03-23 14:32:27,238 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1125
galaxy.jobs ERROR 2021-03-23 14:32:27,419 [p:484294,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1126
galaxy.web.framework.decorators ERROR 2021-03-25 15:22:15,094 [p:44177,w:1,m:0] [uWSGIWorker1Core3] Uncaught exception in exposed API method:
galaxy.web.framework.decorators ERROR 2021-03-25 15:22:23,517 [p:44177,w:1,m:0] [uWSGIWorker1Core0] Uncaught exception in exposed API method:
galaxy.web.framework.decorators ERROR 2021-03-25 15:22:23,805 [p:44177,w:1,m:0] [uWSGIWorker1Core1] Uncaught exception in exposed API method:
galaxy.jobs DEBUG 2021-03-26 19:13:47,828 [p:180146,w:1,m:0] [LocalRunner.work_thread-3] (1208) setting dataset 1339 state to ERROR
galaxy.jobs DEBUG 2021-03-26 19:21:21,702 [p:180146,w:1,m:0] [LocalRunner.work_thread-2] (1211) setting dataset 1342 state to ERROR
galaxy.jobs DEBUG 2021-04-01 15:51:37,634 [p:180146,w:1,m:0] [LocalRunner.work_thread-3] (1440) setting dataset 1598 state to ERROR
galaxy.jobs DEBUG 2021-04-06 13:45:12,989 [p:180146,w:1,m:0] [LocalRunner.work_thread-3] (1590) setting dataset 1763 state to ERROR
galaxy.jobs DEBUG 2021-04-06 13:58:24,889 [p:180146,w:1,m:0] [LocalRunner.work_thread-0] (1592) setting dataset 1765 state to ERROR
galaxy.jobs DEBUG 2021-04-06 19:19:32,792 [p:180146,w:1,m:0] [LocalRunner.work_thread-3] (1594) setting dataset 1767 state to ERROR
galaxy.jobs.runners ERROR 2021-04-06 19:19:36,931 [p:180146,w:1,m:0] [LocalRunner.work_thread-3] (1597/) Job wrapper finish method failed
galaxy.jobs DEBUG 2021-04-06 19:19:41,142 [p:180146,w:1,m:0] [LocalRunner.work_thread-3] (1598) setting dataset 1772 state to ERROR
galaxy.jobs ERROR 2021-04-06 19:19:41,958 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1601
galaxy.jobs ERROR 2021-04-06 19:19:44,578 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1615
galaxy.jobs ERROR 2021-04-06 19:19:45,945 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1627
galaxy.jobs ERROR 2021-04-06 19:19:47,349 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1637
galaxy.jobs ERROR 2021-04-06 19:19:47,772 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1638
galaxy.tools.toolbox.base ERROR 2021-04-06 19:33:57,892 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-06 19:33:57,893 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-06 19:33:57,894 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-06 19:33:57,895 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'melanoma_tools/scripts_hla_la.xml', '[Errno 2] No such file or directory')
galaxy.tools.toolbox.base ERROR 2021-04-06 19:33:57,902 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-06 19:33:57,903 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-06 19:33:57,904 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-07 11:52:47,046 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-07 11:52:47,048 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-07 11:52:47,049 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-07 11:52:47,049 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'melanoma_tools/scripts_hla_la.xml', '[Errno 2] No such file or directory')
galaxy.tools.toolbox.base ERROR 2021-04-07 11:52:47,056 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-07 11:52:47,057 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-07 11:52:47,058 [p:180146,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.jobs ERROR 2021-04-07 13:04:59,058 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1648
galaxy.jobs ERROR 2021-04-07 13:04:59,275 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1649
galaxy.jobs ERROR 2021-04-07 13:04:59,496 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1650
galaxy.jobs ERROR 2021-04-07 13:05:00,754 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1659
galaxy.jobs ERROR 2021-04-07 13:05:01,143 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1660
galaxy.jobs ERROR 2021-04-07 13:05:01,416 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1661
galaxy.jobs ERROR 2021-04-07 13:05:02,765 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1672
galaxy.jobs ERROR 2021-04-07 13:05:02,974 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1673
galaxy.jobs DEBUG 2021-04-07 13:07:00,262 [p:180146,w:1,m:0] [LocalRunner.work_thread-0] (1683) setting dataset 1867 state to ERROR
galaxy.jobs DEBUG 2021-04-07 13:44:40,904 [p:180146,w:1,m:0] [LocalRunner.work_thread-2] (1721) setting dataset 1910 state to ERROR
galaxy.jobs DEBUG 2021-04-07 13:44:41,303 [p:180146,w:1,m:0] [LocalRunner.work_thread-3] (1722) setting dataset 1911 state to ERROR
galaxy.jobs ERROR 2021-04-07 13:44:51,501 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1729
galaxy.jobs ERROR 2021-04-07 13:44:51,709 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1730
galaxy.jobs ERROR 2021-04-07 13:44:53,034 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1736
galaxy.jobs ERROR 2021-04-07 13:44:54,297 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1748
galaxy.jobs ERROR 2021-04-07 13:44:55,726 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1756
galaxy.jobs ERROR 2021-04-07 13:44:56,257 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1757
galaxy.jobs ERROR 2021-04-07 13:44:57,795 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1761
galaxy.jobs ERROR 2021-04-07 13:44:58,296 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1762
galaxy.jobs ERROR 2021-04-07 13:44:58,882 [p:180146,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1763
galaxy.jobs DEBUG 2021-04-07 14:57:09,115 [p:310870,w:1,m:0] [LocalRunner.work_thread-1] (1770) setting dataset 1965 state to ERROR
galaxy.jobs.runners ERROR 2021-04-07 15:10:17,426 [p:310870,w:1,m:0] [LocalRunner.work_thread-3] (1772/) Job wrapper finish method failed
galaxy.jobs DEBUG 2021-04-07 15:33:59,106 [p:310870,w:1,m:0] [LocalRunner.work_thread-0] (1773) setting dataset 1970 state to ERROR
ERROR conda.core.link:_execute(568): An error occurred while installing package 'conda-forge::_libgcc_mutex-0.1-conda_forge'.
ERROR conda.core.link:_execute(568): An error occurred while installing package 'conda-forge::ncurses-6.2-h58526e2_4'.
ERROR conda.core.link:_execute(568): An error occurred while installing package 'conda-forge::_libgcc_mutex-0.1-conda_forge'.
galaxy.jobs ERROR 2021-04-08 15:53:19,449 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1781
galaxy.jobs.runners ERROR 2021-04-08 15:53:20,554 [p:315295,w:1,m:0] [LocalRunner.work_thread-1] (1780) Failure preparing job
galaxy.jobs ERROR 2021-04-08 15:53:21,096 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1782
galaxy.jobs ERROR 2021-04-08 15:53:21,620 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1783
galaxy.jobs ERROR 2021-04-08 15:53:22,200 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1784
galaxy.jobs ERROR 2021-04-08 15:53:23,566 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1789
galaxy.jobs ERROR 2021-04-08 15:53:23,841 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1790
galaxy.jobs ERROR 2021-04-08 15:53:25,111 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1800
galaxy.jobs ERROR 2021-04-08 15:53:25,339 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1801
galaxy.jobs ERROR 2021-04-08 15:53:25,577 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1802
galaxy.jobs ERROR 2021-04-08 15:53:27,214 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1807
galaxy.jobs ERROR 2021-04-08 15:53:27,455 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1808
galaxy.jobs ERROR 2021-04-08 15:53:28,858 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1817
galaxy.jobs ERROR 2021-04-08 15:53:29,107 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1818
galaxy.jobs ERROR 2021-04-08 16:12:44,256 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1886
galaxy.jobs ERROR 2021-04-08 16:12:45,127 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1887
galaxy.jobs ERROR 2021-04-08 16:12:47,221 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1888
galaxy.jobs ERROR 2021-04-08 16:12:47,971 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1889
galaxy.jobs ERROR 2021-04-08 16:12:50,675 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1837
galaxy.jobs ERROR 2021-04-08 16:12:51,952 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1838
galaxy.jobs ERROR 2021-04-08 16:12:52,820 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1839
galaxy.jobs ERROR 2021-04-08 16:12:55,946 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1899
galaxy.jobs ERROR 2021-04-08 16:13:00,542 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1850
galaxy.jobs ERROR 2021-04-08 16:13:01,122 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1851
galaxy.jobs ERROR 2021-04-08 16:13:01,635 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1907
galaxy.jobs ERROR 2021-04-08 16:13:03,460 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1858
galaxy.jobs ERROR 2021-04-08 16:13:03,886 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1859
galaxy.jobs ERROR 2021-04-08 16:13:05,793 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1870
galaxy.jobs ERROR 2021-04-08 16:13:06,038 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1871
galaxy.jobs ERROR 2021-04-08 16:13:06,241 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1872
galaxy.jobs ERROR 2021-04-08 16:13:06,518 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1873
galaxy.jobs ERROR 2021-04-08 16:13:10,954 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1925
galaxy.jobs ERROR 2021-04-08 16:13:12,082 [p:315295,w:1,m:0] [JobHandlerQueue.monitor_thread] Unable to cleanup job 1927
galaxy.tools.toolbox.base ERROR 2021-04-08 18:00:29,910 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:00:29,912 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:00:29,913 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:00:29,914 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'melanoma_tools/scripts_hla_la.xml', '[Errno 2] No such file or directory')
galaxy.tools.toolbox.base ERROR 2021-04-08 18:00:29,925 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:00:29,940 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:00:29,942 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:12,706 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:12,708 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:12,709 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:12,710 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'melanoma_tools/scripts_hla_la.xml', '[Errno 2] No such file or directory')
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:12,733 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:12,734 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:12,736 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:32,510 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:32,511 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:32,512 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:32,513 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'melanoma_tools/scripts_hla_la.xml', '[Errno 2] No such file or directory')
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:32,526 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:32,528 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-08 18:05:32,532 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-09 13:23:00,375 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-09 13:23:00,376 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-09 13:23:00,378 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-09 13:23:00,378 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'melanoma_tools/scripts_hla_la.xml', '[Errno 2] No such file or directory')
galaxy.tools.toolbox.base ERROR 2021-04-09 13:23:00,385 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-09 13:23:00,386 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-09 13:23:00,387 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.workflow.run ERROR 2021-04-09 15:48:56,907 [p:315295,w:1,m:0] [WorkflowRequestMonitor.monitor_thread] Failed to execute scheduled workflow.
galaxy.tools.toolbox.base ERROR 2021-04-09 15:55:26,230 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-09 15:55:26,231 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-09 15:55:26,232 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-09 15:55:26,233 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'melanoma_tools/scripts_hla_la.xml', '[Errno 2] No such file or directory')
galaxy.tools.toolbox.base ERROR 2021-04-09 15:55:26,253 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/codingSnps.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/codingSnps.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-09 15:55:26,254 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'evolution/add_scores.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/add_scores.loc'")
galaxy.tools.toolbox.base ERROR 2021-04-09 15:55:26,256 [p:315295,w:1,m:0] [Thread-1] ("Error reading tool configuration file from path '%s': %s", 'phenotype_association/sift.xml', "[Errno 2] No such file or directory: '/data/galaxy/tool-data/sift_db.loc'")
galaxy.jobs DEBUG 2021-04-09 16:02:21,691 [p:315295,w:1,m:0] [LocalRunner.work_thread-1] (2005) setting dataset 2226 state to ERROR
galaxy.workflow.run ERROR 2021-04-09 16:19:44,894 [p:315295,w:1,m:0] [WorkflowRequestMonitor.monitor_thread] Failed to execute scheduled workflow.

It seems like:

  • When i invoke a workflow with some tool versions not equal to installed, i get the yellow "Some tools are being executed with different versions compared to those available when this workflow was last saved because the other versions are not or no longer available on this Galaxy instance. To upgrade your workflow and dismiss this message simply edit the workflow and re-save it.
    " warning, and then the aforementioned error.

  • When i got this error, every other workflow invocation on the same history will result in the same error, even if i save the workflow again so it contains correct tool versions.

  • However, if i “copy” the history, i have a good chance to schedule workflow correctly from this copy.

  • But if i e. g. schedule the workflow with wrong parameters, purge this history and then invoke the workflow another time, or schedule two workflows in short time, it can result in either the same error or “An error occurred while updating information with the server. Please contact a Galaxy administrator if the problem persists. Ok”. The latter error also manifestates in partial scheduling of the last workflow (e. g. 16 steps of 60).

Do not run a workflow that has tool version changes. The user interface warns you if that is the case. Please edit the workflow and select a version that is installed. If the versions don’t match references to tool steps may be invalid, and scheduling can’t proceed beyond the step that has an invalid reference.
If you are on the latest galaxy release I would also recommend going to the best practices panel in the workflow and make sure you have no disconnected inputs.

When i got this error, every other workflow invocation on the same history will result in the same error, even if i save the workflow again so it contains correct tool versions.

Please make sure you’re actually running the newly saved workflow, and that the tool versions have been properly updated. Older releases of Galaxy did not register that a change occurred if the only change happened when loading the editor, so after you click through the upgrade message you had to make some other change, like moving the position of a step slightly. That’s not necessary anymore, but it might be on 20.09.

  • When i got this error, every other workflow invocation on the same history will result in the same error, even if i save the workflow again so it contains correct tool versions.

There is no state attached to histories that would cause this. If you have any hints on how to reproduce this we can look into it.

  • However, if i “copy” the history, i have a good chance to schedule workflow correctly from this copy.

If you have any instructions on how to reproduce this I’d be very interested. There is no mechanism that I can think of that would cause such behavior.

2 Likes

I encountered this error on 21.01. However i did not perform many actions in short period of time this time, i only deleted an history and (several minutes later) tried to run my workflow.

Workflow submission failed
The server could not complete the request. Please contact the Galaxy Team if this error persists.

{
    "new_history_name": "h1801_e1t1_rmdup",
    "history_id": null,
    "resource_params": {},
    "replacement_params": {},
    "parameters": {
        "0": {
            "input": "h1801"
        },
        "1": {
            "input": {
....

What logs can i post to make the things clear?

Is that 21.01 or 21.05 ?

You should have either an exception in the logs or a non-200 status code. If you can post that that might help

We did finally find a situation where this could happen, [21.05] Do not persist tool state when invoking workflow by mvdbeek · Pull Request #12141 · galaxyproject/galaxy · GitHub
It might help to update to the latest commit on the 21.05 branch.

2 Likes

Is that 21.01 or 21.05 ?

21.01 of course.

There are errors, exceptions and non-200 http codes since 9:00:

galaxy.web.framework.decorators ERROR 2021-06-17 09:01:41,081 [pN:main.web.1,p:428449,w:1,m:0,tN:uWSGIWorker1Core3] Uncaught exception in exposed API method:
binascii.Error: Odd-length string
The above exception was the direct cause of the following exception:
binascii.Error: decoding with 'hex' codec failed (Error: Odd-length string)
During handling of the above exception, another exception occurred:
    raise exceptions.MalformedId(msg, id=str(id))
galaxy.exceptions.MalformedId: Malformed id ( undefined ) specified, unable to decode
172.16.2.70 - - [17/Jun/2021:09:01:41 +0000] "GET /dataset/get_edit?dataset_id=undefined HTTP/1.1" 500 - "http://172.16.2.200:8080/datasets/edit" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 
Safari/537.36"
galaxy.web.framework.decorators ERROR 2021-06-17 09:02:14,437 [pN:main.web.1,p:428449,w:1,m:0,tN:uWSGIWorker1Core2] Uncaught exception in exposed API method:
binascii.Error: Odd-length string
The above exception was the direct cause of the following exception:
binascii.Error: decoding with 'hex' codec failed (Error: Odd-length string)
During handling of the above exception, another exception occurred:
    raise exceptions.MalformedId(msg, id=str(id))
galaxy.exceptions.MalformedId: Malformed id ( undefined ) specified, unable to decode
172.16.2.70 - - [17/Jun/2021:09:02:14 +0000] "GET /dataset/get_edit?dataset_id=undefined HTTP/1.1" 500 - "http://172.16.2.200:8080/datasets/edit" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 
Safari/537.36"
galaxy.web.framework.decorators ERROR 2021-06-17 09:02:44,496 [pN:main.web.1,p:428449,w:1,m:0,tN:uWSGIWorker1Core1] Uncaught exception in exposed API method:
binascii.Error: Odd-length string
The above exception was the direct cause of the following exception:
binascii.Error: decoding with 'hex' codec failed (Error: Odd-length string)
During handling of the above exception, another exception occurred:
    raise exceptions.MalformedId(msg, id=str(id))
galaxy.exceptions.MalformedId: Malformed id ( undefined ) specified, unable to decode
172.16.2.70 - - [17/Jun/2021:09:02:44 +0000] "GET /dataset/get_edit?dataset_id=undefined HTTP/1.1" 500 - "http://172.16.2.200:8080/datasets/edit" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
galaxy.web.framework.decorators ERROR 2021-06-17 09:02:48,604 [pN:main.web.1,p:428449,w:1,m:0,tN:uWSGIWorker1Core2] Uncaught exception in exposed API method:
binascii.Error: Odd-length string
The above exception was the direct cause of the following exception:
binascii.Error: decoding with 'hex' codec failed (Error: Odd-length string)
During handling of the above exception, another exception occurred:
    raise exceptions.MalformedId(msg, id=str(id))
galaxy.exceptions.MalformedId: Malformed id ( undefined ) specified, unable to decode
172.16.2.70 - - [17/Jun/2021:09:02:48 +0000] "GET /dataset/get_edit?dataset_id=undefined HTTP/1.1" 500 - "http://172.16.2.200:8080/datasets/edit" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
galaxy.web.framework.decorators ERROR 2021-06-17 09:02:58,019 [pN:main.web.1,p:428449,w:1,m:0,tN:uWSGIWorker1Core0] Uncaught exception in exposed API method:
binascii.Error: Odd-length string
The above exception was the direct cause of the following exception:
binascii.Error: decoding with 'hex' codec failed (Error: Odd-length string)
During handling of the above exception, another exception occurred:
    raise exceptions.MalformedId(msg, id=str(id))
galaxy.exceptions.MalformedId: Malformed id ( undefined ) specified, unable to decode
172.16.2.70 - - [17/Jun/2021:09:02:57 +0000] "GET /dataset/get_edit?dataset_id=undefined HTTP/1.1" 500 - "http://172.16.2.200:8080/datasets/edit" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 32712/32963] 127.0.0.1 () {42 vars in 661 bytes} [Thu Jun 17 09:05:00 2021] GET /datasets/error => generated 116779 bytes in 50 msecs (HTTP/1.1 200) 8 headers in 393 bytes (1 switches on core 0)
galaxy.web.framework.decorators ERROR 2021-06-17 09:05:02,081 [pN:main.web.1,p:428449,w:1,m:0,tN:uWSGIWorker1Core3] Uncaught exception in exposed API method:
binascii.Error: Odd-length string
The above exception was the direct cause of the following exception:
binascii.Error: decoding with 'hex' codec failed (Error: Odd-length string)
127.0.0.1 - - [17/Jun/2021:09:05:02 +0000] "GET /api/datasets/undefined HTTP/1.1" 500 - "http://localhost:8080/datasets/error" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
127.0.0.1 - - [17/Jun/2021:09:05:02 +0000] "GET /api/users/current HTTP/1.1" 403 - "http://localhost:8080/datasets/error" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
127.0.0.1 - - [17/Jun/2021:09:05:02 +0000] "GET /api/histories/2d9035b3fc152403/contents?limit=500&offset=0&order=hid&v=dev&q=deleted&q=purged&q=visible&qv=False&qv=False&qv=True HTTP/1.1" 403 - "http://localhost:8080/datasets/error" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
127.0.0.1 - - [17/Jun/2021:09:05:02 +0000] "GET /api/histories/2d9035b3fc152403?keys=size%2Cnon_ready_jobs%2Ccontents_active%2Chid_counter HTTP/1.1" 403 - "http://localhost:8080/datasets/error" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
172.16.2.70 - - [17/Jun/2021:10:08:41 +0000] "GET /welcome HTTP/1.1" 302 271 "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [17/Jun/2021:10:14:13 +0000] "GET /welcome HTTP/1.1" 302 271 "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
galaxy.web.framework.decorators ERROR 2021-06-17 10:57:16,532 [pN:main.web.1,p:428449,w:1,m:0,tN:uWSGIWorker1Core1] Uncaught exception in exposed API method:
binascii.Error: Odd-length string
The above exception was the direct cause of the following exception:
binascii.Error: decoding with 'hex' codec failed (Error: Odd-length string)
During handling of the above exception, another exception occurred:
    raise exceptions.MalformedId(msg, id=str(id))
galaxy.exceptions.MalformedId: Malformed id ( undefined ) specified, unable to decode
172.16.2.70 - - [17/Jun/2021:10:57:16 +0000] "GET /dataset/get_edit?dataset_id=undefined HTTP/1.1" 500 - "http://172.16.2.200:8080/datasets/edit" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [17/Jun/2021:12:00:40 +0000] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [17/Jun/2021:12:00:47 +0000] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"

These are not related (this is when you navigate using the browser back button for instance, which misses the dataset id). You’re looking for POST /api/workflows/<workflow_id>/invocations

I obtained it with cat galaxy.log | grep -v ‘HTTP/1.1" 200’ | grep -Pi ‘HTTP/1.1"|excep|error’ . What command should give relevant errors?

It was in the scollbox actually, can you see if there is any additional context around

172.16.2.70 - - [17/Jun/2021:12:00:47 +0000] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
transgen@transgen-4:~/galaxy$ cat galaxy.log | grep -iC 10 '1cd8e2f6b131e891/invocations HTTP/1.1" 400' --color=always 
172.16.2.70 - - [17/Jun/2021:12:00:23 +0000] "POST /api/tools/scripts_optitype/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 33157/33420] 172.16.2.70 () {46 vars in 955 bytes} [Thu Jun 17 12:00:23 2021] POST /api/tools/scripts_optitype/build => generated 6781 bytes in 91 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
172.16.2.70 - - [17/Jun/2021:12:00:23 +0000] "POST /api/tools/bwa_geno/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 33157/33421] 172.16.2.70 () {46 vars in 939 bytes} [Thu Jun 17 12:00:23 2021] POST /api/tools/bwa_geno/build => generated 7746 bytes in 380 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 0)
172.16.2.70 - - [17/Jun/2021:12:00:23 +0000] "POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 33157/33422] 172.16.2.70 () {46 vars in 1063 bytes} [Thu Jun 17 12:00:23 2021] POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build => generated 26789 bytes in 380 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [17/Jun/2021:12:00:26 +0000] "POST /api/tools/star_geno/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 33159/33423] 172.16.2.70 () {46 vars in 941 bytes} [Thu Jun 17 12:00:27 2021] POST /api/tools/star_geno/build => generated 7375 bytes in 327 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
172.16.2.70 - - [17/Jun/2021:12:00:26 +0000] "POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 33159/33424] 172.16.2.70 () {46 vars in 1063 bytes} [Thu Jun 17 12:00:26 2021] POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build => generated 26644 bytes in 339 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 2)
172.16.2.70 - - [17/Jun/2021:12:00:40 +0000] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 33160/33425] 172.16.2.70 () {44 vars in 938 bytes} [Thu Jun 17 12:00:40 2021] POST /api/workflows/1cd8e2f6b131e891/invocations => generated 211 bytes in 71 msecs (HTTP/1.1 400) 3 headers in 148 bytes (1 switches on core 0)
[pid: 428449|app: -1|req: -1/33426] 172.16.2.70 () {38 vars in 887 bytes} [Thu Jun 17 12:00:40 2021] GET /static/dist/assets/b242d750b24c13ee4fea0eda1d0efc69.png => generated 826 bytes in 0 msecs via offload() (HTTP/1.1 200) 3 headers in 111 bytes (0 switches on core 1)
172.16.2.70 - - [17/Jun/2021:12:00:47 +0000] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 33161/33427] 172.16.2.70 () {44 vars in 938 bytes} [Thu Jun 17 12:00:47 2021] POST /api/workflows/1cd8e2f6b131e891/invocations => generated 211 bytes in 72 msecs (HTTP/1.1 400) 3 headers in 148 bytes (1 switches on core 3)
172.16.2.70 - - [17/Jun/2021:15:43:52 +0000] "GET /workflow/editor?id=1cd8e2f6b131e891 HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/list" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 33162/33428] 172.16.2.70 () {42 vars in 964 bytes} [Thu Jun 17 15:43:52 2021] GET /workflow/editor?id=1cd8e2f6b131e891 => generated 142648 bytes in 137 msecs (HTTP/1.1 200) 2 headers in 88 bytes (1 switches on core 2)
[pid: 428449|app: -1|req: -1/33429] 172.16.2.70 () {42 vars in 854 bytes} [Thu Jun 17 15:43:52 2021] GET /favicon.ico => generated 15086 bytes in 0 msecs via offload() (HTTP/1.1 200) 3 headers in 128 bytes (0 switches on core 0)
172.16.2.70 - - [17/Jun/2021:15:43:52 +0000] "GET /api/webhooks HTTP/1.1" 200 - "http://172.16.2.200:8080/workflow/editor?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 33163/33430] 172.16.2.70 () {40 vars in 824 bytes} [Thu Jun 17 15:43:52 2021] GET /api/webhooks => generated 7467 bytes in 13 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [17/Jun/2021:15:43:53 +0000] "GET /api/datatypes/types_and_mapping HTTP/1.1" 200 - "http://172.16.2.200:8080/workflow/editor?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 33165/33431] 172.16.2.70 () {38 vars in 810 bytes} [Thu Jun 17 15:43:53 2021] GET /api/datatypes/types_and_mapping => generated 111916 bytes in 19 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
172.16.2.70 - - [17/Jun/2021:15:43:53 +0000] "GET /api/licenses HTTP/1.1" 200 - "http://172.16.2.200:8080/workflow/editor?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 428449|app: 0|req: 33165/33432] 172.16.2.70 () {38 vars in 772 bytes} [Thu Jun 17 15:43:53 2021] GET /api/licenses => generated 214936 bytes in 21 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 2)

I met this error again, this time on 21.05 21.01. I “purged” running workflow and tried to schedule the same workflow from the same history, and the “Workflow submission failed” appeared.

cat galaxy.log | grep -v 'HTTP/1.1" 200' | grep -Pi 'HTTP/1.1"|excep|error' --color=always | tail -n 300
.....
.....
172.16.2.70 - - [15/Jul/2021:17:00:09 +0300] "GET /welcome HTTP/1.1" 302 271 "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [15/Jul/2021:17:02:29 +0300] "GET /welcome HTTP/1.1" 302 271 "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [15/Jul/2021:17:10:15 +0300] "GET /welcome HTTP/1.1" 302 271 "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [15/Jul/2021:17:13:03 +0300] "GET /welcome HTTP/1.1" 302 271 "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
            echo "Failed to activate conda environment! Error was:"
} ; _JAVA_OPTIONS=${_JAVA_OPTIONS:-'-Xmx2048m -Xms256m'} && export _JAVA_OPTIONS &&  picard FastqToSam  FASTQ="/home/transgen/galaxy/database/objects/b/d/e/dataset_bdedba0c-ff8d-483b-92e9-736c0b67edd0.dat" FASTQ2="/home/transgen/galaxy/database/objects/3/f/d/dataset_3fd576a7-ed49-4ed7-b96b-10e69e67d9a9.dat"  QUALITY_FORMAT="Standard" OUTPUT="/home/transgen/galaxy/database/objects/5/0/4/dataset_5047626c-e115-4f64-bfa2-16182b651ea5.dat" READ_GROUP_NAME="h1905_normal" SAMPLE_NAME="h1905_normal"  LIBRARY_NAME="LIB1"  PLATFORM_UNIT="UNIT1"  PLATFORM="ILLUMINA"   PREDICTED_INSERT_SIZE="0"     MIN_Q="0" MAX_Q="93" STRIP_UNPAIRED_MATE_NUMBER="false" ALLOW_AND_IGNORE_EMPTY_LINES="false"  SORT_ORDER=coordinate VALIDATION_STRINGENCY="LENIENT" QUIET=true VERBOSITY=ERROR  `if [ -n "$TMPDIR" ] ; then echo 'TMP_DIR=$TMPDIR' ; else if [ -n "$TEMP" ] ; then echo 'TMP_DIR=$TEMP' ; fi ; fi`]
            echo "Failed to activate conda environment! Error was:"
} ; _JAVA_OPTIONS=${_JAVA_OPTIONS:-'-Xmx2048m -Xms256m'} && export _JAVA_OPTIONS &&  picard FastqToSam  FASTQ="/home/transgen/galaxy/database/objects/6/9/f/dataset_69fbeba5-4bfc-418d-8201-2d65651f9b60.dat"  QUALITY_FORMAT="Standard" OUTPUT="/home/transgen/galaxy/database/objects/b/a/2/dataset_ba27560e-46cd-4130-b71e-2645445ee770.dat" READ_GROUP_NAME="A" SAMPLE_NAME="sample-a"          MIN_Q="0" MAX_Q="93" STRIP_UNPAIRED_MATE_NUMBER="false" ALLOW_AND_IGNORE_EMPTY_LINES="false"  SORT_ORDER=coordinate VALIDATION_STRINGENCY="LENIENT" QUIET=true VERBOSITY=ERROR  `if [ -n "$TMPDIR" ] ; then echo 'TMP_DIR=$TMPDIR' ; else if [ -n "$TEMP" ] ; then echo 'TMP_DIR=$TEMP' ; fi ; fi`]
            echo "Failed to activate conda environment! Error was:"
} ; _JAVA_OPTIONS=${_JAVA_OPTIONS:-'-Xmx2048m -Xms256m'} && export _JAVA_OPTIONS &&  picard FastqToSam  FASTQ="/home/transgen/galaxy/database/objects/d/6/4/dataset_d64de6bf-b09d-41fb-a735-20fdf890e741.dat" FASTQ2="/home/transgen/galaxy/database/objects/4/2/7/dataset_427df8bb-451b-48df-b657-a408003b3847.dat"  QUALITY_FORMAT="Standard" OUTPUT="/home/transgen/galaxy/database/objects/3/1/f/dataset_31f5ae01-dae5-4b10-a169-ebaad39e8217.dat" READ_GROUP_NAME="h1905_tumor" SAMPLE_NAME="h1905_tumor"  LIBRARY_NAME="LIB1"  PLATFORM_UNIT="UNIT1"  PLATFORM="ILLUMINA"   PREDICTED_INSERT_SIZE="0"     MIN_Q="0" MAX_Q="93" STRIP_UNPAIRED_MATE_NUMBER="false" ALLOW_AND_IGNORE_EMPTY_LINES="false"  SORT_ORDER=coordinate VALIDATION_STRINGENCY="LENIENT" QUIET=true VERBOSITY=ERROR  `if [ -n "$TMPDIR" ] ; then echo 'TMP_DIR=$TMPDIR' ; else if [ -n "$TEMP" ] ; then echo 'TMP_DIR=$TEMP' ; fi ; fi`]
galaxy.tools.error_reports DEBUG 2021-07-15 17:20:52,067 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-21] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs DEBUG 2021-07-15 17:20:52,555 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1222', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.tools.error_reports DEBUG 2021-07-15 17:20:52,742 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:20:52,754 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1218
galaxy.exceptions.ObjectNotFound: No such object found.
galaxy.jobs DEBUG 2021-07-15 17:20:53,162 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1224', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.tools.error_reports DEBUG 2021-07-15 17:20:53,939 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:20:53,968 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1221
galaxy.exceptions.ObjectNotFound: No such object found.
galaxy.tools.error_reports DEBUG 2021-07-15 17:20:54,225 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-9] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.tools.error_reports DEBUG 2021-07-15 17:20:55,613 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-17] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.tools.error_reports DEBUG 2021-07-15 17:20:55,613 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-17] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs DEBUG 2021-07-15 17:20:55,755 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1226', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.tools.error_reports DEBUG 2021-07-15 17:20:55,936 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:20:55,979 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1223
galaxy.exceptions.ObjectNotFound: No such object found.
galaxy.jobs DEBUG 2021-07-15 17:20:57,696 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1231', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.jobs DEBUG 2021-07-15 17:20:58,085 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1233', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.jobs DEBUG 2021-07-15 17:20:58,222 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1234', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:00,502 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:00,502 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:21:00,511 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1228
galaxy.exceptions.ObjectNotFound: No such object found.
galaxy.jobs DEBUG 2021-07-15 17:21:01,465 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1235', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.jobs DEBUG 2021-07-15 17:21:01,498 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1236', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:02,020 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:02,020 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:21:02,061 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1230
galaxy.exceptions.ObjectNotFound: No such object found.
galaxy.jobs DEBUG 2021-07-15 17:21:03,823 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1237', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:03,975 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:21:03,984 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1232
galaxy.exceptions.ObjectNotFound: No such object found.
galaxy.jobs DEBUG 2021-07-15 17:21:10,706 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1252', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:10,821 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:21:10,866 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1249
galaxy.exceptions.ObjectNotFound: No such object found.
galaxy.jobs DEBUG 2021-07-15 17:21:12,526 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1254', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:12,778 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:21:12,794 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1253
galaxy.exceptions.ObjectNotFound: No such object found.
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:14,697 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:21:14,701 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1255
galaxy.exceptions.ObjectNotFound: No such object found.
172.16.2.70 - - [15/Jul/2021:17:22:43 +0300] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
cat galaxy.log | grep -iC 10 'invocations HTTP/1.1" 400'
--
172.16.2.70 - - [14/Jul/2021:17:08:10 +0300] "POST /api/tools/scripts_optitype/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11072/11117] 172.16.2.70 () {46 vars in 954 bytes} [Wed Jul 14 17:08:10 2021] POST /api/tools/scripts_optitype/build => generated 6791 bytes in 108 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [14/Jul/2021:17:08:10 +0300] "POST /api/tools/bwa_geno/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11072/11118] 172.16.2.70 () {46 vars in 938 bytes} [Wed Jul 14 17:08:10 2021] POST /api/tools/bwa_geno/build => generated 7746 bytes in 397 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 0)
172.16.2.70 - - [14/Jul/2021:17:08:10 +0300] "POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11072/11119] 172.16.2.70 () {46 vars in 1062 bytes} [Wed Jul 14 17:08:10 2021] POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build => generated 26789 bytes in 404 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
172.16.2.70 - - [14/Jul/2021:17:08:16 +0300] "POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [14/Jul/2021:17:08:16 +0300] "POST /api/tools/star_geno/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11074/11120] 172.16.2.70 () {46 vars in 1062 bytes} [Wed Jul 14 17:08:16 2021] POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build => generated 26644 bytes in 324 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 2)
[pid: 2153100|app: 0|req: 11074/11121] 172.16.2.70 () {46 vars in 940 bytes} [Wed Jul 14 17:08:16 2021] POST /api/tools/star_geno/build => generated 7375 bytes in 321 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [14/Jul/2021:17:09:17 +0300] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11075/11122] 172.16.2.70 () {44 vars in 937 bytes} [Wed Jul 14 17:09:17 2021] POST /api/workflows/1cd8e2f6b131e891/invocations => generated 211 bytes in 121 msecs (HTTP/1.1 400) 3 headers in 148 bytes (1 switches on core 0)
[pid: 2153100|app: -1|req: -1/11123] 172.16.2.70 () {38 vars in 887 bytes} [Wed Jul 14 17:09:17 2021] GET /static/dist/assets/b242d750b24c13ee4fea0eda1d0efc69.png => generated 826 bytes in 0 msecs via offload() (HTTP/1.1 200) 3 headers in 111 bytes (0 switches on core 3)
172.16.2.70 - - [14/Jul/2021:17:09:30 +0300] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11076/11124] 172.16.2.70 () {44 vars in 938 bytes} [Wed Jul 14 17:09:30 2021] POST /api/workflows/1cd8e2f6b131e891/invocations => generated 211 bytes in 69 msecs (HTTP/1.1 400) 3 headers in 148 bytes (1 switches on core 2)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "POST /api/histories HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11077/11125] 172.16.2.70 () {46 vars in 905 bytes} [Wed Jul 14 17:10:11 2021] POST /api/histories => generated 565 bytes in 426 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /history/set_as_current?id=63cd3858d057a6d1 HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11078/11126] 172.16.2.70 () {40 vars in 881 bytes} [Wed Jul 14 17:10:11 2021] GET /history/set_as_current?id=63cd3858d057a6d1 => generated 565 bytes in 74 msecs (HTTP/1.1 200) 7 headers in 207 bytes (1 switches on core 0)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /history/current_history_json HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11079/11127] 172.16.2.70 () {40 vars in 854 bytes} [Wed Jul 14 17:10:11 2021] GET /history/current_history_json => generated 565 bytes in 41 msecs (HTTP/1.1 200) 7 headers in 207 bytes (1 switches on core 3)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /api/users/f2db41e1fa331b3e HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11082/11128] 172.16.2.70 () {40 vars in 850 bytes} [Wed Jul 14 17:10:11 2021] GET /api/users/f2db41e1fa331b3e => generated 492 bytes in 51 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 2)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /api/histories/63cd3858d057a6d1/contents?limit=500&offset=0&order=hid&v=dev&q=deleted&q=purged&q=visible&qv=False&qv=False&qv=True HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11082/11129] 172.16.2.70 () {40 vars in 1055 bytes} [Wed Jul 14 17:10:11 2021] GET /api/histories/63cd3858d057a6d1/contents?limit=500&offset=0&order=hid&v=dev&q=deleted&q=purged&q=visible&qv=False&qv=False&qv=True => generated 7937 bytes in 87 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /history/set_as_current?id=63cd3858d057a6d1 HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11082/11130] 172.16.2.70 () {38 vars in 829 bytes} [Wed Jul 14 17:10:11 2021] GET /history/set_as_current?id=63cd3858d057a6d1 => generated 565 bytes in 114 msecs (HTTP/1.1 200) 7 headers in 207 bytes (1 switches on core 0)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /api/histories/63cd3858d057a6d1?keys=size%2Cnon_ready_jobs%2Ccontents_active%2Chid_counter HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11083/11131] 172.16.2.70 () {40 vars in 975 bytes} [Wed Jul 14 17:10:11 2021] GET /api/histories/63cd3858d057a6d1?keys=size%2Cnon_ready_jobs%2Ccontents_active%2Chid_counter => generated 124 bytes in 45 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
172.16.2.70 - - [14/Jul/2021:17:10:18 +0300] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11084/11132] 172.16.2.70 () {44 vars in 938 bytes} [Wed Jul 14 17:10:18 2021] POST /api/workflows/1cd8e2f6b131e891/invocations => generated 211 bytes in 72 msecs (HTTP/1.1 400) 3 headers in 148 bytes (1 switches on core 2)
172.16.2.70 - - [14/Jul/2021:17:10:21 +0300] "GET /api/workflows HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/list" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11085/11133] 172.16.2.70 () {38 vars in 753 bytes} [Wed Jul 14 17:10:21 2021] GET /api/workflows => generated 2776 bytes in 55 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [14/Jul/2021:17:10:25 +0300] "GET /workflows/run?id=1cd8e2f6b131e891 HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/list" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11086/11134] 172.16.2.70 () {40 vars in 929 bytes} [Wed Jul 14 17:10:25 2021] GET /workflows/run?id=1cd8e2f6b131e891 => generated 136508 bytes in 81 msecs (HTTP/1.1 200) 7 headers in 200 bytes (1 switches on core 0)
172.16.2.70 - - [14/Jul/2021:17:10:26 +0300] "GET /api/webhooks HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11087/11135] 172.16.2.70 () {40 vars in 822 bytes} [Wed Jul 14 17:10:26 2021] GET /api/webhooks => generated 7467 bytes in 17 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
172.16.2.70 - - [14/Jul/2021:17:10:26 +0300] "GET /api/webhooks HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11088/11136] 172.16.2.70 () {40 vars in 822 bytes} [Wed Jul 14 17:10:26 2021] GET /api/webhooks => generated 7467 bytes in 14 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 2)
172.16.2.70 - - [14/Jul/2021:17:10:26 +0300] "GET /api/webhooks HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
--
172.16.2.70 - - [15/Jul/2021:17:22:29 +0300] "GET /api/histories/cbbbf59e8f08c98c/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=1970-01-01T00%3A00%3A00.000Z&qv=False&qv=False HTTP/1.1" 200 - "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16919/16990] 172.16.2.70 () {40 vars in 1046 bytes} [Thu Jul 15 17:22:29 2021] GET /api/histories/cbbbf59e8f08c98c/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=1970-01-01T00%3A00%3A00.000Z&qv=False&qv=False => generated 2644 bytes in 51 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 0)
172.16.2.70 - - [15/Jul/2021:17:22:29 +0300] "GET /api/histories/cbbbf59e8f08c98c?keys=size%2Cnon_ready_jobs%2Ccontents_active%2Chid_counter HTTP/1.1" 200 - "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16920/16991] 172.16.2.70 () {40 vars in 942 bytes} [Thu Jul 15 17:22:29 2021] GET /api/histories/cbbbf59e8f08c98c?keys=size%2Cnon_ready_jobs%2Ccontents_active%2Chid_counter => generated 122 bytes in 57 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 2)
172.16.2.70 - - [15/Jul/2021:17:22:29 +0300] "GET /api/users/f2db41e1fa331b3e HTTP/1.1" 200 - "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16921/16992] 172.16.2.70 () {40 vars in 817 bytes} [Thu Jul 15 17:22:29 2021] GET /api/users/f2db41e1fa331b3e => generated 491 bytes in 29 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [15/Jul/2021:17:22:35 +0300] "POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16923/16993] 172.16.2.70 () {46 vars in 1063 bytes} [Thu Jul 15 17:22:35 2021] POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build => generated 22979 bytes in 174 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
172.16.2.70 - - [15/Jul/2021:17:22:35 +0300] "POST /api/tools/star_geno/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16923/16994] 172.16.2.70 () {46 vars in 941 bytes} [Thu Jul 15 17:22:35 2021] POST /api/tools/star_geno/build => generated 5028 bytes in 186 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 0)
172.16.2.70 - - [15/Jul/2021:17:22:43 +0300] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16924/16995] 172.16.2.70 () {44 vars in 938 bytes} [Thu Jul 15 17:22:43 2021] POST /api/workflows/1cd8e2f6b131e891/invocations => generated 211 bytes in 421 msecs (HTTP/1.1 400) 3 headers in 148 bytes (1 switches on core 2)
galaxy.jobs.runners.util.process_groups DEBUG 2021-07-15 17:24:50,620 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-1] check_pg(): No process found in process group 2854858
galaxy.jobs.runners.local DEBUG 2021-07-15 17:24:50,621 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-1] execution finished: /home/transgen/galaxy/database/jobs_directory/001/1266/galaxy_1266.sh
galaxy.model.metadata DEBUG 2021-07-15 17:24:50,687 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-1] loading metadata from file for: HistoryDatasetAssociation 1729
galaxy.jobs INFO 2021-07-15 17:24:51,063 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-1] Collecting metrics for Job 1266 in /home/transgen/galaxy/database/jobs_directory/001/1266
galaxy.jobs DEBUG 2021-07-15 17:24:51,101 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-1] job_wrapper.finish for job 1266 executed (453.734 ms)
galaxy.jobs.runners.util.process_groups DEBUG 2021-07-15 17:31:54,736 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-3] check_pg(): No process found in process group 2854909
galaxy.jobs.runners.local DEBUG 2021-07-15 17:31:54,737 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-3] execution finished: /home/transgen/galaxy/database/jobs_directory/001/1274/galaxy_1274.sh
galaxy.model.metadata DEBUG 2021-07-15 17:31:54,835 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-3] loading metadata from file for: HistoryDatasetAssociation 1737
galaxy.model.metadata DEBUG 2021-07-15 17:31:54,997 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-3] loading metadata from file for: HistoryDatasetAssociation 1738
--
172.16.2.70 - - [14/Jul/2021:17:08:10 +0300] "POST /api/tools/scripts_optitype/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11072/11117] 172.16.2.70 () {46 vars in 954 bytes} [Wed Jul 14 17:08:10 2021] POST /api/tools/scripts_optitype/build => generated 6791 bytes in 108 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [14/Jul/2021:17:08:10 +0300] "POST /api/tools/bwa_geno/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11072/11118] 172.16.2.70 () {46 vars in 938 bytes} [Wed Jul 14 17:08:10 2021] POST /api/tools/bwa_geno/build => generated 7746 bytes in 397 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 0)
172.16.2.70 - - [14/Jul/2021:17:08:10 +0300] "POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11072/11119] 172.16.2.70 () {46 vars in 1062 bytes} [Wed Jul 14 17:08:10 2021] POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build => generated 26789 bytes in 404 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
172.16.2.70 - - [14/Jul/2021:17:08:16 +0300] "POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [14/Jul/2021:17:08:16 +0300] "POST /api/tools/star_geno/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11074/11120] 172.16.2.70 () {46 vars in 1062 bytes} [Wed Jul 14 17:08:16 2021] POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build => generated 26644 bytes in 324 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 2)
[pid: 2153100|app: 0|req: 11074/11121] 172.16.2.70 () {46 vars in 940 bytes} [Wed Jul 14 17:08:16 2021] POST /api/tools/star_geno/build => generated 7375 bytes in 321 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [14/Jul/2021:17:09:17 +0300] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11075/11122] 172.16.2.70 () {44 vars in 937 bytes} [Wed Jul 14 17:09:17 2021] POST /api/workflows/1cd8e2f6b131e891/invocations => generated 211 bytes in 121 msecs (HTTP/1.1 400) 3 headers in 148 bytes (1 switches on core 0)
[pid: 2153100|app: -1|req: -1/11123] 172.16.2.70 () {38 vars in 887 bytes} [Wed Jul 14 17:09:17 2021] GET /static/dist/assets/b242d750b24c13ee4fea0eda1d0efc69.png => generated 826 bytes in 0 msecs via offload() (HTTP/1.1 200) 3 headers in 111 bytes (0 switches on core 3)
172.16.2.70 - - [14/Jul/2021:17:09:30 +0300] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11076/11124] 172.16.2.70 () {44 vars in 938 bytes} [Wed Jul 14 17:09:30 2021] POST /api/workflows/1cd8e2f6b131e891/invocations => generated 211 bytes in 69 msecs (HTTP/1.1 400) 3 headers in 148 bytes (1 switches on core 2)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "POST /api/histories HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11077/11125] 172.16.2.70 () {46 vars in 905 bytes} [Wed Jul 14 17:10:11 2021] POST /api/histories => generated 565 bytes in 426 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /history/set_as_current?id=63cd3858d057a6d1 HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11078/11126] 172.16.2.70 () {40 vars in 881 bytes} [Wed Jul 14 17:10:11 2021] GET /history/set_as_current?id=63cd3858d057a6d1 => generated 565 bytes in 74 msecs (HTTP/1.1 200) 7 headers in 207 bytes (1 switches on core 0)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /history/current_history_json HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11079/11127] 172.16.2.70 () {40 vars in 854 bytes} [Wed Jul 14 17:10:11 2021] GET /history/current_history_json => generated 565 bytes in 41 msecs (HTTP/1.1 200) 7 headers in 207 bytes (1 switches on core 3)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /api/users/f2db41e1fa331b3e HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11082/11128] 172.16.2.70 () {40 vars in 850 bytes} [Wed Jul 14 17:10:11 2021] GET /api/users/f2db41e1fa331b3e => generated 492 bytes in 51 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 2)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /api/histories/63cd3858d057a6d1/contents?limit=500&offset=0&order=hid&v=dev&q=deleted&q=purged&q=visible&qv=False&qv=False&qv=True HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11082/11129] 172.16.2.70 () {40 vars in 1055 bytes} [Wed Jul 14 17:10:11 2021] GET /api/histories/63cd3858d057a6d1/contents?limit=500&offset=0&order=hid&v=dev&q=deleted&q=purged&q=visible&qv=False&qv=False&qv=True => generated 7937 bytes in 87 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /history/set_as_current?id=63cd3858d057a6d1 HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11082/11130] 172.16.2.70 () {38 vars in 829 bytes} [Wed Jul 14 17:10:11 2021] GET /history/set_as_current?id=63cd3858d057a6d1 => generated 565 bytes in 114 msecs (HTTP/1.1 200) 7 headers in 207 bytes (1 switches on core 0)
172.16.2.70 - - [14/Jul/2021:17:10:11 +0300] "GET /api/histories/63cd3858d057a6d1?keys=size%2Cnon_ready_jobs%2Ccontents_active%2Chid_counter HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11083/11131] 172.16.2.70 () {40 vars in 975 bytes} [Wed Jul 14 17:10:11 2021] GET /api/histories/63cd3858d057a6d1?keys=size%2Cnon_ready_jobs%2Ccontents_active%2Chid_counter => generated 124 bytes in 45 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
172.16.2.70 - - [14/Jul/2021:17:10:18 +0300] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11084/11132] 172.16.2.70 () {44 vars in 938 bytes} [Wed Jul 14 17:10:18 2021] POST /api/workflows/1cd8e2f6b131e891/invocations => generated 211 bytes in 72 msecs (HTTP/1.1 400) 3 headers in 148 bytes (1 switches on core 2)
172.16.2.70 - - [14/Jul/2021:17:10:21 +0300] "GET /api/workflows HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/list" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11085/11133] 172.16.2.70 () {38 vars in 753 bytes} [Wed Jul 14 17:10:21 2021] GET /api/workflows => generated 2776 bytes in 55 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [14/Jul/2021:17:10:25 +0300] "GET /workflows/run?id=1cd8e2f6b131e891 HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/list" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11086/11134] 172.16.2.70 () {40 vars in 929 bytes} [Wed Jul 14 17:10:25 2021] GET /workflows/run?id=1cd8e2f6b131e891 => generated 136508 bytes in 81 msecs (HTTP/1.1 200) 7 headers in 200 bytes (1 switches on core 0)
172.16.2.70 - - [14/Jul/2021:17:10:26 +0300] "GET /api/webhooks HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11087/11135] 172.16.2.70 () {40 vars in 822 bytes} [Wed Jul 14 17:10:26 2021] GET /api/webhooks => generated 7467 bytes in 17 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
172.16.2.70 - - [14/Jul/2021:17:10:26 +0300] "GET /api/webhooks HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 11088/11136] 172.16.2.70 () {40 vars in 822 bytes} [Wed Jul 14 17:10:26 2021] GET /api/webhooks => generated 7467 bytes in 14 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 2)
172.16.2.70 - - [14/Jul/2021:17:10:26 +0300] "GET /api/webhooks HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
--
172.16.2.70 - - [15/Jul/2021:17:22:29 +0300] "GET /api/histories/cbbbf59e8f08c98c/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=1970-01-01T00%3A00%3A00.000Z&qv=False&qv=False HTTP/1.1" 200 - "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16919/16990] 172.16.2.70 () {40 vars in 1046 bytes} [Thu Jul 15 17:22:29 2021] GET /api/histories/cbbbf59e8f08c98c/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=1970-01-01T00%3A00%3A00.000Z&qv=False&qv=False => generated 2644 bytes in 51 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 0)
172.16.2.70 - - [15/Jul/2021:17:22:29 +0300] "GET /api/histories/cbbbf59e8f08c98c?keys=size%2Cnon_ready_jobs%2Ccontents_active%2Chid_counter HTTP/1.1" 200 - "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16920/16991] 172.16.2.70 () {40 vars in 942 bytes} [Thu Jul 15 17:22:29 2021] GET /api/histories/cbbbf59e8f08c98c?keys=size%2Cnon_ready_jobs%2Ccontents_active%2Chid_counter => generated 122 bytes in 57 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 2)
172.16.2.70 - - [15/Jul/2021:17:22:29 +0300] "GET /api/users/f2db41e1fa331b3e HTTP/1.1" 200 - "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16921/16992] 172.16.2.70 () {40 vars in 817 bytes} [Thu Jul 15 17:22:29 2021] GET /api/users/f2db41e1fa331b3e => generated 491 bytes in 29 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 1)
172.16.2.70 - - [15/Jul/2021:17:22:35 +0300] "POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16923/16993] 172.16.2.70 () {46 vars in 1063 bytes} [Thu Jul 15 17:22:35 2021] POST /api/tools/toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_FastqToSam/2.18.2.1/build => generated 22979 bytes in 174 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
172.16.2.70 - - [15/Jul/2021:17:22:35 +0300] "POST /api/tools/star_geno/build HTTP/1.1" 200 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16923/16994] 172.16.2.70 () {46 vars in 941 bytes} [Thu Jul 15 17:22:35 2021] POST /api/tools/star_geno/build => generated 5028 bytes in 186 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 0)
172.16.2.70 - - [15/Jul/2021:17:22:43 +0300] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[pid: 2153100|app: 0|req: 16924/16995] 172.16.2.70 () {44 vars in 938 bytes} [Thu Jul 15 17:22:43 2021] POST /api/workflows/1cd8e2f6b131e891/invocations => generated 211 bytes in 421 msecs (HTTP/1.1 400) 3 headers in 148 bytes (1 switches on core 2)
galaxy.jobs.runners.util.process_groups DEBUG 2021-07-15 17:24:50,620 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-1] check_pg(): No process found in process group 2854858
galaxy.jobs.runners.local DEBUG 2021-07-15 17:24:50,621 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-1] execution finished: /home/transgen/galaxy/database/jobs_directory/001/1266/galaxy_1266.sh
galaxy.model.metadata DEBUG 2021-07-15 17:24:50,687 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-1] loading metadata from file for: HistoryDatasetAssociation 1729
galaxy.jobs INFO 2021-07-15 17:24:51,063 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-1] Collecting metrics for Job 1266 in /home/transgen/galaxy/database/jobs_directory/001/1266
galaxy.jobs DEBUG 2021-07-15 17:24:51,101 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-1] job_wrapper.finish for job 1266 executed (453.734 ms)
galaxy.jobs.runners.util.process_groups DEBUG 2021-07-15 17:31:54,736 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-3] check_pg(): No process found in process group 2854909
galaxy.jobs.runners.local DEBUG 2021-07-15 17:31:54,737 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-3] execution finished: /home/transgen/galaxy/database/jobs_directory/001/1274/galaxy_1274.sh
galaxy.model.metadata DEBUG 2021-07-15 17:31:54,835 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-3] loading metadata from file for: HistoryDatasetAssociation 1737
galaxy.model.metadata DEBUG 2021-07-15 17:31:54,997 [pN:main.web.1,p:2153100,w:1,m:0,tN:LocalRunner.work_thread-3] loading metadata from file for: HistoryDatasetAssociation 1738