Workflow run disappeared in galaxy 21.01

, ,
1

Hello!

I was trying to run multiple (3 at this time) workflows in galaxy and upload data for another 6 workflow invocations. Everything went fine (not exactly, but not with workflow of question), but then i saw that one of my workflow runs has totally vanished! I am 146% sure it was running (moreover, it had been running for more than one hour). But there are no errors in web interface nor in the log!

transgen@transgen-4:~/galaxy$ cat galaxy.log | grep -Pv 'Failed to activate conda environment|VERBOSITY=ERROR|HTTP/1.1" 200' | grep -Pi 'HTTP/1.1"|excep|error' --color=always | tail -n 20
galaxy.jobs DEBUG 2021-07-15 17:21:03,823 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1237', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:03,975 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:21:03,984 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1232
galaxy.exceptions.ObjectNotFound: No such object found.
galaxy.jobs DEBUG 2021-07-15 17:21:10,706 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1252', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:10,821 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:21:10,866 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1249
galaxy.exceptions.ObjectNotFound: No such object found.
galaxy.jobs DEBUG 2021-07-15 17:21:12,526 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Pausing Job '1254', Execution of this dataset's job is paused because its input datasets are in an error state.
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:12,778 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:21:12,794 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1253
galaxy.exceptions.ObjectNotFound: No such object found.
galaxy.tools.error_reports DEBUG 2021-07-15 17:21:14,697 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7fc87b032220> generated response None
galaxy.jobs ERROR 2021-07-15 17:21:14,701 [pN:main.web.1,p:2153100,w:1,m:0,tN:JobHandlerQueue.monitor_thread] Unable to cleanup job 1255
galaxy.exceptions.ObjectNotFound: No such object found.
172.16.2.70 - - [15/Jul/2021:17:22:43 +0300] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [15/Jul/2021:17:34:21 +0300] "POST /api/workflows/1cd8e2f6b131e891/invocations HTTP/1.1" 400 - "http://172.16.2.200:8080/workflows/run?id=1cd8e2f6b131e891" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [15/Jul/2021:17:42:24 +0300] "GET /welcome HTTP/1.1" 302 271 "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [15/Jul/2021:18:01:17 +0300] "GET /welcome HTTP/1.1" 302 271 "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
172.16.2.70 - - [15/Jul/2021:18:16:02 +0300] "GET /welcome HTTP/1.1" 302 271 "http://172.16.2.200:8080/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"

So the last error message was at 18:16:02, but the run disappeared at about 19:15.

2

I think it is a web interface bug cos this run appeared again, but another run disappeared.

1 Like
3

Hi @wormball

Maybe upgrade to 21.05?

Otherwise, this just looks like your server is not quite keeping up with database updates. But @mvdbeek can correct me.

1 Like
4

Go to User → Workflow Invocation, it should be there.

2 Likes
5

Maybe upgrade to 21.05?

I definitely will do this, but slightly later. Actually i thought i did this, but it turned out i did it on another server. :frowning:

this just looks like your server is not quite keeping up with database updates

It may be the case. I also encountered some error like “the server could not complete the request” yesterday when about 6 workflows were running and i tried to run another workflow, and this workflow ran fine today. However a day earlier the server was even more loaded, and i did not encounter this error.