How to kill a failed job without gxadmin?

1

Help! I’m running a local dev instance of Galaxy purely to write a wrapper for an in-house R script and I have a failed job preventing Galaxy from loading at all, at least I think that’s the issue:

galaxy.jobs.handler ERROR 2021-01-14 13:54:24,285 [p:90,w:1,m:0] [JobHandlerQueue.monitor_thread] failure running job 621
Traceback (most recent call last):
File “lib/galaxy/jobs/handler.py”, line 408, in __handle_waiting_jobs
self.dispatcher.put(self.job_wrappers.pop(job.id))
File “lib/galaxy/jobs/handler.py”, line 1007, in put
self.job_runners[runner_name].put(job_wrapper)
File “lib/galaxy/jobs/runners/init.py”, line 152, in put
job_wrapper.enqueue()
File “lib/galaxy/jobs/init.py”, line 1456, in enqueue
self._set_object_store_ids(job)
File “lib/galaxy/jobs/init.py”, line 1478, in _set_object_store_ids
object_store_populator.set_object_store_id(dataset)
File “lib/galaxy/objectstore/init.py”, line 1082, in set_object_store_id
data.dataset.object_store_id = self.object_store_id
AttributeError: ‘NoneType’ object has no attribute ‘dataset’
galaxy.jobs.mapper DEBUG 2021-01-14 13:54:24,305 [p:90,w:1,m:0] [JobHandlerQueue.monitor_thread] (622) Mapped job to destination id: local
galaxy.jobs.handler DEBUG 2021-01-14 13:54:24,318 [p:90,w:1,m:0] [JobHandlerQueue.monitor_thread] (622) Dispatching to local runner
galaxy.jobs DEBUG 2021-01-14 13:54:24,464 [p:90,w:1,m:0] [JobHandlerQueue.monitor_thread] (622) Persisting job destination (destination id: local)
galaxy.jobs.handler ERROR 2021-01-14 13:54:24,473 [p:90,w:1,m:0] [JobHandlerQueue.monitor_thread] failure running job 622
Traceback (most recent call last):
File “lib/galaxy/jobs/handler.py”, line 408, in __handle_waiting_jobs
self.dispatcher.put(self.job_wrappers.pop(job.id))
File “lib/galaxy/jobs/handler.py”, line 1007, in put
self.job_runners[runner_name].put(job_wrapper)
File “lib/galaxy/jobs/runners/init.py”, line 152, in put
job_wrapper.enqueue()
File “lib/galaxy/jobs/init.py”, line 1456, in enqueue
self._set_object_store_ids(job)
File “lib/galaxy/jobs/init.py”, line 1478, in _set_object_store_ids
object_store_populator.set_object_store_id(dataset)
File “lib/galaxy/objectstore/init.py”, line 1082, in set_object_store_id
data.dataset.object_store_id = self.object_store_id
AttributeError: ‘NoneType’ object has no attribute ‘dataset’

As recommended here: Killing jobs or dead processes after a database crash on galaxy - #3 by Slugger70, I tried installing and running gxadmin but of course I haven’t set up a postgresql db for my dev instance, so I get this failure:

gxadmin mutate fail-job 621
/usr/bin/gxadmin: line 258: psql: command not found

I am completely stuck. Do I just need to install psql even though Galaxy is using SQLite? We’re going to have actual professionals help with Galaxy admin in about a month or two but I’d really like to use my local instance before then. Any recommendations would be great. I’m this close to gnawing off my own leg to get out of the Galaxy trap (i.e. clean reinstall).

1 Like
2

Have you tried using the Jobs tab under the Admin panel?

3

Alas, the job does not show up. When I load the Galaxy website I get this error: “An error occurred while updating information with the server. Please contact a Galaxy administrator if the problem persists.” In general, no information at all shows up, even though I am logged in as admin (history is missing too.)

4

Hey @KeelyDulmage I think when we suggested gxadmin to you, we didn’t know you were running with sqlite! This makes a lot of sense.

I think for that specific error you’ll need to run the sql in your sqlite3 database potentially.

Something like

sqlite3 universe.sqlite # Or wherever your DB is
update job set state='failed' where id = 621;
1 Like
5

Thank you @hexylena, that did it!

In case other naïve people doing Galaxy dev have the same question some day, my default sqlite db was in galaxy/database.

1 Like