Dear GalaxyHelp,
I have a problem with my local instance of galaxy (20.01). I suddenly noticed it was not starting any new jobs or uploading files (staying grey forever). When I looked at the galaxy log, it was constantly starting new jobs, but the database/job_directory folders galaxy created seem ‘empty’ (empty working and output folder). As this is a development instance, I tried to completely remove the tmp and job_directory. Still, if I restart galaxy, it will try to start new jobs. This instance doesn’t consist of users that started any jobs, so it is very confusing. I must say that I am working on an automated updating of libraries via the bioblend api. I’m curious if it is possible that this api invokes certain jobs that are stuck in a loop?
After some dependency checks which it usually does, my logfile looks like this:
Starting server in PID 11717.
serving on http://xxx:8080
galaxy.model.database_heartbeat DEBUG 2020-06-05 13:52:26,856 [p:11717,w:1,m:0] [database_heartbeart_main.web.1.thread] main.web.1 is config watcher
galaxy.jobs.mapper DEBUG 2020-06-05 13:52:27,216 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80530) Mapped job to destination id: local
galaxy.jobs.handler DEBUG 2020-06-05 13:52:27,232 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80530) Dispatching to local runner galaxy.jobs DEBUG 2020-06-05 13:52:27,253 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80530) Persisting job destination (destination id: local) galaxy.jobs DEBUG 2020-06-05 13:52:27,253 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80530) Working directory for job is: /home/galaxy/galaxy/database/jobs_directory/080/80530
galaxy.jobs.runners DEBUG 2020-06-05 13:52:27,278 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] Job [80530] queued (45.707 ms)
galaxy.jobs.handler INFO 2020-06-05 13:52:27,288 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80530) Job dispatched
galaxy.jobs.mapper DEBUG 2020-06-05 13:52:27,297 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80531) Mapped job to destination id: local
galaxy.jobs.handler DEBUG 2020-06-05 13:52:27,316 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80531) Dispatching to local runner
galaxy.jobs DEBUG 2020-06-05 13:52:27,337 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80531) Persisting job destination (destination id: local)
galaxy.jobs DEBUG 2020-06-05 13:52:27,339 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80531) Working directory for job is: /home/galaxy/galaxy/database/jobs_directory/080/8
0531
galaxy.jobs.runners DEBUG 2020-06-05 13:52:27,389 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] Job [80531] queued (72.643 ms)
galaxy.jobs.handler INFO 2020-06-05 13:52:27,408 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80531) Job dispatched
galaxy.jobs.mapper DEBUG 2020-06-05 13:52:27,436 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80532) Mapped job to destination id: local
galaxy.jobs.handler DEBUG 2020-06-05 13:52:27,469 [p:11717,w:1,m:0] [JobHandlerQueue.monitor_thread] (80532) Dispatching to local runner
etc. creating more and more job directories continuously.
Did any of you every experience something similar? I am on the edge of reinstalling the development instance, but I want to avoid this happening again of course. I was wondering if someone has an idea of forcing to stop this process for now.
Thank you so much in advance!
Annabel Dekker
