Kraken2 jobs and how these process at UseGalaxy.org

Hello,

I have several jobs that have remained in the queued state for an unusually long time on usegalaxy.org.

  • Kraken2: Two jobs (SRR24821405 and SRR24821387) have been queued for more than a day, while the other 11 samples in the same collection have completed successfully.

  • Filter: A simple filtering job in a separate history has also remained queued for over 12 hours.

Could you please check whether there is an issue with job scheduling or the execution environment?

Thank you for your help.

Welcome @Sangeun_Lee

Sorry to hear you are experiencing delays.

For reference, there are on average 35-45, Kraken2 jobs executing at any particular time at this server, including over the last several days. This can be up to 50, but rarely less than 20, so there some variability. Kraken2 requires very large indexes, and the jobs route to some of our highest memory cluster nodes. These are usually quite busy!!

It sounds like some of these were in the first batch of your jobs to execute. The rest should follow. The sorting of which jobs executed initially is likely not related to the sample itself, instead, this probably was just chance. But we can confirm for persistent problems of course!

Our best advice is to leave your jobs queued when the service is busy! If you have samples that do not process the first time, or with a rerun (to eliminate a cluster hiccup), please let us know and we can help to look into that specific failure closer to help resolve it.

Then, for the filtering job, I’m wondering which tool was involved and which input? If that routed to a high memory cluster node due to the tool, the query, or the input sizes, then a queue wait seems possible. Or, if you simply already had many other jobs already running, then any other new job might queue for a short time, as the server implements a fair access strategy per cluster and account. Some of your jobs run, some of other people’s jobs run, more of yours, repeat until all are completed.

The clusters are significant and tend to chomp through even large projects pretty quickly, but these are still “cluster dynamics” and difficult to predict with precision. Giving projects a few days to complete is a good strategy. Using workflows will maximise the throughput too as everything is queued at the same time and streams out and back, automatically, once cluster capacity frees up.

Please let us know how this goes! A shared history would be helpful for remaining troubleshooting. That can be pasted back here in a reply, or we can start up a private message. :slight_smile: