Hi,
I deleted a few jobs while they were running and also purged them. But they still seem to be running and I am not able to run any new jobs (they are staying grey). Could the admin manually clear the stuck jobs?
My account username: raghini
History name: RR23 RNAseq
Deleted & purged tools still running: MarkDuplicates and GeneBody Coverage
Please let me know if you need any other information.
Thanks.
This is on usegalaxy . org
Welcome @Raghini_Rajaram
I’m taking a look at your account and will report back once done, but wanted to explain quickly about how this works.
A scheduled job goes through a few stages (as you know I’m sure, but let’s put the details for whoever may read this later). The gray state is when a job is staged, then yellow for the execution runtime, and finally green or red depending on what happened at runtime. FAQ: Understanding job statuses
Canceling a job
If the dataset is still gray, then deleting/purging it will aborted the process immediately, because all of the actions for this queuing step are available and local to the Galaxy database.
If a dataset is yellow, and is executing the tool, then deleting/purging is itself a sort of staged process. Why? Jobs are dispatched to cluster nodes to run. There isn’t any communication between that cluster node and the database until the job arrives back, to deliver the results or to update about a failure. Attempting to force a cluster node to abort, with manual ad-hoc intervention, is something we avoid, even for our own jobs!
What about workflows? When jobs processed together in a workflow, if the upstream job failed (the input for a tool) then the downstream job will be paused (blue dataset). Once that happens you can delete/purge those datasets too.
What to do
If you ran a workflow (and sent the output to a new history, which should be the case most of the time), you can purge the entire output history, since this is faster and catches “all” the jobs associated with a workflow run together. For individual jobs, you could think about copying datasets to a new history, then purging the original history. This doesn’t make the process happen faster, since the communications are always based on the dataset status, but it can be a useful fast way to get rid of larger batches of work.
I’m going to check your account to see what is going on, however the general advice is to delete what you do not want anymore, queue up the replacements, then allow the scheduling system to manage the throughput for you.
More soon! 
Some details
datasets 1448, 1449, 1662 and others are all examples of job that were scheduled to be queued, and are still grey, and some are deleted and others purged. These jobs will never start up or be sent to a cluster. Instead, when those datasets move up to the top of the list, they’ll be aborted.
Other ideas
You could decide to Purge All Deleted Content.
In addition, once the queued Samtools jobs complete (one has already finished!), I would suggest using Copy History to start over and clear up the clutter from the old jobs. That way the new work will go into a simpler history. At the end, you could purge the original history. This will make it easier to navigate in the non-active tabs (less guessing about hidden datasets and which jobs they belong to). Or, you can do this at the end, or not at all! Your choice!
Tools are sorted out to appropriate cluster nodes, so even if your other deleted jobs were fully cleared from the listing for that set of cluster nodes, it wouldn’t necessarily help your queued jobs to run faster. All of the jobs are in the queue list, along with anyone else’s jobs: any with status deleted will be aborted (not dispatched), any still active will be dispatched, and any with status executing will be updated when that job communicates back from the server. This is similar to how most distributed cluster systems work.
I hope this helps to explain about the situation. Follow up questions are welcome! 
This is what I did - I did not know, sorry.
Thank you @jennaj Very helpful information in your replies - I appreciate it.
I have now cleared the clutter by copying my history and deleting the old history. The new tasks are still not running (staying grey for a long time). Would you be able to check what is going wrong?
Hi @jennaj I ran tool: RSeQC gene_body_coverage as a collection at 7:30 am (GMT+1) and it immediately turned to 4 orange (running) / rest grey. After 8+ hours (at 4 pm), it is still showing 4 running / rest grey with zero turnover in this time. Would you be able to check the job status server-side (whether these are actually alive on the compute nodes or zombied)? Thanks a ton for your help.
Great, I see your current history. Yes, some jobs in the history have started up and some are still queued.
The BAM datasets are quite large, so they will take some time to process. Next time you could do something like removing unmapped first, which may help each job to process quicker.
Then, this RSeqQC tool requires a cluster node with a minimum of 8 GB of memory. These nodes will generally be busier than others (more people sending jobs to them plus those runs taking longer to execute = slower turnover).
However, these jobs are only for statistics, correct? You could continue with your next pipeline steps since you already have your counts from Featurecounts ready to go.
For the server, yes, everything is working Ok. The server UseGalaxy.org server is about as busy as the other UseGalaxy servers right now.
Hope this helps!
