Workflow execution stalled

Having the same problem posted in this thread (Cannot run my workflow - #2 by BenSamy2020). Individual analysis works fine, but when running workflow, it started successfully with the green check box, but then nothing happens. Been on this page for the whole day and can’t do anything.

1 Like

Hi @THYeh

Please review this topic instead, it has an example workflow + history that is created correctly: Invocation scheduling failed - Trying to run a simple mapping workflow - #5 by jennaj

In short, you need need to add one of the “Inputs” types to the very start of your workflow. Then, because you are adding/adjusting inputs, disconnect all existing “noodles” between the inputs/tools and reconnect them in order of execution. This resets the workflow metadata.

Tutorials: Search Tutorials

Thanks!

Hi @jennaj
Thank you for the help!
I’ve checked and did several adjustments according to the topic you posted but it still doesn’t work.

  1. I make sure all the input files are on the very start.
  2. Disconnecting and reconnect noodles between added input files and the following tasks.
  3. Switch from cutadapt to trimmomatic.
    When I did the first two adjustments, noodles between input and cutadapt is still disconnected every time after running the workflow. After changing to trimmomatic, noodles seem stable but the workflow is still not working. :frowning:

Update: Suddenly all the workflow I tried to run in the past two days have been successfully scheduled (~300 steps) this noon. I deleted all the ~300 steps from history and re-run one workflow. It’s successfully scheduled but non of the tasks is completed except import data sets after waiting for 2 hours, even a simple task such as FASTQ Groomer or splitter.

1 Like

Hi @THYeh

Thank you for all the feedback. This confirms that your workflow is correct.

Workflow invocation can be considered as a “meta” job. The invocation job is queued, then runs as server resources are available. Tool job queuing is also expected (whether the job is run with a workflow or directly from the history).

The usegalaxy.org server (and most other public Galaxy servers) are usually very busy. All incorporate computing resources that are shared among all users on the server.

The best strategy is to start up the workflow and allow it to fully complete. Workflow invocation is a process that “batch schedules” the individual steps (tool jobs) in the workflow. Those tool jobs will start off as queued (based on the time of invocation/scheduling) and will complete as resources specific to that step become available. There isn’t a set time frame for how long a workflow invocation or tool job will queue. A full day for some steps/tools is not unusual. While the tool type and input size can be a factor, how busy the server/clusters are is an even larger factor. That is why you’ll see an even smaller jobs queue – it is waiting for resources to become available.

If you haven’t done this already, consider updating your workflow before the next run to include email notifications at steps that are important to you. This will give you progress updates without having to log into Galaxy (or even be logged in to Galaxy).

That is one of the huge benefits of workflows: start it up, let it run, come back once done. No fiddling around with starting up tool jobs 1 by 1. A workflow places all jobs within that workflow into the job queue during invocation and tools will run sequentially without intermediate manual intervention. In short, workflows are not only an effective method to run work reproducibly but a practical method to avoid delays and get work completed quicker.

Very glad your workflows are scheduling now, and that you know they produce the desired output. It has been about a day since you wrote in, so I would expect the workflow you posted back to have started up by now – if not, it soon will. Allow that to complete. If you delete and rerun again, that will place all of it back at the end of the queue again, increasing queue time.

Thanks!