Workflow is not operating on a collection

[Ti_Litt]

Hi,

I’m trying to use an RNAseq analysis workflow on a collection constructed by NCBI fastq dump of paired-end reads.

My workflow involves Trim Galore!, FastQC, HISAT2 and featurecounts; all I believe set to the needed settings to deal with paired-end collections. However despite the workflow claiming to be invoked nothing happens. Similar workflows just for paired-end files will work however the workflow is not operating for collections. I can use the programs individually on the collection and they work but never together as a workflow.

I have attached links to the workflow and an example collection for which it isn’t working for:

workflow: https://usegalaxy.org/u/t_little/w/rnaseqpairedendpairedcollection

history with dataset example: https://usegalaxy.org:/u/t_little/h/test

Best wishes,
Tim.

[jennaj]

Hi - Add in an “input” paired-end collection dataset to your workflow before the Trim Galore! step.

This screenshot of a sample workflow in the editor shows how to make the connection:

[Ti_Litt]

Hi Jen,

Apologies for the delay in following this up, and thank you for your response. I’m having an issue with this still as when I create the Input Dataset Collection and set it to ‘list:paired’ Trim Galore! still won’t accept it as an input. It will accept ‘Paired’ input however this will do the same thing as above when invoking the workflow; it will say its been started but not actually do anything.

Best,
Tim.

[jennaj]

Hi Tim,

Yes, this can happen when new “inputs” are added to a workflow. We are working on a correction. This prior Q&A covers how to work around the problem – disconnect all “noodles”, then reconnect starting from the “input” dataset through the downstream tools.

Workflow doesn't execute all steps after successfully being invoked.

Update: We created a new ticket to address this usage problem. If interested, the ticket is here: https://github.com/galaxyproject/galaxy/issues/7431

In short, what you noticed and what I thought was going on turned out to still be an open issue. This manifests as incompatible connections when adding in a new or different “input type” if tools are already connected downstream based on a different (or absent) initial “input type”. We agree it would be much better if the “input type” was re-evaluated on the fly, during a workflow editing session (and that is what the ticket above is describing).

Removing all connections, then linking them back up (starting with the “input”) is how to get the “input type” to reset so that noodles will connect correctly between tools.

Save a copy of your original workflow back first in case something else goes wrong and you need to start over but don’t want to completely start over!

[Ti_Litt]

Hi Jen,

This has corrected my issue with adding the paired collection input, however I still cannot invoke the workflow successfully.

Best,
Tim.

[jennaj]

I still cannot invoke the workflow successfully

Does the workflow actually launch or does it not submit at all?

If it does launch, datasets can take some time to appear in the target history. Maybe check now? An hour should be long enough.

[Ti_Litt]

Hi Jen,

Unfortunately it submits but does not do anything. I had tried to invoke the workflow a few hours before my response above and had not seen any changes occurring. Mysteriously my history appears to have 60GB of data which
doesn’t reflect what I actually have in it, however there’s no extra datasets hidden or not permanently deleted.

Best wishes,

Tim.

[jennaj]

Ok, thanks for explaining. The Galaxy Main https://usegalaxy.org public server has been extremely busy today. There is a bit of a backlog in job processing that is being worked through.

I suspect that your workflow jobs are partially staged. I would recommend leaving everything “as-is” for at least the next 12 hours, ideally 24. Restarting/rerunning is unlikely to help and instead only cause more delays.

Note: This assumes the workflow was edited/fixed – the “mystery” history size implies that some part of the workflow has already started up, but that isn’t really confirmed yet.

If you want more review (now, or later after waiting a bit longer for the server to catch up), please send me a direct message with a share link to the history (or histories, if “send output to a new history” was used) and to the workflow, or you can post those links publically in this thread – your choice. It would be difficult to find the right data and troubleshoot, even if your account email here is the same as at the server. I don’t need your password and you should never share that with anyone, ever.

[jennaj]

Update:

Hi Tim, I checked the history/workflow links from the original post and those appear to still be where you are currently testing. The results match the workflow and are dated from today.

Maybe it finished?

[Ti_Litt]

Hi Jen,

Yes you are correct, next time I’ll be more careful about using that test history! However these jobs are the outputs of running each software individually, the workflow still does not appear to have been invoked. However I think the data may be more memory heavy than I suspected so I may try again but using less of the disc quota in case this is the issue.

I’ll send a direct message if this continues being an issue with more exact details.

Best,
Tim.