Hi @Jon_Colman
Yes, each server connects to multiple, large, academic clusters. The nodes have different technical specifications. There are complex rules about which resources a tool needs and which clusters are appropriate. Kraken2 needs a very high memory node for the reference data. These are busy nodes! Some of the jobs take a longer time to process due to the complexity. So, even a small query may queue waiting for a node to free up.
Error about read counts with mapping tool usually mean that the pairs of reads are missing one of the ends. Meaning, one mate end didn’t pass through the upstream step. This is expected when “mapping → more mapping”. The first mapping can lose an end.
Do the errors look like this?
The -2 here means the reverse reads, and -1 means the forward reads.
Certain mapping parameters are strict and both mated ends are required. QA trimming tools can be adjusted to only report intact pairs, but if this happens later on in an analysis, you can filter for intact pairs a few ways. The tools below are one, but also see the Seqtk tool suite for many more complex manipulations related to this.
To check for unmated pairs, this is one tool to use. It also reports issues with compression status, versus the datatype, or just in general (truncated, or corrupted for some reason).
- FASTQ info validates single or paired fastq files
To correct unmated pairs, this usually involves creating an interleaved file, dropping unmatched pairs, then splitting the files out again. You could put these two into a mini workflow for reuse directly, or include it as a subworkflow in another existing workflow. (And, I just saw that you are using this already, but let’s leave it for anyone reading later on!).
- FASTQ interlacer on paired end reads
- then, FASTQ de-interlacer on paired end reads
Then for
Oh, this is curious.
The EU server has an automatic rerun for failures (just one). In the history this can look like a job is running, then it goes back to queued, then finally runs again. The job either happened to hit a bad node, or the original error indicated that the job needs more resources, so it is being dispatched to a larger cluster node.
Running again would start that process over. If the error was by chance, then the new job has a chance to succeed! My immediate guess would be that the compress/uncompress functions are busy. Some tools understand compressed fastq, and other need uncompressed. Galaxy will transform the data if needed (you can see this in the history sometimes) with the implicit conversion functions.
Does this seem to be what is going on? Because this is happening with a few tools it makes be think it is at a lower level, and not the tool. If not, you are welcome to share a history with the error and I’ll review, and bring in EU people as needed. If those transformations have some issue it seems worth looking at. It could be related to the data size or something else.
All of these servers crunch through massive amounts of data! We just ran some statistics and it turns out that the public servers combined process an analysis job somewhere every single second! Tuning this is an ongoing process. 