Large discrepancies between processing identical files on Org vs EU site

[jennaj]

Hi @M.r.t

Thanks for sharing the histories! Very helpful.

It looks like the reads from the pair didn’t load completely from the SRA at the UseGalaxy.eu server. This then lead to the downstream result discrepancy when mapping. Some mapping tools may have trapped this but I wouldn’t count on that – always verify the read content yourself. Even if trapped, the error messages can be odd and unclear, but a dedicated QA tool will usually report something actionable.

So, I was able to discover this by running some QA on the reads. The jobs failed, then I reviewed the job details view for the job using the i-icon. On those views, I saw these messages.

Then, if I used the bug-icon, the error message could be interpreted by our new Galaxy AI assistant! It interpreted this really well! It is new so I wouldn’t expect it to be perfect yet for every error plus so much depends on how the original author trapped error cases. But anything unclear you can always ask about here still. And anything really odd that seems to be a server issue, you can go ahead and submit the bug report and the server administrators will see it.

Expand to read the text from the gxy chat bot directly

Hello jenj,

The error message you’re encountering is from a tool known as FastQC, which is commonly used for quality checking of sequencing data in FASTQ format. Here’s a simplified breakdown of the error you are seeing:

Error Explanation:

• Failed to process file ERR6068440_2_fastq_gz.gz: This indicates that the tool encountered a problem when trying to process this specific FASTQ file.
• SequenceFormatException: This error suggests an issue with the format of the sequence data within your FASTQ file.
• Midline ‘AATGCTGTCTCTC<TTCTTTCA9:8186)CCTTTGGTCCTGTAGGTCACAACTGTGTTTAT’ didn’t start with ‘+’ at 39603447: In a FASTQ file, each sequence entry consists of four lines:
  1. A sequence identifier line beginning with ‘@’
  2. The raw sequence letters
  3. A separator line beginning with ‘+’
  4. A quality score lineThe error indicates that at byte position 39,603,447 (or somewhere in the file), a line expected to start with a ‘+’ doesn’t comply with this format, which is required as the third line for each sequence entry.

Suggested Solution:

1. Check File Integrity:

• Verify if the file got corrupted during download or transfer. You might want to redownload or decompress the file again if it was compressed, then re-upload it to Galaxy.

2. Examine the Problematic Line:

• If you are familiar with command-line tools, you can inspect the file using a text editor or tools like less or head to navigate to the specified position and check if there is an unexpected character or misalignment in the file.

3. Re-run the Tool:

• Once you’ve verified or corrected any issues with the file, try running the FastQC tool in Galaxy again.

If these steps don’t resolve the issue, the file might require sequence-specific investigations to ensure all entries conform to the FASTQ standard. Should you need further assistance, consulting with your data provider or a colleague familiar with the sequence data format might be beneficial.

I hope this helps guide you towards resolving the issue. If you have more questions, feel free to ask!

---

---

What to do

Whenever data is newly loaded up to Galaxy, run some simple checks to make sure the transfer happened correctly. That might just be inspecting the data to confirm the datatype but can also involve dedicated content checking tools. Fastq reads are sensitive to what is in the inside, too, the content, so assessing that is usually a good idea, along with manipulations like trimming.

More about Upload → Getting Data into Galaxy

Then, more about QA/QC is covered in this prior topic. The workflow at the end here would likely be a good fit for you. Getting the reads with the dedicated SRA read fetching will also add in some stability versus using free URLs, too, since that is handled a bit differently by the SRA data servers. That will also avoid potential content loss with multiple data transfer steps – example: from the cloud somewhere down to your computer, back up into a cloud resource. Cloud to cloud is one less hop.

@jennaj

How to get from the SRA “paired list” output collection shape to a simple “list” collection shape?

Use the Faster Download and Extract Reads in FASTQ format from NCBI SRA tool

Then, when running FastQC on a paired end collection, it is useful to first apply the Collection Operations -> Flatten Collection tool. This assigns a unique sample identifier to each end of the pair – giving the sequences a distinct collection identifier. MultiQC will then provide the results for each sequence end, and summarize correctly.

Single dataset QA example

• Hands-on: Quality Control / Quality Control / Sequence analysis

Multiple paired end dataset QA example

• Hands-on: Reference-based RNA-Seq data analysis / Reference-based RNA-Seq data analysis / Transcriptomics (see the initial steps)
• More examples → Galaxy Training!

A demonstration is in the Quality Control Q20-L20 public workflow.
How to find it →

@jennaj

• Search here GalaxyProject.org → GTN Pan-Galactic Workflow Search
• With the term Quality Control Q20-L20

What to do

1. Run the Flatten Collection tool on the output from Faster Download and Extract Reads in FASTQ
2. Run FastQC on the result (a simple “list” collection format)
3. Run MultiQC on the result.

Or, you can try the workflow. You can customize the trimming step, or ignore it, or remove that step in your copy of the workflow.

---

---

I hope this helps! We can follow up more about any questions you have, here or in the private chat.

XRef → Search results for 'uk.ac.babraham.FastQC.Sequence.SequenceFormatException' - Galaxy Community Help