# Large discrepancies between processing identical files on Org vs EU site **URL:** \ **Category:** usegalaxy.eu support\ **Tags:** workflow, upload, mapping, fastqgz, collections, collection-shape\ **Created:** [April 21, 2025, 4:24pm UTC](https://help.galaxyproject.org/t/large-discrepancies-between-processing-identical-files-on-org-vs-eu-site/15265 "2025-04-21T16:24:00Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![M.r.t](https://avatars.discourse-cdn.com/v4/letter/m/2acd7d/32.png) [@M.r.t](https://help.galaxyproject.org/u/M.r.t)\ **Post date:** [April 21, 2025, 4:24pm UTC](https://help.galaxyproject.org/t/large-discrepancies-between-processing-identical-files-on-org-vs-eu-site/15265/1 "2025-04-21T16:24:00Z")
Hello, I am very new to this, so forgive me if I’m missing something but I’ve come across massive discrepancies between processing the same files on the same supposed tools on the EU vs Org sites. The first issue I had was when trying to map paired fastq files on the EU site using “Map with BWA-MEM” on default settings. These files were part of a larger dataset I got from ENA, and every other paired files in the batch had been processed without issue. However, this one failed and I was presented an error. I repeated this twice and still ended up with an error. I tried to process this paired set using the same exact tool on the org site and to my surprise it was processed without issue. The second crazy discrepancy was when I processed another paired set on the same tool (on the EU site) and it was marked as completed without issue, but upon further inspection I saw that the resulting BAM file was much smaller than the other completed BAM files (2 GB vs the average 10-25 GBs). Again, I decided to process the same paired files on the same exact tool on the org site and the resulting BAM file was a much more reasonable 21 GBs. What’s going on? Why are there very large discrepancies using, allegedly, identical tools on the same settings? How can I expect to have a consistent QC when co-analyzing datasets processed on EU vs ORG? I’m not even sure I can rely on the EU site ---
**Author:** ![jennaj](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/jennaj/32/27_2.png) [@jennaj](https://help.galaxyproject.org/u/jennaj)\ **Post date:** [April 21, 2025, 5:37pm UTC](https://help.galaxyproject.org/t/large-discrepancies-between-processing-identical-files-on-org-vs-eu-site/15265/2 "2025-04-21T17:37:10Z")
Hi @M.r.t Sorry to hear that you have been having problems. The tools across servers should be exact. Maybe we can discover what the differences are so you can get the results synced up? Using a workflow would be one way to do this since all of the details would be in the workflow: tool versions, parameter settings, reference data. Then you would control the inputs. You could even move the history with the inputs across servers, intact, then run the workflow against both copies, and then expect identical results. Now, some tools will have non-deterministic algorithms, but the differences between runs even on the same server would be expected to vary a bit, too, not just the tool running on different cluster nodes, since time would be a variable, no matter where you are working. I would like to review some of the differences you found. You can share back the histories with the data. If you do not want to share those publicly, we can move into a private message here instead, and only add in other administrators as needed (I’ll ask first!). This error would be interesting to review: the history share link for both servers. Please note the datasets involved. I’ll be able to see the other details from just that. > [@M.r.t](#): > > The second crazy discrepancy was when I processed another paired set on the same tool (on the EU site) and it was marked as completed without issue, but upon further inspection I saw that the resulting BAM file was much smaller than the other completed BAM files (2 GB vs the average 10-25 GBs). Again, I decided to process the same paired files on the same exact tool on the org site and the resulting BAM file was a much more reasonable 21 GBs. And, this one would also be interesting to review. > [@M.r.t](#): > > However, this one failed and I was presented an error. I repeated this twice and still ended up with an error. I tried to process this paired set using the same exact tool on the org site and to my surprise it was processed without issue. * * * * * * How to generate a history share link is in the banner of this forum, also here → > [@How to get faster help with your question](https://help.galaxyproject.org/t/how-to-get-faster-help-with-your-question/11469/1): > > ## **How to get faster help with your question** > > We love details! 🧑‍🔬 Consider [**Sharing your History**](https://training.galaxyproject.org/training-material/faqs/galaxy/histories_sharing.html) or posting screenshots of your [**Job Information Details**](https://training.galaxyproject.org/training-material/faqs/galaxy/analysis_troubleshooting.html) page. > > Please confirm the [**Server URL**](https://training.galaxyproject.org/training-material/faqs/galaxy/analysis_troubleshooting_reporting.html) (your web browser at the top!) and any [**Tutorial**](https://training.galaxyproject.org/) links involved. Let me know if you want to post those into a chat message instead of here. The screenshots will probably not be enough for this one since we are looking at the deeper technical details too. Thanks! 🙂 ---
**Author:** ![M.r.t](https://avatars.discourse-cdn.com/v4/letter/m/2acd7d/32.png) [@M.r.t](https://help.galaxyproject.org/u/M.r.t)\ **Post date:** [April 21, 2025, 8:01pm UTC](https://help.galaxyproject.org/t/large-discrepancies-between-processing-identical-files-on-org-vs-eu-site/15265/4 "2025-04-21T20:01:20Z")
Hi @jennaj . A chat message will be better. I’m not sure how to send one here ---
**Author:** ![jennaj](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/jennaj/32/27_2.png) [@jennaj](https://help.galaxyproject.org/u/jennaj)\ **Post date:** [April 21, 2025, 8:08pm UTC](https://help.galaxyproject.org/t/large-discrepancies-between-processing-identical-files-on-org-vs-eu-site/15265/5 "2025-04-21T20:08:35Z")
I just sent you a message to get it started. I included instructions for how to share in there too. Find your messages at the very top right at this site, under your account icon. ![Screen Shot 2025-04-21 at 1.05.48 PM](https://us1.discourse-cdn.com/flex020/uploads/galaxy/original/2X/8/81995640f72fae46eb66f1eacf66c2892798b921.png) ---
**Author:** ![jennaj](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/jennaj/32/27_2.png) [@jennaj](https://help.galaxyproject.org/u/jennaj)\ **Post date:** [April 24, 2025, 5:49pm UTC](https://help.galaxyproject.org/t/large-discrepancies-between-processing-identical-files-on-org-vs-eu-site/15265/6 "2025-04-24T17:49:47Z")
Hi @M.r.t Thanks for sharing the histories! Very helpful. It looks like the reads from the pair didn’t load completely from the SRA at the [UseGalaxy.eu](http://UseGalaxy.eu) server. This then lead to the downstream result discrepancy when mapping. Some mapping tools may have trapped this but I wouldn’t count on that – always verify the read content yourself. Even if trapped, the error messages can be odd and unclear, but a dedicated QA tool will usually report something actionable. So, I was able to discover this by running some QA on the reads. The jobs failed, then I reviewed the job details view for the job [using the i-icon](https://training.galaxyproject.org/training-material/faqs/galaxy/datasets_icons.html). On those views, I saw these messages. ![Screen Shot 2025-04-24 at 10.25.49 AM](https://us1.discourse-cdn.com/flex020/uploads/galaxy/original/2X/2/27324c0a066e7b1f3fa5efadc028649e90498e0c.png) ![Screen Shot 2025-04-24 at 10.21.21 AM](https://us1.discourse-cdn.com/flex020/uploads/galaxy/original/2X/7/782926117a82fb533cec789c5dfecd92d6119897.png) Then, if I used the [bug-icon](https://training.galaxyproject.org/training-material/faqs/galaxy/datasets_icons.html), the error message could be interpreted by our new Galaxy AI assistant! It interpreted this really well! It is new so I wouldn’t expect it to be perfect yet for every error plus so much depends on how the original author trapped error cases. But anything unclear you can always ask about here still. 🙂 And anything _really odd_ that seems to be a server issue, you can go ahead and submit the bug report and the server administrators will see it. ![Screen Shot 2025-04-24 at 10.32.20 AM](https://us1.discourse-cdn.com/flex020/uploads/galaxy/original/2X/2/20d81f61b629c82290ed9bd3601500bd90ea97aa.png) > **Expand to read the text from the gxy chat bot directly** > > Hello jenj, > > The error message you’re encountering is from a tool known as **FastQC** , which is commonly used for quality checking of sequencing data in FASTQ format. Here’s a simplified breakdown of the error you are seeing: > > ### Error Explanation: > > - **Failed to process file `ERR6068440_2_fastq_gz.gz`:** This indicates that the tool encountered a problem when trying to process this specific FASTQ file. > - **SequenceFormatException:** This error suggests an issue with the format of the sequence data within your FASTQ file. > - **Midline ‘AATGCTGTCTCTC\ 1. A sequence identifier line beginning with ‘@’ > 2. The raw sequence letters > 3. A separator line beginning with ‘+’ > 4. A quality score lineThe error indicates that at byte position 39,603,447 (or somewhere in the file), a line expected to start with a ‘+’ doesn’t comply with this format, which is required as the third line for each sequence entry. > > ### Suggested Solution: > > 1. **Check File Integrity:** > > - Verify if the file got corrupted during download or transfer. You might want to redownload or decompress the file again if it was compressed, then re-upload it to Galaxy. > > 1. **Examine the Problematic Line:** > > - If you are familiar with command-line tools, you can inspect the file using a text editor or tools like `less` or `head` to navigate to the specified position and check if there is an unexpected character or misalignment in the file. > > 1. **Re-run the Tool:** > > - Once you’ve verified or corrected any issues with the file, try running the FastQC tool in Galaxy again. > > If these steps don’t resolve the issue, the file might require sequence-specific investigations to ensure all entries conform to the FASTQ standard. Should you need further assistance, consulting with your data provider or a colleague familiar with the sequence data format might be beneficial. > > I hope this helps guide you towards resolving the issue. If you have more questions, feel free to ask! * * * * * * **What to do** Whenever data is newly loaded up to Galaxy, run some simple checks to make sure the transfer happened correctly. That might just be inspecting the data to confirm the datatype but can also involve dedicated content checking tools. Fastq reads are sensitive to what is in the inside, too, the content, so assessing that is usually a good idea, along with manipulations like trimming. **More about Upload** → [Getting Data into Galaxy](https://help.galaxyproject.org/t/getting-data-into-galaxy/10868) Then, more about QA/QC is covered in this prior topic. The workflow at the end here would likely be a good fit for you. Getting the reads with the dedicated SRA read fetching will also add in some stability versus using free URLs, too, since that is handled a bit differently by the SRA data servers. That will also avoid potential content loss with multiple data transfer steps – example: from the cloud somewhere down to your computer, back up into a cloud resource. Cloud to cloud is one less hop. > [@Quality Control Start Here! multQC issue and guidance?](https://help.galaxyproject.org/t/quality-control-start-here-multqc-issue-and-guidance/14914/2): > > #### How to get from the SRA “paired list” output collection shape to a simple “list” collection shape? > > Use the `Faster Download and Extract Reads in FASTQ format from NCBI SRA` tool > > Then, when running `FastQC` on a paired end collection, it is useful to first apply the `Collection Operations -> Flatten Collection` tool. This assigns a unique sample identifier to each end of the pair – giving the sequences a distinct collection identifier. `MultiQC` will then provide the results for each sequence end, and summarize correctly. > > Single dataset QA example > > - [Hands-on: Quality Control / Quality Control / Sequence analysis](https://training.galaxyproject.org/training-material/topics/sequence-analysis/tutorials/quality-control/tutorial.html) > > Multiple paired end dataset QA example > > - [Hands-on: Reference-based RNA-Seq data analysis / Reference-based RNA-Seq data analysis / Transcriptomics](https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/ref-based/tutorial.html) (see the initial steps) > - More examples → [Galaxy Training!](https://training.galaxyproject.org/training-material/by-tool/iuc/sra_tools/fasterq_dump.html) > > A demonstration is in the **Quality Control Q20-L20** public workflow. > How to find it → > > > [@Trimming with different parameters/tools give vastly different end results.](https://help.galaxyproject.org/t/trimming-with-different-parameters-tools-give-vastly-different-end-results/14815/2): > > > > - Search here [GalaxyProject.org](http://GalaxyProject.org) → [GTN Pan-Galactic Workflow Search](https://training.galaxyproject.org/training-material/workflows/list) > > - With the term `Quality Control Q20-L20` > > **What to do** > > 1. Run the `Flatten Collection` tool on the output from `Faster Download and Extract Reads in FASTQ` > 2. Run `FastQC` on the result (a simple “list” collection format) > 3. Run `MultiQC` on the result. > > Or, you can try the workflow. You can customize the trimming step, or ignore it, or remove that step in your copy of the workflow. * * * * * * I hope this helps! We can follow up more about any questions you have, here or in the private chat. 🚀 XRef → [Search results for 'uk.ac.babraham.FastQC.Sequence.SequenceFormatException' - Galaxy Community Help](https://help.galaxyproject.org/search?q=uk.ac.babraham.FastQC.Sequence.SequenceFormatException) ---
**Author:** ![jennaj](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/jennaj/32/27_2.png) [@jennaj](https://help.galaxyproject.org/u/jennaj)\ **Post date:** [May 13, 2025, 11:57pm UTC](https://help.galaxyproject.org/t/large-discrepancies-between-processing-identical-files-on-org-vs-eu-site/15265/7 "2025-05-13T23:57:06Z")
Split this topic on May 13, 2025, 11:57pm UTC A post was split to a new topic: [Human genotype analysis](https://help.galaxyproject.org/t/human-genotype-analysis/15410) ---
**Author:** ![jennaj](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/jennaj/32/27_2.png) [@jennaj](https://help.galaxyproject.org/u/jennaj)\ **Post date:** [May 13, 2025, 11:56pm UTC](https://help.galaxyproject.org/t/large-discrepancies-between-processing-identical-files-on-org-vs-eu-site/15265/8 "2025-05-13T23:56:31Z")
Closed on May 13, 2025, 11:56pm UTC