I noticed an issue with some of my sequencing, I first noticed just looking at my forward and reverse reads, and saw that read numbers didn’t match on the forward and reverse. Some double checking and looking at my raw reads (which are sequenced on Element Aviti), just looking at the first handfull of reads and find them out of order.
I tried to fix this by converting to a Sorted BAM and then back to FASTQ. Read ordering looks perfect in the BAM file, yet after converting back to fastq Read2 looks like the same ordering as the BAM, but Read1 the order is messed up. I have tried various ways to fix, but it always comes back to the converting back to FASTQ that things get mixed up.
Any help on trying to fix this?? I’ve tried everything I can think of.
I think I found my fix. I ran FASTP with all options disabled. I believe no trimming is supposed to be done, but it did do some type of trimming as I have some shorter reads now. But the order looks ok.
It is rather uncommon that the original reads are not properly paired/sorted.
You can try FASTQ interlacer followed by FASTQ de-interlacer, or just interlacer and use reads in interlacer/interleaved format. The original tools are very slow on compressed data. Maybe look at modern alternatives. I recommend seqtk package, but it requires more steps for data processing. Look at:
seqtk_mergepe interleave two unpaired FASTA/Q files for a paired-end file
seqtk_dropse drop unpaired from interleaved Paired End FASTA/Q
seqtk_seq common transformation of FASTA/Q - with this tool deinterlacing requires two jobs, one with extract 2n-1 reads, another with extractr 2n reads
Yes, I thought it was strange too!!! My Assumption, was they were properly paired from the Lab. Since I found the issue on reads that I had processed, I went back to look at my original files. All of my 6 samples from Element Aviti sequencer had issues, my 2 samples from Novaseq 6000 looked fine.
For interlacer, wouldn’t that just take the reads and interlace unpaired reads???
I will try the seqtk tools, will the mergepe interleave find the pairs that aren’t in order??
I did process with FASTP with no options selected, that seemed to make things better, but not fixed. Just surprised that when I did a name sorted BAM that it wouldn’t keep the order as the BAM.
At the moment since I know several of the strains that are in my files, I’m mapping those initially at 98% match with Bowtie2, I’m assuming they will only match if the reads are properly paired? When I reduce the file size I will look again at getting proper paired.
interlacer takes two files with F and R reads, filter out unpaired reads and produce properly paired reads in interlaced format. It also writes “orphan” reads. It is one step operation, but the tool is very slow and it seems it is sensitive to input data.
As for the mapping step: I’d rather use properly paired data. Many tools have shortcuts for faster data processing, and may not handle non-standard data well.
I would like properly paired as well, it’s been driving me crazy, been having issues for a LONG TIME!!! Now I find it’s probably due to not having original properly paired reads.
So what mapping tools can I use interlaced reads??? I’m sure it’s easier keeping them in order if I keep them interlaced as much as possible.
I just did the interlace tool, how do I separate the properly paired ones?? Just a normal de-interlace, or is there a different program. I see a mix of interlaced ones, and ones that didn’t. I would like to try and repair the ones that didn’t properly interlace.
The interlacer returns several output files. One is with interlaced reads. This file contain properly paired reads. If you want F and R reads in separate files, use interlaced reads as input for deinterlacer. As for files with singletons - I expect the reads in these files do not have a pair. Of cause, Aviti data may have some features, so, check the results to make sure the tools produce proper outputs.
HiSAT2 and BWA-MEM2 can use interlaced/interleaved reads, but users should select this data format in job setup.
I like Hisat2 overall, I know BWA-MEM is popular, but as far as I understand, it’s local alignment only. Several of the strains I’m dealing with are Eukaryotic and very similar to human dna, many reads are only 1-2bp apart in similarity. I like BBmap as well, which takes interlaced reads, but is EXTREMELY slow.
I actually suspect most reads do have a pair, as my BAM file showed paired reads for ones that lost the pairing when converting to FASTQ. So I’m hoping that I can separate the properly paired ones, then trying to BAM sort the singles. We will see.
I’m literally getting nowhere here, everything I do to try and repair the paired ordering of my original R1/R2 fastq pair, still ends up with the same issues. If I take the ordering of Read1, Read2 should take the same ordering. This seems to show this in the BAM files. Then when converting back to fastq, I believe Read1 has the correct ordering, then Read2 starts to insert other reads into the order. It feels like there is some anchoring in the pairing from the original pairing, and perhaps pulling it back into what should be a new output.
Is there perhaps a tool that strips any tags that might be present, so that when I try to sort that it’s only using it’s current new pairing??
Looking online the only two tools that seem to mention this is BBmap-Repair, and FASTQ-Pair
Running the Seqtk tools should be able to sort this out for you in the fastq data.
Then, when mapping, some reads will map and some will not. You can filter the raw BAM file after mapping on the different flags you are interested in. You can retain proper pairs Yes or No with different filter runs to review subsets closer.
The ordering in the BAM is not intended to be reviewed directly since it is heavily encoded, despite some of it being in plain text (based on the fastq data) when uncompressed to SAM.
Does this actually help? It is hard to guess more without seeing the exact data.
Yes, I understand the tools you mentioned, Igor had mentioned Seqtk mergepe, which I tried and didn’t work. I read about the tool, it says reads have to be in the same order, since they aren’t in the same order, it didn’t work. Actually the Interlacer tool, actually says that reads don’t have to be in the same order, it will find them. Those that don’t pair will go to single. No tools threw out any singles, as both reads have the same number.
When I converted to BAM and either sorted or collated, and viewed the first part, all looked paired. But upon converting to FASTQ, Read1 had the same order as the BAM and Read1 lost it’s pairing.
Mapping with Hisat2, I would get mapped reads that had different read numbers for Read1/Read2.
@@@ Ok, I can kinda see what the issue is, but it doesn’t make any sense.
I know that Read1 and Read2 should be in the same order, as that’s the order the tools take them from. So when I compare Read1 and Read2, let’s assume that Read1 is in the correct order, but Read2 is not in the same order. I only need to go down about 5 reads until they don’t match.
To make it easier to see, I used the tools “Select first lines”, so I took 200 lines for 50 separate reads. First from Read1, next from Read2. These SHOULD be in the same order as the original Read1 and Read2, correct???
Here is what happened, I should now have two sets of 50 reads in the same order as Read1 & Read2. What I actually got is two sets that are NOW in the same order??? This makes zero sense, but I think whatever is happening this is causing the issue.
What’s weird, is that this is only on the Element Aviti reads, where I have 3 sets from one lab, and 4 sets from a different lab.
I’m going to attach the history here to show what I did, perhaps you can make sense of this??
I just did another test, along the same lines as I just mentioned.
I converted the FASTQ to Tabular, everything looked great, both reads appeared in order. Then upon converting back to FASTQ then the read order converted back to the original out of order. I did those on a different history, so I will move them over to the one I shared.