I noticed an issue with some of my sequencing, I first noticed just looking at my forward and reverse reads, and saw that read numbers didn’t match on the forward and reverse. Some double checking and looking at my raw reads (which are sequenced on Element Aviti), just looking at the first handfull of reads and find them out of order.
I tried to fix this by converting to a Sorted BAM and then back to FASTQ. Read ordering looks perfect in the BAM file, yet after converting back to fastq Read2 looks like the same ordering as the BAM, but Read1 the order is messed up. I have tried various ways to fix, but it always comes back to the converting back to FASTQ that things get mixed up.
Any help on trying to fix this?? I’ve tried everything I can think of.
I think I found my fix. I ran FASTP with all options disabled. I believe no trimming is supposed to be done, but it did do some type of trimming as I have some shorter reads now. But the order looks ok.
It is rather uncommon that the original reads are not properly paired/sorted.
You can try FASTQ interlacer followed by FASTQ de-interlacer, or just interlacer and use reads in interlacer/interleaved format. The original tools are very slow on compressed data. Maybe look at modern alternatives. I recommend seqtk package, but it requires more steps for data processing. Look at:
seqtk_mergepe interleave two unpaired FASTA/Q files for a paired-end file
seqtk_dropse drop unpaired from interleaved Paired End FASTA/Q
seqtk_seq common transformation of FASTA/Q - with this tool deinterlacing requires two jobs, one with extract 2n-1 reads, another with extractr 2n reads
Yes, I thought it was strange too!!! My Assumption, was they were properly paired from the Lab. Since I found the issue on reads that I had processed, I went back to look at my original files. All of my 6 samples from Element Aviti sequencer had issues, my 2 samples from Novaseq 6000 looked fine.
For interlacer, wouldn’t that just take the reads and interlace unpaired reads???
I will try the seqtk tools, will the mergepe interleave find the pairs that aren’t in order??
I did process with FASTP with no options selected, that seemed to make things better, but not fixed. Just surprised that when I did a name sorted BAM that it wouldn’t keep the order as the BAM.
At the moment since I know several of the strains that are in my files, I’m mapping those initially at 98% match with Bowtie2, I’m assuming they will only match if the reads are properly paired? When I reduce the file size I will look again at getting proper paired.
interlacer takes two files with F and R reads, filter out unpaired reads and produce properly paired reads in interlaced format. It also writes “orphan” reads. It is one step operation, but the tool is very slow and it seems it is sensitive to input data.
As for the mapping step: I’d rather use properly paired data. Many tools have shortcuts for faster data processing, and may not handle non-standard data well.
I would like properly paired as well, it’s been driving me crazy, been having issues for a LONG TIME!!! Now I find it’s probably due to not having original properly paired reads.
So what mapping tools can I use interlaced reads??? I’m sure it’s easier keeping them in order if I keep them interlaced as much as possible.
I just did the interlace tool, how do I separate the properly paired ones?? Just a normal de-interlace, or is there a different program. I see a mix of interlaced ones, and ones that didn’t. I would like to try and repair the ones that didn’t properly interlace.
The interlacer returns several output files. One is with interlaced reads. This file contain properly paired reads. If you want F and R reads in separate files, use interlaced reads as input for deinterlacer. As for files with singletons - I expect the reads in these files do not have a pair. Of cause, Aviti data may have some features, so, check the results to make sure the tools produce proper outputs.
HiSAT2 and BWA-MEM2 can use interlaced/interleaved reads, but users should select this data format in job setup.
I like Hisat2 overall, I know BWA-MEM is popular, but as far as I understand, it’s local alignment only. Several of the strains I’m dealing with are Eukaryotic and very similar to human dna, many reads are only 1-2bp apart in similarity. I like BBmap as well, which takes interlaced reads, but is EXTREMELY slow.
I actually suspect most reads do have a pair, as my BAM file showed paired reads for ones that lost the pairing when converting to FASTQ. So I’m hoping that I can separate the properly paired ones, then trying to BAM sort the singles. We will see.