I’m having issues since yesterday. Seems maybe there were some server errors yesterday that caused some format issues on my files that I couldn’t figure out how to fix. So started from scratch again today. Now it seems none of my workflows are working, I try to invoke them and they either do nothing, or will result in errors. I expected to see some issues posted, but haven’t seen anything.
Facing the same issue here!
1 Like
They did an upgrade on the system yesterday.
Since then - nothing is working.
The new generated data is not recognized as the correct type. So it can not be used for the next step.
The workflow is not working under the new interface. If you run the same command from tools and configure it manually - it may work.
So in general all kind of bugs.
I LOVE upgrades!!!
1 Like
HI all!
Things should have been more steady today. We had a smaller cluster issue time with the update but all of that is resolved now.
Please let us know if you still notice anything odd and we can followup! That includes anything new you see in the UI and would like feedback about. The release notes are still being developed but I can share that back here still – or post supplemental help of course! 
I still have problems with the new update. Just running simple tasks even. Just now I was TRYING to concatenate a couple of files, and my output was “0”. So many things are problematic, and not sure what is going to run correctly. I’m currently trying to do most things on usegalaxy.eu
I would be interesting in reviewing that job. The release doesn’t impact individuals tools, just the application around those tools to host them. We can address whatever you may find the usual way. Shared histories are the fastest way to review.
Here is the concatenate one
1 Like
Hi @Jon_Colman
Thanks for sharing the history. I see your issue, but when I rerun the jobs, they hit a different cluster and the work is successful. This is happening mostly by chance and is not a new situation but could be trapped better.
The compressed fastq data is not expected to be used with this tool. The tool is designed for plain text (any datatype), but might be able to handle compressed fastq or other compressed dataypes in some situations (on certain clusters). We cannot limit the tool to those “handle compressed fastq” clusters due to how often it is used (would explode the wait time when anyone attempted to use it).
Trapping this situation is difficult – but we do try to give some warnings on the tool form. The Concatenate function is very simple: any data in a text format, it stacks input files on top of eachother, and outputs it. Context checks are not involved at all (on purpose!). This utility is different from most other tools that are screening based on file types and providing some simple checks to ensure the inputs are appropriate and the output is expected.
One alternative is this tool if you are trying to put the ends back together into the same file. This will create an “interleaved” fastq format. Then you can split the ends out again with other Seqtk tools, place the reads into a paired end collection, etcetera as needed.
- seqtk_mergepe interleave two unpaired FASTA/Q files for a paired-end file
- Handles any fastq data, compressed or uncompressed formats
The other solution is to uncompress the fastq data first. That will create plain text fastq files that can be used with the original Concatenate tool, and any cluster will understand how to process the uncompressed data (no intermediate manipulations).
For now the best advice I have is to try a rerun with one of those options. If you have any other situations, you are welcome to share those here and I’ll take a look.
I don’t quite understand on this, as I look at the concatenate tool concatenate (CAT) one, it does list fastq.gz as an input so I assume fastqsanger.gz also works. On this I was trying to just take host reads that I ran on different tool settings to combine back into one set.
Another item that I don’t know if it’s an issue for me or not. So taking a pair of reads and using BBmerge to ONLY error correct the reads without merging, this appears to work fine, and gives me an interleaved set in an order starting with read_R1 followed by read_R2. But then I de-interleave to bring them back into separate forward and reverse, but it changes the order giving me I believe read_R2 /r1 and read_R1 /r2 ?? But if they are first extracted from a BAM file which changes the read_R1 to read /r1 and read_R2 to read /r2 then they deinterleave normally. Does this mater that they get reversed??? Or am I using the wrong tool?
The other nightmare that I just realized when using a mapping program with a BAM output and then having to work to convert back to fastq, I was using samtofastq and my assumption that Mapped + Unmapped would equal the same amount of reads that I started with. I just had to start over from scratch again. I’m using the FASTX BAM to interleaved fastq now, which seems to retain my reads.
I also found that when I used mapping with BWA-aln for short reads, that it didn’t keep my reads in a pair, if single reads aligned it would only pull those. Is this only an issue with BWA or does Minimap2 do this too???
Thanks Jennifer
1 Like
Hi @Jon_Colman
This is all great feedback. I’ll go through these and open tickets as needed, or explain what is going on if expected. If you have any tiny examples for these it will speed things up ..although I think I understand what each is and can create those (I do this all the time anyway as part of my role). Some of these details are how the 3rd party tools “work” but let’s be sure and come up with parameter combinations, or a different tool choice when possible.
For the concatenate tools, there are several base tool versions and some do not specify the input format (the most expansive flavor). One of these is probably a better choice in your workflow give the input datatype (fastqsanger.gz, a subtype of fastq.gz). I’ll be sure to include this review but wanted to let you know in case you can identify it first and not wait on me. These tools are a bit special and differ in small but meaningful ways. I’ll try to clarify for your use case.
I’ll try to get to these this week. I don’t think there are bugs exactly but we can confirm. I want to fully resolve your issues here. More soon. 