There is a problem with computematrix (and maybe also other tools) when you choose a bedgraph file as an input. Galaxy will let you choose a “bedgraph (as bigwig)” file as an input but will then fail to run and will tell you that your input dataset is in an error state. At first I thought that something was wrong with my bedgraph dataset that came out of MACS2 but I found that if I just convert to bigwig format separately first (with Wig/Bedgraph to bigwig) then computematrix will run. So now that I know this, it isn’t a big problem since conversion from bedgraph is straightforward, but other people choosing bedgraph as bigwig files as inputs may be confused by the ambiguous “error state” message and not know how to resolve this. Previously (as recently as a few months ago), I could use a bedgraph (as bigwig) file as an input and there would not be an error so something seems to have changed since then. It would be better to have tools that require bigwig inputs not show bedgraph files on the available score file list at all.
Hi @TTP
This is interesting behavior. Is it still reproducible? I wondering if the converter tool is missing or has some new bug!
If you want to isolate and share back an example history with the problem, we can confirm and reach out to the administrators to get it fixed.
As an aside, I’ve seen something similar but it wasn’t a problem with the converter program specifically – it was a problem about a missing “database” assignment. The database key is what links a dataset to a fasta index, and that is required for the auto-conversion.
If Galaxy doesn’t host your genome, you can create a custom database key (aka “custom build”) and it works the same way. https://training.galaxyproject.org/training-material/faqs/galaxy/#reference%20genomes
Yes, other people I know have had a similar problem. I just ran another job to illustrate this:
[
](https://usegalaxy.org/u/tpaull/h/hb155-to-hb174)
The other problem that is visible in this history is that I am getting a lot of “used more memory than it was allocated” errors these days. I am not sure how to get around this. These are ChIP datasets and the problem seems to be the very large size of the ChIP input datasets.
Thanks,
-Tanya
Thanks, I’m reviewing.
Odd … this should be happening less frequently given recent boosts in resources. I see an example in your history so will do a check for that as well 
OK, this is what is going on:
- The version of the converter available from the tool panel includes an option to trim “overhanging coordinates”, instead of failing. It is toggled on by default.
- The version of the converter available from the “auto-convert” between the datatypes does not include a trimming function by default. Why? Because that would introduce a data change that is not explicitly exposed/chosen by the user. Converters are convenience functions … and never directly change data content, on purpose.
- MACS can output coordinates that can extend past the chromosome ends. This is a known “feature” of MACS, and everyone would have to adjust for that with certain functions or downstream tools. This is definitely not just in Galaxy. Review this discussion at the MACS forum if you are curious. The tool they recommend is the same one used in Galaxy, and the trim option is available in the version from the tool panel, but not in the auto-convert method’s version (again, on purpose).
What to do:
- Extract a workflow from one of your successful histories, and make sure that it includes the direct conversion step that includes the trimming. Or, add it directly to a workflow by editing it.
- Use your workflow. It will work just like a single tool if you “favorite” it and hide the intermediate outputs, and will even show up in the tool panel.
Workflows are much easier than people assume, and will save you even more time, over time, especially since you are doing the same thing across a bunch of samples. You could even just organize and tag all of your samples at once, then run the workflow once on the whole thing. Or, keep it simple between the pairs like you are doing now. Either is fine.
Now, your files are large. If the direct conversion fails for memory reasons when using the trim option, and the database assignment is actually correct, then you can trust the error message about runtime memory. Some data will actually exceed public computational resources of course! A scaled up private server is a common solution. See the Galaxy Platform directory for options – there are both free and pay-for-use choices (apply your grant funding, etc).
Hope that helps!
Thanks for this response. I am now always converting from the MACS2 output bedgraph to bigwig format directly using Wig/Bedgraph to bigwig, but today somehow the tool seems to have disappeared. There is a tool called “Convert Bedgraph to bigwig” but this is giving an error with my input files. Could the original conversion tool be put back?
Hi @TTP
First, if you are working at UseGalaxy.org, please try a rerun now. The server was undergoing updates earlier and a few jobs randomly failed.
Would you please clarify the following, then we can follow up about what might be going on? Tools shouldn’t be lost of course!
- The server URL where you are working.
- The full tool name/version of the prior tool you were using.
- The full tool name/version of the new tool that is failing, and all of the job logs (“i” icon will have all three, not the bug icon). If any logs are in the dataset as shown in the history, include that too please or screenshot the expanded dataset (contains an optional 4th log).
- If you are using a workflow, consider sharing it as well.
Xref → Troubleshooting resources for errors or unexpected results (how to do everything mentioned)
Let’s start there, thanks!
Thanks. I am using usegalaxy.org and saw problems earlier but more recently it seems to be working fine apart from this problem.
Previous tool was Wig/BedGraph-to-bigWig converter. Where did this go?
tool that is failing is Convert Bedgraph to Bigwig.
There were other errors earlier today, but the current error is a memory allocation. This doesn’t make sense though since a much larger file went through before this (2.3Gb) while this file that failed to convert was only 369 Mb. Seems like there is something else that is wrong here. The previous converter would not fail randomly like this, so if that were available, I would use that instead.
I am using a custom genome build here but the file before this that worked also used this build.
Hi @TTP
Ok, I have more details.
- We have reorganized some of these functions to make the usage clearer
- As part of that, the tool you were using was hidden from the tool panel (on purpose)
- But since that results in a loss of functionality (overhang coordinate clip), we are installing it fully again. It should be available again in the tool panel by the end of the week, possibly today. Ticket Install wig_to_bigwig from TS by mvdbeek · Pull Request #698 · galaxyproject/usegalaxy-tools · GitHub
- If you are in a rush, the EU and AU servers still have the tool loaded.
Thanks for following up and explaining!
I have checked a few times but this tool is still not findable through its previous name; is it there under another name? Previously called Wig/BedGraph-to-bigWig converter
Thanks
Hi @TTP
This version of the tool should work. Be sure to assign a database to your datasets, and to use the advanced options to clip the overhang sequence.
wigtobigwig bedGraph or Wig to bigWig converter link to tool
Is it possible to increase the memory allocation threshold for the wigtobigwig tool? I get this memory error:
Hi @TTP
Yes, maybe, but we’ll need the exact example.
Please send in a bug report from that error, and include a link to this topic discussion in the comments. We’ll review and let you know via email.
Hi- I have been using “Convert BedGraph to BigWig” with MACS output bedgraph files recently and now the tool seems to have disappeared again. ?? This happened a few months ago and then was restored.
Thanks
Hi @TTP
I see both tools at UseGalaxy.org today:
One with the option for trimming overhanging coordinates (the version that is best to use after MACS2) → https://usegalaxy.org/root?tool_id=toolshed.g2.bx.psu.edu/repos/iuc/ucsc_wigtobigwig/ucsc_wigtobigwig/447+galaxy0
And one with the direct conversion (can produce errors with sensitive downstream tools, if the data is from MACS2) → https://usegalaxy.org/root?tool_id=CONVERTER_bedgraph_to_bigwig
And for the error about the memory allocation – I can’t remember if you sent something in? Did you include a link to this topic? That will help me to associate the two. If you want to send that in again, I’ll review again
Hope this helps!
Thanks; yes the tool is back again. Not sure why it was briefly missing. Everything working fine now.
2 posts were split to a new topic: MACS2 bdgcmp troubleshooting