Memory Not Clearing

I’m having memory issues. My console shows 220 GB being used, yet I look at my Active Storage and it only shows 124 GB. I’ve cleared everything and deleted everything I can, but it won’t clear. When trying to “refresh” the memory after clearing, the button isn’t doing anything. User ID jpcnorthwest

Hi @Jon_Colman

Try giving this a full day to clear. The data purging processes run in the background. Refresh can help this to happen quicker but it is not immediate.

Meanwhile, remember that you have access to all of the UseGalaxy servers and your account at each has a distinct data storage allocation. An overview is here if this interests you. → What should I do if my data exceeds the given 250GB of storage?

If this doesn’t clear by tomorrow, please let us know and we can help you to look into this more. Confirming you are working at the UseGalaxy.eu server is a good place to start. Thanks! :slight_smile:

Ok, I’m moving some of my stuff to the Usegalaxy.org

A question on compressing files. When compressing a fastqsanger file on Usegalaxy.org, the standard tool just changed to compress, which just compresses to a .gz file, which I think have to change to fastqsanger.gz. Is there a reason the other tool was removed??

Hi @Jon_Colman

You can simply convert a fastqsanger file to a fastqsanger.gz file. It creates another dataset in the history.

Or, you can copy all of the files you want to move into a new history, compress the history to a file, then move that over instead. Since you are short on space, this is probably the better option. Why? Identical copies of files in your account do not take up more space, and the temporary compressed version of the history takes up no extra space – instead it just expires after a day or so.

Now, the larger the history the longer this will take to move over, so just keep that in mind. Fewer or no deleted datasets helps, that’s why I am suggesting to copy the files into a new fresh history.

With that context, I am not sure I understand your question. Do you want to share some screenshots to explain?

If you are asking about the datatype detected when moving a single file between servers, that is hard to answer, but in general anytime you are loading a file – including via link from another server – the datatype is detected. If the “guess” is wrong, that indicates a problem. Maybe the file was really not the type you throught it was, or the data is a complex type that would benefit from specifying the format during upload (very rare, and would be unexpected for read data).

The great part about moving an entire history is that everything is moved over with the original context: datatype names, the datatype formats, collection structures, all of it. I use this function nearly every day for all of those reasons, back and forth between all of the UseGalaxy servers. For histories with just one or two datasets, and with several hundred. You won’t need to do any extra compression steps – the history compression already applies the smallest size. Then it all uncompresses back to however the original state was.

Hope this helps and you can ask followup questions! Learning how to use all of the servers is something everyone can take advantage of. :scientist:

Image1 shows the compress files option on usegalaxy.org, this takes a fastqsanger file and changes it to “gz” only, not fastqsanger.gz.

Image2 shows compress files option on usegalaxy.eu, this does the expected fastqsanger to fastqsanger.gz, with also options to decompress. No option to decompress files on galaxy.org, both sites used to be the same as on usegalaxy.eu

Image3 shows my issue with storage, showing I have over 250GB, which I don’t have. When trying to refresh the storage it shows a server error.

Image4 shows what is actually in my history. After sitting overnight, there are no changes. So even though I deleted files, they aren’t coming off my current allocation.

Hi @Jon_Colman

Find the convert options under the pencil icon. Then if you use the rerun button after, you’ll see the convert tool used. I’ll also compare the tools listed in the tool panel, maybe something changed and we can address that, but you shouldn’t need to wait.

Then for your account/data issue, we can ask the administrators to see if there is a known issue. Your username is probably enough for them to find your account. Find this under User → Preferences → Manage information.

Ping @wm75 or @bjoern.gruening



How to check for deleted data

Starting at User → Preferences → Storage Dashboard

Go to Manage your account storage

Is this only happening since 24h? I will have a look.

Yes, approximately 24 hours

This should be fixed now! We had a problem with our MQ/celery cluster and not all tasks have been processed. This should now be fixed.

Thanks for reporting!

1 Like

Thanks Bjoern, memory looks cleaned now!!
One other thing, I was running Kraken2 on usegalaxy.org and transferring the Kraken Classification and Kraken Report to another server. For some reason the report is being imported as an interval file???

Hi @Jon_Colman

Glad this is all working!

For this part

You can try assigning the datatype format to prevent the file format detection from getting this wrong.

Have you tried assigning the same type as was in the original history? If not, I’d suggest trying that next. :slight_smile:

Yes, I did that, just wanted to mention it as a possible issue.

1 Like

Ok, thanks @Jon_Colman for following up.

I’m going to look into that specific example more. Maybe we can enhance the file format detection to have this fall back to the less specific tabular type instead of greedily attempting to make it fit interval. I’ll post the ticket back here.

I’m not sure why it’s doing that, it didn’t do that before. It’s even named as a tabular file when it transfers

It has to do with the way new data files are “sniffed” to detect the datatype.

It is seeing columns with content that resembles chromosome names and coordinates. I’m investigating what may have changed and if/why the servers are acting differently. We have some tutorials with warnings but I thought this was addressed. We had some recent updates that may be impacting this.

For now, you can assign the datatype directly. Or, if you are moving entire histories, the files will restore on the other server however they were labeled on the origin server. This is one reason why I prefer that method – the original file name, datatype, collection structure – all is preserved.

More soon. Thanks for alerting us about the oddness. :slight_smile: