Uploaded files depleted memory on Galaxy Cloudman?

, , ,
#1

Hello Galaxy community,

I was trying to upload my sequencing data (~2.3 Gb/file x 32 files=73 Gb) to my Galaxy on Cloudman 2.0 on an EC2 instance. The uploading process stoped and failed when ~ 75% of the files were done, showing " Warning: Please make sure the file is available. (500)". I check the “cluster status” page, it showed that the “memory cached” kept increasing and depleted the memory of the instance. However, the disk space usage did not grow at all. I guess the files were kept in the memory instead of being written into the disk storage, thus depleted the memory. I wonder how to solve this problem?

I am new to galaxy and really appreciate it if anyone can help!

1 Like
#2

I’ll try to replicate this but in the meantime, something to try would be to upload fewer (eg, 5) files at once instead of all 32.

Your comment also made us realize we do not expose an option to set the size of disk for launched instances. By default this is 90GB, which would likely be a problem for you if you have 73GB of input data. We’ll fix that as well and post here.

BTW, did you launch the GVL or the CloudMan 2.0 appliance from CloudLaunch? I’d suggest using the GVL one in the future (the CloudMan one is largely considered a development version).

2 Likes
#3

Thank you Enis! I will try to load less files each time. It happened on Cloudman 2.0, where I set the root volume to 300GB but the problem still occured. Maybe I confused the root volume and the storage?

1 Like
#4

The root volume is not actually used for storing Galaxy data so that would unfortunately not help. We’ll roll out an updated version in the next few days.

2 Likes
#5

Hello, I am having this very same problem, trying to upload fastq.gz files onto Galaxy (which is running off of GVL) and have only uploaded half my files and keep getting the same error message " Warning: Please make sure the file is available. (500)".
Its not the files as I can load on the use.galaxy.org no problem, and I should have plenty of space (allocation is 1000GB). Please help!

1 Like
#6

Heh, I did say we’d make it possible to set the size of Galaxy disk within a few days… Yet, here we are a few months later and this still has not been done.
Unfortunately, I don’t have a solution for you at this point. I’ll see tomorrow if we can commit to a timeline and post an update here.

1 Like
#7

Hey any progress?

#8

Well, not actual progress, but at least we know we won’t be able to add this feature before next week at the earliest. Sorry for the delay.

1 Like
#9

:frowning: thanks for the update tho!

1 Like
#10

Please forgive my persistence, but I am at a standstill with analysis until I can upload files. Has there been any movement?

1 Like
#11

This is still a work-in-progress