Hello there,
I need to use galaxy.org for features it offers compared to galaxy.eu, however, the dataset I am working with is larger than 1.2 TB. Is there a way to connect to my google cloud storage as I have it set up on my galaxy.eu account.
Hello there,
I need to use galaxy.org for features it offers compared to galaxy.eu, however, the dataset I am working with is larger than 1.2 TB. Is there a way to connect to my google cloud storage as I have it set up on my galaxy.eu account.
Welcome @ticecm
Some of the data resource connections are currently exclusive to the EU server. This includes external data storage, but this will change over time.
For now, you can move data between the servers by URL. This means you can use the EU server as a sort of portal to your external resources by sending data from other public Galaxy server to/from it.
However, a single file over 1 TB is very large! The EU server offers a different type of computational cluster, and it can sometimes scale to run much larger jobs. This may mean that moving the data over to the ORG server might later result in failed jobs due to the resource differences, even if the data storage was scaled larger (on the server itself, or with external storage).
If the data size is actually a history, and not an individual file, you could move just the data you want to process over? The ORG server has a 1 TB temporary storage space for larger work.
In short, a single file that is 1 TB would be unlikely to process at the ORG server, even if it processes at the EU server.
Please let me know if you have any questions about this!
Hello there! @jennaj
Thank you for your response. It is not one single file over 1TB! I am sorry if it was interpreted that way. I have around 150 WGS genomes that are causing it to go over that threshold. Since this is the case, do you think it would be best to load it on the EU server and then move it to the server?
Thank you!
Hi @ticecm
Ok, glad it was not a single 1 TB file!
Getting the data up into a cloud environment makes it easier to move it around to other cloud services since you can do that with URLs. You can certainly use the different Galaxy servers as staging areas.