Setting up temporary storage on a private Galaxy instance

1

I administer a private Galaxy instance. Looking around I see that since last year the public instances have an option to set up temporary storage locations, where data gets deleted after a certain amount of time, together with a permanent storage for user results. I would like to use this for my own server, since I believe it would help cut down on user storage quite a bit.

Has this capability been ported to the public Galaxy releases? Is there any documentation on setting this up?

2

Welcome @Empir

Yes, as an administrator you can set up multiple local storage locations then set custom rules for how long the data persists.

Administration docs → Connecting Users and Data — Galaxy Project 24.2.1.dev0 documentation

Keep in mind that data can be technically on the same physical location but have metadata designating it for either permanent, temporary or scheduled persistence.

Examples:

  • UseGalaxy.org offers two levels of storage. One is permanent and the other is temporary.

  • UseGalaxy.org.au offers everything under a longer default persistence, then will begin removing older data on a schedule. They send out emails to warn users about pending data deletions (touching the data resets the clock).

  • UseGalaxy.eu has a schedule where everything is persistent, unless the entry point was through a training link. Any account involved in a training session without a later return session is removed – the account itself and the data.

Hopefully this helps and you can ask follow up questions! :slight_smile:

Xref

3

Thanks for the link. I found this initially when searching around for this feature, but I was unable to find any actual details on that page on how to set up a temporary (sub-)storage. Searching the page for “temporary” or “persistent” or variations does not return anything.

Reading a bit more carefully through that page and the various linked config files, my understanding is that with those object store definitions we can set up different storage locations for the users, declare one of them to be non-persistent (in name only), but the actual cleanup of data needs to be done in the backend with gxadmin? As detailed in Redirecting… .

I think I see how to at least test this now, but I think it would be good to offer some clean documentation on setting up such a temp storage. It’s a very useful feature I am sure other Galaxy admins could make use of, but it’s hard to piece together. As it is the “feature” is unfortunately split across at least 3 different documentation pages.

1 Like
4

Hi @Empir

Glad you are able to make sense of this, and yes, I agree that there could be a sort of developer note to demonstrate how all of this could come together for the short term storage example.

I can think of two ways to get this shared:

  1. Write up the example and propose it as an addition directly in the docs. The developers would help to tune this. Your PR would go against here → galaxy/doc at dev · galaxyproject/galaxy · GitHub

  2. Write up a blog post about your experiences and how those were added to your server. This could even be linked from the docs, e.g. an example local server implementation. See → The Galactic Blog - Galaxy Community Hub

The second here is probably a good place to start. It would get the discussion started, and we’ve had developer blog content promoted into the core docs before. :slight_smile:

5

Yeah I realise it’s not great etiquette to suggest “there should be some documentation on this” and do nothing else myself :slight_smile: . This is a bit at the bottom of my priority queue right now, but it was important to know if it was possible for deployment reasons. When I get around to it, I can look into turning it into a short tutorial as you mention, I think it would be good (if the Galaxy team doesn’t beat me to it). But it will take some time. At least there is this thread now in case somebody else goes looking for the same information.

1 Like