Hi all,
I’m trying to figure out how can I move multiple files from regular storage to scratch. I can go to each dataset individually and change storage location in the options there, but It becomes unrealistic for dozens of datasets…
Welcome @Evgeni_Bolotin
Yes, you are correct. Moving many datasets between storage locations is super tedious and unfortunately we don’t have a batch method for this yet.
This is one of our summaries for optimizations we are working through. See “Batch Move Between Object Stores”.
I wish we had some workaround to offer! The first time through this after hitting a quota limit is usually the worst.
The best advice we have is to consider setting your default storage preference to the scratch storage, then to set specific named histories to the permanent storage location. I’ll initially load my reference data or anything else really important into a dedicated history with a “permanent” storage location, then copy those datasets into other “temporary” storage histories where I might have large amounts of raw data loaded for processing. If you are using workflows, this can get quite sophisticated by setting certain tool steps or classes of data (example: intermediate outputs) to specific storage locations to override those defaults.
This sets the default to temporary, allowing work to keep processing, then I can flip to permanently save anything that is expiring, which is usually much less data, and data that I care about more (summary results). This is usually the reverse of what it seems most people are doing but that is intentional – we don’t want anyone to lose their data unexpectedly, so the default is always permanent. We have been thinking about adding in a note during account confirmation about recommended defaults and data storage strategies when setting up the account for larger projects. Maybe in a future release!
This was a good question! Let’s leave this open until fully resolved in the application. 