S3 objectstore with a different download/upload option?

Hi,

I’ve been testing with S3 objectstore using Galaxy version 19.01. It is great stuff! Here is my plan, I have Galaxy installed on a small instance (head node), then use condor runner to assign jobs to different worker instances. I wish that workers can pull datasets from S3 and push datasets to S3 directly, not through the head node that Galaxy is installed on. But so far I can only figure out how to pull and push through the head node, that is all the data is cached in the object_store_cache directory. My question is, is it possible to pull from S3 and push to S3 during the job execution, so that the dataset is downloaded/uploaded through the worker instances? Or do you have any suggestions to make it work?Thank you

2 Likes