Suppose I have a nested directory structure, something like data > si_n > (si_n.cell, si_n.param) with multiple values of n. Is there a way to upload this folder into a Galaxy collection with the same nested structure?
I think if this is available for your respective instance, FTP upload can do this for you.
But FTP upload has been decommissioned for usegalaxy.org, so this won’t work there. Does anybody have any suggestions for how to do this without FTP available? Is there some kind of --recursive functionality for the galaxy-upload tool?
1 Like
I’m not exactly sure and can’t test right now due to the UseGalaxy.org server being down for maintenance today.
Once back up – I would test to see if the upload string will accept standard unix commands, something like one of these.
Per n
…/data/*n*
Per file extension
…/data/*.cell
…/data/*.param
This would probably fail due to the . and .. and any other dot files getting picked up.
…/data/*
Let us know what you think. That tool can be updated if there is a use case we missed on the first pass. We liked FTP too but had to change out for alternative methods due to updated security requirements on our academic services.
What’s working currently is:
find <top-level-folder> -type f -exec galaxy-upload --url https://usegalaxy.org --api-key $API_KEY --history-id <id_of_history_to_upload_to> --file-name "{}" "{}" \;
This will upload all files in <top-level-folder> as individual datasets to the history with ID <id_of_history_to_upload_to>.
Once the data is there you can select all of it and choose Build Collection from Rules. Since the above upload command preserves paths into subfolders, you can use them with the Rule builder to structure your collection like you want.
I leave it up to @jennaj to explain individual steps of this in detail. For the rule-based uploader I also recommend Rule Based Uploader and Rule Based Uploader: Advanced.
@nate maybe galaxy-upload could make this more convenient than the current hack?
1 Like