I intend to upload a large (1000+) number of files to using galaxy using FTP, but being new to this method of uploading files I decided to try with just 20.
Following the tutorials available online, I was able to successfully upload my 20 files to the FTP server (ftp.usegalaxy.eu), and I can see them in the upload screen; the problem arises after I select “Choose remote file > FTP Directory > {my files} > OK > Start”. The files appear greyed out in the history, get processed shortly thereafter, and then systematically return an error:
" Failed to fetch url gxftp://name_of_file. [Errno2] No such file or directory: ‘/data/jwd01/incoming/my_email@address/name_of_file’ "
I can reproduce the FTP problem at the UseGalaxy.eu server. With both an older file and three new ones. It seems the problem is with that final loading step.
This is my shared history, and one of the job identifiers (for the EU admins to use).
I’ve cross-reported this over to their chat, but I would expect them to post the update back here. To find this chat you can use this link, or see the bottom of the EU server homepage (click on the Galaxy icon in the upper left then scroll down).
Upload this way seems to be working fine, as @gbbio suggests using. You can drag and drop 10 files at a time there too. Interrupted connections can be resumed as needed. I wouldn’t drop 1000 unless on a really fast connection but you might be able to do 100 at a time.
URL fetching
This is definitely the best way to batch load data. Put your files in a publicly accessible location. The URLs cannot be password protected – but you also don’t need to leave them up there for long, just long enough to fetch the data.
If you use the Rule Builder you can organized your collections at the very start this way. It is worth learning and we have some tutorials to show how it works.
I’m not in a rush to upload all my files, and since the chat link @jennaj sent seems to suggest the issue is temporary, I think I’ll just wait before the big upload.
I have used the local upload in the meantime to test some workflows (again, with less files) and it has been working well, which suits me just fine for now (although I’m having trouble with scheduled jobs taking forever but that’s another story, and not something I would consider an issue just yet…).
