ftp transferred files not seen by galaxy and cannot sign into ftp://usegalaxy.org

#1

Hi, I’m a new user of galaxy and trying to upload my data files (>15GB each) for analysis. As prompted by the ftp tutorial (https://galaxyproject.org/ftp-upload/), I used filezilla to transfer files to usegalaxy.org . My filezilla window says that my 12 files are now transferred. When I go to get data->upload file from your computer->choose ftp file, the window states “your ftp directory does not contain any files”. And when I open a new tab with ftp://usegalaxy.org and type in my galaxy credentials, the sign in box keeps reloading with blank fields and “your connection is not private”.

Please assist with how to access the files transferred to usegalaxy.org via filezilla.

Thanks!

#2

I just tested it and it seems to work fine for my account. Could you please double check that you are using an account with exactly the same email address for both FTP and usegalaxy.org? That is how the ftp uploads and UI pair.

And when I open a new tab with ftp://usegalaxy.org and type in my galaxy credentials, the sign in box keeps reloading with blank fields and “your connection is not private”.

You don’t need to visit the ftp address from your browser. https://usegalaxy.org is the interface that you will select your files from.

1 Like
#3

Hi Marten,

Thanks for following up. I was using identical accounts. I had to re-establish a connection via filezilla by typing in ftp://usegalaxy.org, which reverted to usegalaxy.org when I hit enter. But with this new connection, smaller test files seemed to transfer and be visible by the get data command on galaxy. I have now restarted my 12 large files which took over 9 days to upload the first time, and hope that they will now be accessible by galaxy the way the small test files were.

I appreciate the clarification and I am glad that I don’t need to visit the ftp address from my browser, since that doesn’t seem to be possible. Perhaps when no files are on an account’s ftp, the popup suggestion in the get data->upload file->choose ftp file window should be updated to reflect the https://usegalaxy.org address rather than the ftp address, if that is the correct one?

#4

I think you might be clashing with autocleanup, from this page https://galaxyproject.org/ftp-upload/

Files not imported within 3 days will be cleaned up from the FTP site.

Can you upload from a faster connection?

#5

Hi @kcgfarbmuren Agree, the pop-up is confusing. We’ll look into this URL change (is a bit new) and why/how it was done. The model used to just state “usegalaxy.org”.

Screenshot with one file loaded last week at https://usegalaxy.org with Filezilla:

ftp-upload-popup-usegalaxy-org
ftp-upload-popup-usegalaxy-org.png1272×442 40.9 KB

Meanwhile, the URL is:

  • usegalaxy.org (Filezilla, or other FTP clients)
  • usegalaxy.org or ftps://usegalaxy.org (command-line)

As @marten states, FTP is often not needed anymore for files over 2 GB. It depends a bit on how much data you are loading, your connection speed, and how the data is organized within the Upload tool.

FAQ: https://galaxyproject.org/ftp-upload/

#6

Hi Jenna,

It appears my connection speed is slower than what you all expect. I have now successfully transferred 5 of my 12 files, but at least 2 of the remaining were partially transferred and removed from the ftp because of the time limit I presume? Now my ftp says it is empty when I try to add data to my console from it. Please advise as to how you would get the remaining 7 18-22GB files onto galaxy for analysis.

Many thanks,

Kate

#7

Once a file is loaded by FTP, you have 3 days to move it into a history.

Try to get files loaded completely within that time frame. Use FTP “resume” functions – within your Filezilla or other client or line-command. Try one at a time if you really need to.

12 files that are ~20 GB in size is going to put you nearly over quota or actually over. And if these are compressed fastq data, that is right on the edge of what tools on public servers can process, including usegalaxy.org. The amount of storage in your account (quota) is distinct from the compute/memory resourced needed to execute jobs, so bumping your quota up would be unlikely to help.

It is beginning to sound like you’ll need to set up your own Galaxy server to process such large data. Cloud Galaxy servers can be a good choice but there are others.

Thanks!