FTP Upload in Galaxy (with FileZilla) - Maximum FTP data file size is 50 GB each

I am trying to upload a database of all bacterial nucleotide sequences (gigantic file - 107 GB). I am using FTP via FileZilla, and it got to 80.5 Gb before giving the error “552 Transfer aborted. File too large”. I read something about a 250Gb limit but I think I should be within that - even with files already on Galaxy, it says I am only using 2.7Gb. Not sure what to do (I am definitely no bioinformatician and it took me two weeks to figure out how to download this file -_-). If there’s an alternative to use a database like this on Galaxy that is already uploaded I am all ears. Below is the full set of messages that keep iterating until I stop the queue.

Error: File transfer failed after transferring 294,912 bytes in 1 second
Status: Starting upload of /Volumes/MBLab/Bacterial Nucleotide Database - October 2020.fasta
Status: Retrieving directory listing of “/”…
Command: PASV
Response: 227 Entering Passive Mode (129,114,60,56,119,69).
Command: REST 0
Response: 350 Restarting at 0. Send STORE or RETRIEVE to initiate transfer
Command: MLSD
Response: 150 Opening BINARY mode data connection for MLSD
Response: 226 Transfer complete
Command: PASV
Response: 227 Entering Passive Mode (129,114,60,56,120,177).
Command: REST 80530636800
Response: 350 Restarting at 80530636800. Send STORE or RETRIEVE to initiate transfer
Command: STOR Bacterial Nucleotide Database - October 2020.fasta
Response: 150 Opening BINARY mode data connection for Bacterial Nucleotide Database - October 2020.fasta
Response: 552 Transfer aborted. File too large
1 Like

Discussed via email with @aroebuck

Items that may help others:

  1. The maximum size of an FTP file is 50 GB at public Galaxy servers. BAMs are a bit different, the maximum size is 25-35 GB.
  1. Setting up a local Galaxy.
  1. Alternatives, including cloud options. Many can be easier to administrate – including the GVL of Cloudman.

More help and prior Q&A can be found at this site or at the GTN tutorial site. Start by with keywords like “upload” or “cloud” or “ftp”. If your issue remains, start a new topic and be as specific as possible for context: How using Galaxy, where using Galaxy (URL if public server), version of local/cloud Galaxy, special configurations…

Best,