Downloading a collection from usegalaxy.org fails to complete

usegalaxy.org support
, ,
#1

Hi,
I’m trying to download a collection of wiggle files from the usegalaxy.org server, using instructions from this page: https://galaxyproject.org/support/download-data/. I have tried using Chrome and using curl command in terminal. Downloads start fine, however, both methods fail after ±1GB has been downloaded. Terminal output below (links redacted). Would appreciate any suggestions to make this work.

mlp-varo-010:~ k.boogerd$ curl -o collection.tar.gz https://usegalaxy.org/api/dataset_collections/REDACTED/download?key=REDACTED
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1029M    0 1029M    0     0  2028k      0 --:--:--  0:08:39 --:--:-- 3283k
curl: (18) transfer closed with outstanding read data remaining

mlp-varo-010:~ k.boogerd$ curl -C - -o collection.tar.gz https://usegalaxy.org/api/dataset_collections/REDACTED/download?key=REDACTED
** Resuming transfer from byte position 1079838960
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume.
1 Like
Larger files not downloaded completely? Try using a History Archive
#2

Hi - Could you give this another try? I cannot reproduce the problem with a 1.7G BAM collection.

The incomplete transfer may have been due to a transient server issue or downloading from a slower internet connection.

Small gotcha: The resume command can fail like this with older versions of curl when the file is already downloaded completely. That doesn’t seem to be your case if the original collection size is over 1029M – that said, updating curl anyway to eliminate any legacy issues with older versions would be a good idea.

If after updating curl, rerunning, and the complete collection fails to download completely, please write back and we can troubleshoot from there.

Thanks!

#3

Thanks for the suggestions. It appears that my local settings indeed are the issue. Although not resolved yet, I’ve been able to download the files using another PC. Thanks!

1 Like
#4

I believe this was due to the nginx buffer temp file max size being 1GB by default. We probably can’t support buffer temp files large enough for collections, but I have disabled buffering for collection downloads on usegalaxy.org entirely, which may fix this issue (hopefully without creating more issues). Please give it a try and let us know if it’s still not working.

1 Like
#5

Hi @nate … I believe we have the same issue on our server. Could you share the part of the nginx configuration that you changed?