Hi,
I’m trying to download a collection of wiggle files from the usegalaxy.org server, using instructions from this page: https://galaxyproject.org/support/download-data/. I have tried using Chrome and using curl command in terminal. Downloads start fine, however, both methods fail after ±1GB has been downloaded. Terminal output below (links redacted). Would appreciate any suggestions to make this work.
mlp-varo-010:~ k.boogerd$ curl -o collection.tar.gz https://usegalaxy.org/api/dataset_collections/REDACTED/download?key=REDACTED
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1029M 0 1029M 0 0 2028k 0 --:--:-- 0:08:39 --:--:-- 3283k
curl: (18) transfer closed with outstanding read data remaining
mlp-varo-010:~ k.boogerd$ curl -C - -o collection.tar.gz https://usegalaxy.org/api/dataset_collections/REDACTED/download?key=REDACTED
** Resuming transfer from byte position 1079838960
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume.
Hi - Could you give this another try? I cannot reproduce the problem with a 1.7G BAM collection.
The incomplete transfer may have been due to a transient server issue or downloading from a slower internet connection.
Small gotcha: The resume command can fail like this with older versions of curl when the file is already downloaded completely. That doesn’t seem to be your case if the original collection size is over 1029M – that said, updating curl anyway to eliminate any legacy issues with older versions would be a good idea.
If after updating curl, rerunning, and the complete collection fails to download completely, please write back and we can troubleshoot from there.
Thanks for the suggestions. It appears that my local settings indeed are the issue. Although not resolved yet, I’ve been able to download the files using another PC. Thanks!
I believe this was due to the nginx buffer temp file max size being 1GB by default. We probably can’t support buffer temp files large enough for collections, but I have disabled buffering for collection downloads on usegalaxy.org entirely, which may fix this issue (hopefully without creating more issues). Please give it a try and let us know if it’s still not working.