Downloading a collection from usegalaxy.org fails to complete

download
dataset-collection
curl
#1

Hi,
I’m trying to download a collection of wiggle files from the usegalaxy.org server, using instructions from this page: https://galaxyproject.org/support/download-data/. I have tried using Chrome and using curl command in terminal. Downloads start fine, however, both methods fail after ±1GB has been downloaded. Terminal output below (links redacted). Would appreciate any suggestions to make this work.

mlp-varo-010:~ k.boogerd$ curl -o collection.tar.gz https://usegalaxy.org/api/dataset_collections/REDACTED/download?key=REDACTED
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1029M    0 1029M    0     0  2028k      0 --:--:--  0:08:39 --:--:-- 3283k
curl: (18) transfer closed with outstanding read data remaining
mlp-varo-010:~ k.boogerd$ curl -C - -o collection.tar.gz https://usegalaxy.org/api/dataset_collections/REDACTED/download?key=REDACTED
** Resuming transfer from byte position 1079838960
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume.
1 Like

#2

Hi - Could you give this another try? I cannot reproduce the problem with a 1.7G BAM collection.

The incomplete transfer may have been due to a transient server issue or downloading from a slower internet connection.

Small gotcha: The resume command can fail like this with older versions of curl when the file is already downloaded completely. That doesn’t seem to be your case if the original collection size is over 1029M – that said, updating curl anyway to eliminate any legacy issues with older versions would be a good idea.

If after updating curl, rerunning, and the complete collection fails to download completely, please write back and we can troubleshoot from there.

Thanks!

0 Likes

#3

Thanks for the suggestions. It appears that my local settings indeed are the issue. Although not resolved yet, I’ve been able to download the files using another PC. Thanks!

1 Like