Datasets transfer from usegalaxy.org to usegalaxy.eu

[marcinschmidt]

I wanted to move/copy my dataset collection (fastqsanger.gz files, >2GB each) from usegalaxyorg to usegalaxyeu as on the latter server there is a tool I am going to use. I tried to do as it was written in Question 2, Help 2. But as I copy-paste the URL to the dataset (from the ‘Copy link’ - chains icon) to the upload data tool it shows very small size of the data (few bytes) and the transfer ends with an error ('The uploaded file contains invalid HTML content").

The history is accessible (Share and Publish).

I have a collection (a list of pairs with 21 items) of total 42 fastqsanger.gz files.

Can you help me what am I doing wrong?
According to old posts it should work.

Best regards,
Marcin

[marten]

I am not sure this actually works with datasets in a collection (I just tested it and got the same error). However there should be a possible workaround – every dataset in a collection typically has a corresponding “hidden” dataset in the same history. Can you try using those?

[marcinschmidt]

Hi! I have no idea where to look for the hidden dataset. My collection looks
like a single object (no icons allowing to do anything). It could be expanded
into unnumbered pairs of datasets with names composed from matching parts
of forward and reverse files with extension .fq.
then I can expand a single dataset:

Neither link taken from chains icon nor diskette icon allows to successfully download a file to usegalaxyeu

[marten]

Hidden datasets would be at the history level. In the top part of the History there would be “show hidden” link. Try clicking that.

[marcinschmidt]

Hi! Neither the link that is copied to clipboard by clicking on ‘chain’ icon nor the one that initiates download (by clicking on ‘diskette’ icon) work.

[marten]

What does the link that you get by clicking on the chain look like?

[jennaj]

Hi @marcinschmidt ,

The download function appears to be fine as of now. Instructions for dataset collections are in this FAQ: Downloading Data - Galaxy Community Hub

Be aware that at some Galaxy servers the history must be first set to a shared state (by URL) before a URL will retrieve data due to a larger server-wide default preference. Example: “from” usegalaxy.eu “to” usegalaxy.org would require that a history is in a shared state, but not the reverse. All preferences can be adjusted by you for default permissions.

Thanks for following up!

[marcinschmidt]

Hi! There is a post from ‘jennaj’ so it must be broken but the link looks like this:
https:/
/usegalaxy.org/datasets/bbd44e69cb8906b5fa7d3cf2966452cb/display?to_ext=fastqsanger.gz
(I split the address in the beginning)

[marten]

I think you have those datasets private to your account, so another Galaxy cannot fetch it. If I try to open your link I cannot access it either. The other Galaxy cannot log in as you, the datasets need to have permissions open in order for this transfer to work. Check out the advice above from Jenn regarding the “shared state” of history.

[marcinschmidt]

I have the ‘Make History accessible’ on in the ’ Share or Publish History’

[marten]

What do the permissions look like when you click on the pencil icon on dataset?

[marcinschmidt]

Looks the same

[marcinschmidt]

Hi @marten ! I tried to make the dataset accessible for you but entering your email I had ‘No matches found’ ?!

[marten]

The fact you have your account in the “access” field makes it inaccessible for everyone else. Remove it with the X and save. Then the link for the import to the other Galaxy will work.

[marcinschmidt]

Hi @marten ! Finally it went through, thanks for your help!

[marten]

Glad it works! Sorry it has to be this complicated at the moment. This is certainly an area to improve and part of the current project’s focus.

[marcinschmidt]

Hi again @marten and @jennaj , one more issue. The transfer went through but:
the dataset has strange name (“display?to_ext…”), not the one in the source

on the tile (as you can see the dataset has 2.7 GB) but in Dataset information the file size is only 421.3 MB ?! It was downloaded via link from “unhidden” object in history. When the link was taken digging deep into collection down to the source file (the very same file) it gets totally different name (“display?to_ext…=reverse”):

Is it normal (1) the name from a part of URL and (2) difference in reported dataset size (which one is what really is on the server)?

[marten]

The name is what Galaxy tries to guess from url when you do not give it a name in the upload dialogue. As of the consistency I’d recommend running a checksum (Secure Hash / Message Digest tool) on the two files and comparing that. The actual size on the disk may be an outdated info that will get refreshed at some point.

[marcinschmidt]

Hi @marten and @jennaj
It’s another bug. I entered to upload with name entered (‘AB1_.fq.gz’)

and it appeared in history as:

[jennaj]

Thanks @marcinschmidt and @marten

The dataset renaming in the Upload tool has been ticketed now: Filename is not used when uploading files by URL · Issue #12746 · galaxyproject/galaxy · GitHub

ps: The chain copy behavior corrected itself overnight as far as I can tell. Meaning, clicking on the chain icon does copy the path to the dataset to the clipboard.

Permissions need to be correct as @marten described. Default behavior at usegalaxy.org and usegalaxy.eu differ, but that can be changed under User > Preferences at either. Or, the history can be set to a shared state, and datasets contained in it can also optionally be set to a shared state.