Direct transfer of ~109 GB Novogene RNA-seq data to UseGalaxy.eu and recommended storage

Uncategorized
1

Dear Galaxy community,

Following a suggestion from the Galaxy Europe support team, I am posting our question here so that the answer may also be useful to other users.

We have a paired-end mRNA-seq dataset consisting of 24 samples and approximately 109 GB of compressed raw data. The data are currently hosted on the Novogene data delivery system.

Novogene provides us with a temporary signed HTTPS download URL for a large .tar archive containing the sequencing data. The URL is valid for 7 days and can be regenerated if necessary.

We would like to avoid downloading the entire dataset to a local computer and then uploading it again, because our local internet connection and electricity supply can occasionally be unstable.

We therefore have two questions:

1. Direct server-to-server transfer

What is the recommended way to transfer the dataset directly from the Novogene HTTPS download URL to UseGalaxy.eu?

Should we simply use:

Upload Data → Paste/Fetch Data

and paste the signed HTTPS URL there, or would it be better to configure one of the repositories described here?

https://training.galaxyproject.org/training-material/faqs/galaxy/manage_your_repositories.html

Since Novogene only provides a temporary signed HTTPS URL, rather than an S3 access key, FTP account, WebDAV account, etc., we are not sure which repository option would be appropriate.

2. Storage strategy

The raw dataset is approximately 109 GB, and our downstream workflow will include QC, trimming, STAR alignment, featureCounts and differential expression analysis.

We understand that UseGalaxy.eu provides standard storage as well as a larger short-term storage area.

For a workflow of this size, would you recommend keeping the original raw archive/data in standard storage and placing large temporary/intermediate files such as extracted FASTQ files and STAR outputs in short-term storage?

Or would you recommend a different storage strategy?

Thank you very much for your advice.

2

For a one-off temporary signed URL, Paste/Fetch Data in the Upload tool is the recommended direct route: the fetch happens server-side, so you never need to download the 109 GB yourself, and you can extract the per-sample files from the .tar once it is in your history. Just start the upload promptly since the signed URL expires after 7 days. The external repository options in that FAQ are really meant for persistent data sources you access repeatedly, not a single temporary download, so you can skip that setup here. For storage, your plan is the right idea: keep the raw reads in your standard 250 GB permanent storage and use the scratch space (up to 2 TB per user, purged after 60 days) for intermediate files, since QC/trimming/STAR outputs are regenerable but bulky. Happy to be corrected if I’ve got that wrong.

4

Dear Galaxy Europe Support Team,

Thank you for the previous replies. I think I may not have expressed the remaining issue clearly enough, so I would like to restate it as precisely as possible.

I now understand the following:

  • the 109 GB Novogene dataset can be transferred using Paste/Fetch;

  • the raw data can remain in the 250 GB long-term storage;

  • large intermediate files can be placed in the 2 TB short-term/scratch storage;

  • datasets in the scratch space are retained for approximately 60 days.

The point I am still trying to clarify is what options are available when an active project requires the 2 TB working space for longer than 60 days.

This is an ongoing RNA-seq project involving 24 paired-end samples, including FASTQ processing, QC, STAR alignment, count generation, differential expression analysis, and several downstream analyses. Depending on the progress of the project, some large intermediate datasets may still be required after the 60-day retention period.

For that reason, I would be very grateful for clarification on the following points:

  1. Can the 60-day retention period of the 2 TB scratch storage be extended for an active research project?

  2. If not, what is the recommended solution for researchers who need several hundred GB to approximately 2 TB of active working storage for longer than 60 days?

  3. Does Galaxy Europe provide any paid or co-funded storage option for longer-term retention of large active datasets?

  4. If such an option exists, could you please provide information on the available storage capacity, retention period, approximate cost, and application procedure?

  5. Is it possible to connect external persistent storage so that Galaxy tools, including STAR, can use it for large outputs?

  6. If so, which storage approach would you recommend for this type of RNA-seq workflow?

I would also like to emphasize that I am not requesting unlimited free storage. If additional persistent storage is available through a paid or co-funded arrangement, I would be willing to consider that option.

The data-transfer issue itself has already been resolved, so my question at this stage is specifically about how to retain a large active working dataset beyond the standard 60-day scratch-storage period.

If this question falls under a different Galaxy Europe team or storage administrator, I would also be grateful if you could point me to the appropriate contact or documentation.

Thank you again for your time and assistance.

5

For what it’s worth, I wouldn’t plan on the scratch retention being extendable — it’s designed as transient working space, so treat the 60 days as a hard deadline. In practice I’d keep the raw reads and the final count tables in the 250 GB permanent storage and accept that the bulky STAR intermediates can be regenerated from the FASTQs if needed. On paid storage and external mounts, that’s really a question for the Galaxy Europe team — I don’t want to guess at their policies.

6

Hi @aaak

Coming up with a data management strategy is really important so it is good to be thinking about this at the start of a project where you know in advance that the starting data is large.

For these questions, have you seen the options outlined here for attached storage? Some are commercial options.

For RNA-seq experiments, you would probably not need more space since the data loading is at the start: fastq reads and intermediate BAMs. These are large but temporary, and using the 60 day space should be plenty of time to complete a workflow to produce the reduced count files. Running these in batches should be possible if really needed. Then, accumulating the much smaller count files in the permanent storage would allow ongoing explorations and comparisons that may take more time.

The :gear: IWC Workflow Library has examples for these types of batch workflows. As an example, this workflow will reduce the data down to a manageable size for even large experiments. → RNA-Seq Analysis: Paired-End Read Processing and Quantification | IWC

Am I misunderstanding? Why would the reads and BAMs need to be retained for the longer time period?

7

I got my answer via email and thank you for all responses. I was told that Galaxy EU have no policy for paid addional storage