Repeated DATA_FETCH failures affecting multiple FASTQ datasets

Uncategorized
1

Dear Galaxy Support Team,

I am experiencing repeated failures when uploading several datasets to Microbiology Galaxy (https://microbiology.usegalaxy.eu/).

The problem first occurred with 36_S36_R1_001.fastq.gz, which has now failed twice during the DATA_FETCH step. The error is reproducible and occurs before any downstream analysis is started.

Affected datasets include:

  • 36_S36_R1_001.fastq.gz
  • 39_S39_R2_001.fastq.gz
  • 40_S40_R1_001.fastq.gz
  • 40_S40_R2_001.fastq.gz

This suggests that the issue may not be limited to a single file.

For sample 36_S36_R1_001.fastq.gz:

  • Compressed size: approximately 2.3 GB.
  • I tested the file locally by extracting it with WinRAR/WinZip.
  • The extraction completed successfully without any warnings or errors.
  • The extracted FASTQ file is approximately 11 GB.
  • No corruption, CRC errors, or archive errors were reported during decompression.

Therefore, the file itself does not appear to be damaged.

I also received the Galaxy error report email. The traceback seems to indicate a server-side issue involving SSL/Celery/Kombu communication rather than a FASTQ format problem. Relevant messages include:

ssl.SSLEOFError:

[SSL: UNEXPECTED_EOF_WHILE_READING]

EOF occurred in violation of protocol

AttributeError:

‘ChannelPromise’ object has no attribute ‘_value_’

Failed Job ID: 4838ba20a6d8676576735ac6bebfa8d7

History: Males_microbiota_shotgun

Could you please investigate whether this issue is related to the Galaxy infrastructure, storage backend, or upload service? Since multiple independent datasets are affected and at least one of the files has been successfully validated locally, I suspect the problem may originate on the Galaxy side.

Thank you very much for your help.

2

Hi @rega

I have been able to load data by URL, download it, then load it again by local file browsing. I tested at both UseGalaxy.org and Microbiology.UseGalaxy.org (same account at both but the initial landing page is different). This account is using the default storage area and the files are very small so it may not reflect your exact use case.

If you could send in a bug report from the failed dataset(s), that will allow the EU administrators to review the exact error message closer. If you want to include a link to this topic in the comments, that is usually helpful as they will see it and provide updates back here!

11 GB/2.3 GB is not especially large but much depends on how the data is being loaded. If this is via URL, this tends to work best since the connection is between two servers. It has worked well for others to load up data once to a service like Dropbox, then to load to/from Galaxy that way (to avoid local file upload directly). Uncompressed data can get around exotic gzip compression types if really needed.

More options are included here → FAQ: How do I manage my Galaxy storage? and here FAQ: How do I manage my repositories on Galaxy?.

Thanks and let’s start there! :slight_smile: