Title: Request for higher-memory resource for RSEM v1.3.1 due to OOM
Hello Galaxy Europe Support Team,
I am reprocessing the public PRJEB40416 gastric cancer RNA-seq dataset on usegalaxy.eu.
STAR alignment has completed successfully using GRCh38 and Ensembl release 98. I now need to run RSEM v1.3.1 on the STAR transcriptome-aligned BAM files.
The two transcriptome-aligned BAM files I am currently testing are approximately 7.8 GB and 9.9 GB.
I attempted to run RSEM inside Galaxy JupyterLab, but the current environment is limited to 4 GiB RAM. The RSEM process was terminated by the OOM killer even after reducing the run to one thread and disabling BAM output.
As you have noticed, Galaxy Interactive environments (GIEs, including JupyterLab) hosted at public Galaxy servers have computational limits that can lead to jobs that are too large to process.
Instead, try using the same tool directly. These can be incorporated into Galaxy Workflows for HTP processing.
As an example, for you tool are using, these are the public resources allocated at UseGalaxy servers:
RSEM calculate expression RNA-Seq by Expectation-Maximization
To get you started, these are the HTP workflows we have for Transcriptomics (so far!).
This workflow uses FeatureCounts, but you may adjust any part of it in your own copy of the workflow. The GTN has more example templates. All can be searched in the Galaxy app under Workflows → Public workflows.
There are a couple of reasons we limit Interactive Tools (ITs) to 4 GB.
Our job scheduler (HTCondor) can inspect a running job and scale its resource allocation dynamically, according to the rules shown above. This works for standard jobs, but not for ITs or tools that run in Docker. The reason is that HTCondor has a “docker universe” for natively managing Docker containers, but the Galaxy job runner doesn’t submit jobs through it. Docker containers instead get launched as a subprocess of the job script. Docker itself uses dockerd daemon which manages its own cgroup hierarchy and is controlled through the Docker API over a Unix socket, rather than being a direct child process of the job. That puts the container’s actual resource usage outside the cgroup HTCondor creates and monitors for the job, so HTCondor can’t see or dynamically scale it.
Singularity works differently: it has no daemon and simply execs the container process directly, so the container lands inside the same cgroup as the parent job script and inherits its limits. That’s what lets HTCondor monitor and scale Singularity-based jobs but not Docker-based ones.
