Thanks for the extra info.
Yes, the Gencode GTF looks to be not the problem.
RNA-STAR is a compute-intensive tool. 16 GB is not sufficient for human mapping. The memory needed to run the tool line-command is the same as when running it within Galaxy. See: Mapping RNA-seq Reads with STAR - PMC
Quote from that publication:
Necessary Resources
Hardware
- A computer with Unix, Linux or Mac OS X operating systems.
- RAM requirements: at least 10 x GenomeSize bytes. For instance, human genome of ~3 GigaBases will require ~30 GigaBytes of RAM. 32GB is recommended for human genome alignments.
- Sufficient free disk space (>100 GigaBytes) for storing output files.
Alternative ways to run your own Galaxy server, including Cloud choices. Since you are already using Cloudman, you probably just need to bump up the VM choice to one with more resources: Galaxy Platform Directory: Servers, Clouds, and Deployable Resources - Galaxy Community Hub.
Please also see this recent Galaxy Blog post. This is an alternative for using a local Galaxy with on-demand cloud resources incorporated: The Galactic Blog - Galaxy Community Hub > Enabling cloud bursting for Galaxy
Thanks!