multithreading and parallel processing files

admin
cloudman
jobs

#1

I’ve initialized a Galaxy instance on aws and am an admin on there.
I am wondering how to parallelize the use of some tools on there. For example tophat2 has an option -p where on the command line it allows for parallel processing but that option is not one of the options that can be configured in the User Interface tool. Is there a way around this?

There is a notification on the homepage of Cloudman UI that says this version of Galaxy has configured some tools to run on 4 cores in the file “job_conf.xml.cloud”, but is it possible to edit this file to use more processors?

Thanks


#2

Yes, you can edit the file. You will need to create a new destination with the number of cores you want set. Something like:

        <destination id="slurm_cluster_cpu16" runner="slurm">
            <param id="nativeSpecification">--nodes=1 --ntasks=16</param>
        </destination>

The default can be seen here:

@enis, if this file is edited, how should it be persisted?


#3

To persist job_conf.xml, copy the original template from /mnt/cm/cm/conftemplates/job_conf.xml to /opt/cloudman/config/conftemplates/job_conf.xml and modify as desired before restarting Galaxy via CloudMan. (Note that you’ll have to create the /opt/cloudman/config/conftemplates/ directory.)


#4

Thank you, I’m trying this now. hopefully I’ll be able to get it to work!


#6

which of the files do I modify after copying?


#7

Should be just job_conf.xml to add a specific destination and tools you want to run with more cores.
These slides may help with some examples of how the given file is composed: https://galaxyproject.github.io/dagobah-training/2018-cape-town/16-compute-cluster/compute-cluster.html#7


#8

Sorry I should have been more clear, I meant do I edit the /mnt/cm/cm/conftemplates/job_conf.xml or /opt/cloudman/config/conftemplates/job_conf.xml that I copied from the first destination


#9

You should copy /mnt/cm/cm/conftemplates/job_conf.xml to /opt/cloudman/config/conftemplates/job_conf.xml.


#10

Thank you @enis and @jxtx, it worked.