multithreading and parallel processing files

1

I’ve initialized a Galaxy instance on aws and am an admin on there.
I am wondering how to parallelize the use of some tools on there. For example tophat2 has an option -p where on the command line it allows for parallel processing but that option is not one of the options that can be configured in the User Interface tool. Is there a way around this?

There is a notification on the homepage of Cloudman UI that says this version of Galaxy has configured some tools to run on 4 cores in the file “job_conf.xml.cloud”, but is it possible to edit this file to use more processors?

Thanks

1 Like
2

Yes, you can edit the file. You will need to create a new destination with the number of cores you want set. Something like:

        <destination id="slurm_cluster_cpu16" runner="slurm">
            <param id="nativeSpecification">--nodes=1 --ntasks=16</param>
        </destination>

The default can be seen here:

@enis, if this file is edited, how should it be persisted?

2 Likes
3

To persist job_conf.xml, copy the original template from /mnt/cm/cm/conftemplates/job_conf.xml to /opt/cloudman/config/conftemplates/job_conf.xml and modify as desired before restarting Galaxy via CloudMan. (Note that you’ll have to create the /opt/cloudman/config/conftemplates/ directory.)

1 Like
4

Thank you, I’m trying this now. hopefully I’ll be able to get it to work!

1 Like
6

which of the files do I modify after copying?

1 Like
7

Should be just job_conf.xml to add a specific destination and tools you want to run with more cores.
These slides may help with some examples of how the given file is composed: https://galaxyproject.github.io/dagobah-training/2018-cape-town/16-compute-cluster/compute-cluster.html#7

1 Like
8

Sorry I should have been more clear, I meant do I edit the /mnt/cm/cm/conftemplates/job_conf.xml or /opt/cloudman/config/conftemplates/job_conf.xml that I copied from the first destination

1 Like
9

You should copy /mnt/cm/cm/conftemplates/job_conf.xml to /opt/cloudman/config/conftemplates/job_conf.xml.

1 Like
10

Thank you @enis and @jxtx, it worked.

3 Likes
11

Do anyone have a list of tools that work in multi-core more so I can populate my job_conf file?