On an Macbook Pro M1 Max Arm64 Mac. Even after editing job_conf.xml, I can only use one core

[antomicblitz]

Hi. First off, I am a complete noob, but I have managed to get Galaxy running on a M1 Mac. Unfortunately, I am not able to take advantage of the multiple cores. I know I need to edit the job_conf.xml file so that it will override the GALAXY_SLOTS default value for most of the tools, which is usually 1. I have an 10 CPU core mac with 32gb of memory and I want to try to optimise galaxy to use all of it when necessary.

I have followed several other post and have tried to cobble together my own job_conf.xml file to have the following:

<?xml version="1.0"?>
<job_conf>
<plugins>
  <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="10"/>
<plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="10"/>
</plugins>
<handlers>
<handler id="main"/>
</handlers>

<destinations default="local">
    <destination id="local" runner="local">
    <destination id="local10threads" runner="multilocal">
        <param id="request_cpus">10</param>
        <param id="memory">28G</param>
        <param id="embed_metadata_in_job">False</param>
        <param id="nativeSpecification">-p Galaxy -n 10</param>
        <env file="/data/galaxy/galaxy-central/galaxy_venv/bin/activate" />
    </destination>
</destinations>
<tools>
    <tool id="bowtie2" destination="local10threads"/>
    <tool id="bwa-mem" destination="local10threads"/>
    <tool id="bwa-mem2" destination="local10threads"/>
    <tool id="bowtie2" destination="local10threads"/>
    <tool id="bowtie" destination="local10threads"/>
    <tool id="bowtie-build" destination="local10threads"/>
    <tool id="fastq_groomer_parallel" destination="local10threads"/>
    <tool id="deeptools" destination="local10threads"/>
    <tool id="cufflinks" destination="local10threads"/>
    <tool id="cuffdiff" destination="local10threads"/>
    <tool id="cuffmerge" destination="local10threads"/>
    <tool id="tophat2" destination="local10threads"/>
    <tool id="trimmomatic" destination="local10threads"/>
</tools>
</job_conf>

However, when I run bwa-mem2 or any other tool in galaxy, it still reverts to only using one core or whatever the default value is .

Can someone please help me figure out what to do? I am out of ideas

As an example, this is what Galaxy shows is the resulting argument when I try to run bwa-mem2:

set -o | grep -q pipefail && set -o pipefail; ln -s '/Users/antoniolamb/galaxy/database/objects/e/1/c/dataset_e1c7b303-95f1-4185-9b51-a3e66cacde75.dat' 'localref.fa' && bwa-mem2 index 'localref.fa' && bwa-mem2 mem -t "${GALAXY_SLOTS:-1}" -v 1 'localref.fa' '/Users/antoniolamb/galaxy/database/objects/5/d/1/dataset_5d1ed8fa-99aa-466c-b508-8fff2dfcc65b.dat' '/Users/antoniolamb/galaxy/database/objects/1/9/1/dataset_19188ad0-62b7-492c-9d7b-6263ff735bd0.dat' | samtools sort -@${GALAXY_SLOTS:-2} -T "${TMPDIR:-.}" -O bam -o '/Users/antoniolamb/galaxy/database/objects/0/b/4/dataset_0b48873c-3cf7-4f80-add1-2a086f5237cb.dat'

This is what shows in the destination parameters. I am so confused and frustrated why it won’t use all the cores available?

[antomicblitz]

Hello! After almost a week of troubleshooting, I finally got it working and I now have a local installation of galaxy that uses 9/10 cores and 32 GB of memory on my macbook pro . I left one for the OS so it doesn’t crash. If anyone else has this problem, please use these job_conf.xml settings in your file, and change the number of cores
( 9). It will make all tools default to using the number of cores specified.

<?xml version="1.0"?>
<job_conf>
    <plugins>
        <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner"/>
    </plugins>
    <handlers>
        <handler id="main"/>
    </handlers>
    <destinations default="multicore_destination">
        <destination id="multicore_destination" runner="local">
            <param id="local_slots">9</param>
        </destination>
        <tools>
        <tool id="*" destination="local"/>
    	</tools>
    </destinations>
</job_conf>

Below is proof :). Trimmomatic using 46 threads running on multiple cores. I confirmed it works with BWA-mem as well.