Running an AI Workflow on a cluster

usegalaxy.org support
,
1

Hi,

I’m a newbie to Galaxy.

I’m trying to run the Deep Learning Tutorial workflow on a Slurm cluster.

  1. I’ve managed to run a simple one-step tool on the cluster (small CNN python program). The tool includes a form that asks for the Slurm job parameters (nb of nodes, nb of tasks, nb of GPU, etc.). I’ve used the Dynamic Destination Mapping (python method).

  2. I’ve managed to run the Deep Learning Tutorial which includes 6 tools (keras_model_config, keras_model_build, keras_train_and_evaluation, etc.) on my Galaxy server (which does not belong to the Slurm cluster). Each tool has its own form to request parameters from the user.

How can I combine the two steps above? i.e.

  • First, ask the user for the Slurm job parameters (nb of nodes, nb of tasks, nb of GPU, etc.) only once.

  • Second, run the workflow, each tool in succession on the nodes/GPUs allocated in the previous step.

Thank you for any hint / documentation link.