Web hook/POST to external URL when job begins and completes running

Hello,

I was looking into the web hooks available in Galaxy and from what I can tell, in terms of job scheduling, the only hook available is once the job is submitted by the user. There doesn’t seem to be a hook which can be triggered when a job begins running on a cluster (using Slurm for scheduling on our HPC), or when it finishes (either success or fail).

Would it be possible for me to add something like this to my Galaxy install, or perhaps there’s an easier way to do so. Ideally this would be for all tools, so it’s not quite feasible to edit all the tool XMLs and add custom execution code.

Essentially what I would want is to POST to a URL (external REST API) with a certain message (BEGIN or END FAIL or END SUCCESS, and the submitting user’s username) when jobs begin or finish.

Any suggestions are appreciated!

2 Likes

There are a few options here:
I recommend you poll the API for /api/jobs https://docs.galaxyproject.org/en/latest/api/api.html#galaxy.webapps.galaxy.api.jobs.JobController.index

The other possibility is to hijack the dynamic runner functionality and write a function that makes a request before assigning to another destination: https://github.com/galaxyproject/galaxy/blob/34ee279142a3b3ec60c1d579dc4e2250a229fcd9/lib/galaxy/config/sample/job_conf.xml.sample_advanced#L667-L676

You could also modify the runner directly to make a request: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/drmaa.py

Everything but polling the API will break with future releases.

Thanks for the reply. I think option 2 or 3 would be favourable.

Correct me if I’m wrong, but option 2 would only do my function upon job submission to DRM, since the dynamic runner is just in charge of scheduling/preparing the job and not following it through its life like the DRMAA runner directly.

Is there any reason why these would break with future updates? Is Galaxy changing the way jobs are run?

If what I said above is true, I think I’ll have to edit the DRMAA.py runner file directly to get the functionality I want.

The issue with modifying Galaxy is that your patches will need to be updated with every release that breaks them.

Is there a reason you can’t poll the API?

I’m in the process of developing an iOS app to monitor Galaxy job status across servers, among other things which can all be done via API directly. The goal is to publish it so other users can use it as well. I’m implementing push notifications for job begin/completion, hence my question about Web Hooks for those statuses.

Regarding polling the API, this isn’t something iOS supports when an app is in background state. I could setup a “main” external server to poll all Galaxy servers which users link in the app, however this could quickly degrade performance when scaling to many users and/or many servers.

At this point, I think I will go the route of making a custom (modified version of drmaa) job runner. This would be the one step users need to do manually to get this implemented with their servers, and only if they want push notifications.

I’ll just have to remind users every new release to replace the job runner with the custom one, ensuring I stay up to date with any changes in the runner files as well

I’m not sure if this is something the Galaxy Project has considered, or would like to collaborate on…

You should definitely look at an alternative to modifying Galaxy.
Most users don’t own their copy of Galaxy and you will struggle to convince sys-admins to maintain custom deployments of Galaxy.

I also can’t think of a reason someone would need immediate notification of a job completing. If the job takes longer than the person is willing to wait, they usually go off and do other things and come back when they are ready.

Galaxy does offer the option to send an email upon job completion. You could look into making a contribution to Galaxies code base, extending this existing functionality.

2 Likes

Appreciate your suggestions, thanks. I’ll look into my options some more, cheers!

I just ran across this: https://docs.galaxyproject.org/en/release_20.01/admin/config.html#webhooks-dir

Looks like they provide a way of implementing webhooks.

Thanks for the link, only thing is that the webhooks which exist already out-of-the-box are quite limited in terms of when they’re called, only when you schedule a tool or workflow, for example, and not when it is running/finished.

https://docs.galaxyproject.org/en/release_20.01/admin/special_topics/webhooks.html