Per-job conda environment creation errors

, , ,
#1

We’re in the middle of setting up a new Galaxy server (18.09) and currently installing tools - mixed ToolShed and locally developed code.

Quite a lot of our tools are currently failing with error:
Conda dependency seemingly installed but failed to build job environment.

This includes BCFTools, GATK (3.x), and others. Looking through the manage dependencies panel, the ones with issues often have multiple dependency requirements, with Conda as the resolver and separate environment paths listed. Other, working tools with multiple requirements via conda give the same mulled-v1-HASH environment path for all of them. Now, we don’t have the mulled beta enabled right now, and I’m not sure we ever did, so I don’t know how these were sourced. We also aren’t caching the dependencies, although maybe we need to.

Has anyone seen this behavior, or have any troubleshooting suggestions? Is there a latency factor here - could it fail due to time taken to build the env across filesystems?

1 Like
#2

Hi @dkolbe

If a tool requires multiple dependencies the only we to make this work using Conda is to install all dependencies into a new environment. mulled-v1-<HASH> is a hash of the combination of package names and versions. This is not configurable, and this is what you want to prefer and see for all mutli-dependency tools.
Other options that concern mulled are about building/fetching containers using a similar scheme, so likely don’t concern your use case.

ToolShed installed packages have been deprecated a long time ago because they are difficult to install and maintain. All recent tools from should be installable via Conda.
Mixing of ToolShed packages and Conda packages is possible in principle, but likely won’t work well.
If you encounter cases where this doesn’t work you can try uninstalling the ToolShed pckages. To do this go to Manage dependencies in the admin panel, select the tool you are interested in and hit Install dependencies via Conda.

If you encounter tools that just won’t install via Conda I would suggest contacting the authors and ask them for an update. Unfortunately we won’t update GATK 3 due to the licensing issue. GATK4 had been in works this summer, I hope this will materialize soon.

2 Likes
#3

Thanks @mvdbeek. Does that mean I should enable the mulled beta so that the combined environments get built? Or do I need to build them by hand? If they should be created automatically, for some reason that’s not happening.

P.S. - when I said ToolShed, I just meant obtaining them via toolshed, they’re all actually fulfilled by conda, it looks like.

1 Like
#4

No, it’s all fine. Make sure all your tools that have multiple requirements are using a Conda environment that starts with mulled-v1-. This should happen automatically, but can be very slow. The installation progress is logged in your log file. If for some reason this doesn’t happen you can trigger the installation in the Manage dependencies menu. Have an eye on the logs when you do this, you’ll see the commands that will be executed.

1 Like
#5

Just following up with some details here in case anyone hits a similar issue.

Possible root of problem: bcftools and htslib rely on openssl. Openssl has been recently updated to 1.1, and these libraries say they’re compatible, but they’re not because they explicitly try to load openssl1.0. These dependencies I think flagged red on install through admin UI. I manually (CLI) forced their individual environments back to openssl1.0.2p.

This is the point where I had gotten stuck. The manage dependencies showed the requirements as met, but gave separate environments. Selecting the tool and then triggering install dependencies did create the new shared environment (and did not try to reinstall openssl1.1).

1 Like