webapp galaxy/tools data_fetch.py not inheriting LD_LIBRARY_PATH

usegalaxy.org support
1

I’ve installed galaxy and the python venv it creates. However, my venv is using a system python that is not in a standard location. Galaxy starts up fine because I’ve included the path to the venv’s python libpython in my environment’s LD_LIBRARY_PATH.

However, when I use the Upload Data webapp, it throws an error saying, “error while loading shared libraries: libpython3.9.so.1.0” when trying to execute data_fetch.py

This is certainly because the tool, or webapp, that runs data_fetch.py is not inheriting my env’s LD_LIBRARY_PATH. Or, perhaps not even activating the venv prior to running it’s linked python.

Ideas on how to fix this?
-Lopaka

2

Hi @pakalee

I’ve cross-posted your question at the Admin chat. They may reply in the chat or back here, and feel free to join the chat.

Matrix: You're invited to talk on Matrix
Gitter galaxyproject/admins - Gitter

1 Like
3

If there’s any possibility of using a different Python for this purpose, I would strongly recommend it. A Python that doesn’t have its runtime linker path set properly is fairly broken. I use a version of Python installed from Conda and this works well. Galaxy even installs Conda for its own dependency usage on first startup, so you can use that Conda instance if you like:

$ /path/to/galaxy/database/dependencies/_conda/bin/conda create -n __python@3.9 python=3.9
$ /path/to/galaxy/database/dependencies/_conda/envs/__python@3.9/bin/python3 -m venv /path/to/galaxy/venv

If using a fixed Python isn’t an option, there are a couple things you could do:

  1. Set LD_LIBRARY_PATH in Galaxy’s job configuration as described in the sample.
  2. Replace your venv’s python symlink with a shell script that set LD_LIBRARY_PATH and calls the original Python:
#!/bin/sh
export LD_LIBRARY_PATH='/wherever/lib'
exec /wherever/bin/python3 "$@"

While these may work to solve the immediate problem, I suspect you will continue to run in to related issues down the line.

1 Like
4

Hi Nate. Yeah, I had the same thoughts and you and tried exec’ing python from a script. But that didn’t work for all processes. The solution was to use a statically-compiled python. This is what conda does and why it doesn’t have a problem with python resolving it’s libs.

So there definitely is a problem somehow with galaxy processes not inheriting the env that started run.sh. I can see in the logs that run.sh activates galaxy/.venv that it created on install. And, everything runs fine, until another app, like data_fetch.py is run.