bwa has missing dependencies after using "Install New Tools (Beta)" link

1

Hello,

I have a fresh new installation of Galaxy (version 19.09 on Ubuntu) hooked into PostgreSQL. After users install a version of bwa in Galaxy (revision: 01ac0a5fedc3) from the “Install new tools (Beta)” link under the “Admin” section, the new bwa shows missing dependencies. Conversely, when I install an older version (revision: c71dd035971e) using the “Install New Tools” link (NOT Beta), the dependencies are installed along with bwa. I do not know how to resolve/debug these missing dependencies given my 3 weeks experience with Galaxy. Both bwa and samtools are installed in Ubuntu and included in the PATH variable for operating system user Galaxy. This seems like it should be an obvious thing so I’m seeking guidance on a good approach to debug this. Should I expect all tools under “Install New Tools (Beta)” to work? I wonder whether the PATH is not being properly passed to Galaxy so it can’t locate the operating system installations of bwa and samtools. In the Galaxy app under “Manage Dependencies” I tried selecting “Install checked dependencies using Conda” but it generates errors in the galaxy.log. Sorry, not sure how to include the galaxy.log without going over the 30K limit.

Background Info:
I verified that bwa and samtools are both in the operating system PATH for user galaxy (see below).
I used “apt-get” instead of conda to install bwa on Ubuntu: “(sudo apt-get update -y; sudo apt-get install -y bwa)”.
I’m launching galaxy as “sh /apps/galaxy/galaxy_app/run.sh start > /apps/galaxy/galaxy_app/galaxy_start.log 2>&1”.

BWA operating system verification

galaxy@PB-DEV-01:/apps/galaxy/galaxy_app/config$ which bwa
/usr/bin/bwa
galaxy@PB-DEV-01:/apps/galaxy/galaxy_app/config$ bwa
Program: bwa (alignment via Burrows-Wheeler transformation)
Version: 0.7.17-r1188
Contact: Heng Li lh3@sanger.ac.uk

SAMTOOLS operating system verification

galaxy@PB-DEV-01:/apps/galaxy/galaxy_app/config$ which samtools
/usr/bin/samtools
galaxy@PB-DEV-01:/apps/galaxy/galaxy_app/config$ samtools
Program: samtools (Tools for alignments in the SAM format)
Version: 1.6 (using htslib 1.6)

BWA tool information in Galaxy after installation

Tool shed: toolshed.g2.bx.psu.edu
Name: bwa
Description:
Revision: 01ac0a5fedc3
Owner: devteam
Location: /apps/galaxy/galaxy_app/database/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bwa/01ac0a5fedc3/bwa
Deleted: False

Dependencies:
bwa version:0.7.17 Resolver: none Exact Version: True Installation Status: not installed
samtools version:1.6 Resolver: none Exact Version: True Installation Status: not installed

2

Can you paste the relevant parts of the error log? If not, you can use a service such as gist.github.com to paste the log contents, or put the full log file somewhere that it can be downloaded and paste the link.

Galaxy manages its dependencies, by default using Conda, so the system versions are generally irrelevant to the Galaxy tool (although it will use things it finds on $PATH as a last resort).

3

Here is a partial cut of the galaxy.log. If you need more log info I’ll work to get a more complete log file to you. I deleted TONS of dependency statements at the end like these…:

Dependency biom-format not found.
galaxy.tools.deps DEBUG 2020-01-23 10:47:55,705 [p:111907,w:1,m:0] [uWSGIWorker1Core1]

— this is part of my galaxy.log file showing the conda errors from clicking on the
— “Install checked dependencies using Conda” link in the “Manage Dependencies” section.

— DEBUG is turned on: but I do not think the long list of DEBUG statements at the end
— is related since other tools seem to install fine.

...  most stuff deleted  ...

galaxy.tools.deps.conda_util DEBUG 2020-01-23 10:47:55,596 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Executing command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name mulled-v1-a1698655ddc683fd2767d127c7c583056e87322876f94e167ba3900da02c1fc4 bwa=0.7.17 samtools=1.6
galaxy.tools.deps.conda_util ERROR 2020-01-23 10:47:55,616 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Failed to execute command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name mulled-v1-a1698655ddc683fd2767d127c7c583056e87322876f94e167ba3900da02c1fc4 bwa=0.7.17 samtools=1.6
Traceback (most recent call last):
  File "lib/galaxy/tools/deps/conda_util.py", line 235, in exec_command
    return self.shell_exec(cmd, env=env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 58, in shell
    p = shell_process(cmds, env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 88, in shell_process
    p = subprocess.Popen(cmds, **popen_kwds)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
galaxy.tools.deps.resolvers.conda DEBUG 2020-01-23 10:47:55,616 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Removing failed conda install of [CondaTarget[bwa,version=0.7.17], CondaTarget[samtools,version=1.6]]
galaxy.tools.deps.conda_util DEBUG 2020-01-23 10:47:55,616 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Executing command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __bwa@0.7.17 bwa=0.7.17
galaxy.tools.deps.conda_util ERROR 2020-01-23 10:47:55,634 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Failed to execute command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __bwa@0.7.17 bwa=0.7.17
Traceback (most recent call last):
  File "lib/galaxy/tools/deps/conda_util.py", line 235, in exec_command
    return self.shell_exec(cmd, env=env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 58, in shell
    p = shell_process(cmds, env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 88, in shell_process
    p = subprocess.Popen(cmds, **popen_kwds)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
galaxy.tools.deps.resolvers.conda DEBUG 2020-01-23 10:47:55,635 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Removing failed conda install of bwa, version '0.7.17'
galaxy.tools.deps.conda_util DEBUG 2020-01-23 10:47:55,635 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Executing command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __samtools@1.6 samtools=1.6
galaxy.tools.deps.conda_util ERROR 2020-01-23 10:47:55,650 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Failed to execute command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __samtools@1.6 samtools=1.6
Traceback (most recent call last):
  File "lib/galaxy/tools/deps/conda_util.py", line 235, in exec_command
    return self.shell_exec(cmd, env=env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 58, in shell
    p = shell_process(cmds, env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 88, in shell_process
    p = subprocess.Popen(cmds, **popen_kwds)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
galaxy.tools.deps.resolvers.conda DEBUG 2020-01-23 10:47:55,651 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Removing failed conda install of samtools, version '1.6'
galaxy.tools.deps.conda_util DEBUG 2020-01-23 10:47:55,651 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Executing command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name mulled-v1-9057fa4fae7a852d56c94c9b9ec7d41fbc58918a1c8f7f74859454940ac7d851 bwa samtools
galaxy.tools.deps.conda_util ERROR 2020-01-23 10:47:55,668 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Failed to execute command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name mulled-v1-9057fa4fae7a852d56c94c9b9ec7d41fbc58918a1c8f7f74859454940ac7d851 bwa samtools
Traceback (most recent call last):
  File "lib/galaxy/tools/deps/conda_util.py", line 235, in exec_command
    return self.shell_exec(cmd, env=env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 58, in shell
    p = shell_process(cmds, env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 88, in shell_process
    p = subprocess.Popen(cmds, **popen_kwds)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
galaxy.tools.deps.resolvers.conda DEBUG 2020-01-23 10:47:55,668 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Removing failed conda install of [CondaTarget[bwa,unversioned], CondaTarget[samtools,unversioned]]
galaxy.tools.deps.conda_util DEBUG 2020-01-23 10:47:55,668 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Executing command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __bwa@_uv_ bwa
galaxy.tools.deps.conda_util ERROR 2020-01-23 10:47:55,683 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Failed to execute command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __bwa@_uv_ bwa
Traceback (most recent call last):
  File "lib/galaxy/tools/deps/conda_util.py", line 235, in exec_command
    return self.shell_exec(cmd, env=env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 58, in shell
    p = shell_process(cmds, env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 88, in shell_process
    p = subprocess.Popen(cmds, **popen_kwds)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
galaxy.tools.deps.resolvers.conda DEBUG 2020-01-23 10:47:55,683 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Removing failed conda install of bwa, version 'None'
galaxy.tools.deps.conda_util DEBUG 2020-01-23 10:47:55,683 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Executing command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __samtools@_uv_ samtools
galaxy.tools.deps.conda_util ERROR 2020-01-23 10:47:55,700 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Failed to execute command: /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __samtools@_uv_ samtools
Traceback (most recent call last):
  File "lib/galaxy/tools/deps/conda_util.py", line 235, in exec_command
    return self.shell_exec(cmd, env=env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 58, in shell
    p = shell_process(cmds, env, **kwds)
  File "lib/galaxy/tools/deps/commands.py", line 88, in shell_process
    p = subprocess.Popen(cmds, **popen_kwds)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
galaxy.tools.deps.resolvers.conda DEBUG 2020-01-23 10:47:55,700 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Removing failed conda install of samtools, version 'None'
galaxy.tools.deps DEBUG 2020-01-23 10:47:55,704 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Dependency biom-format not found.
galaxy.tools.deps DEBUG 2020-01-23 10:47:55,705 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Dependency biom-format not found.
galaxy.tools.deps DEBUG 2020-01-23 10:47:55,705 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Dependency biom-format not found.
galaxy.tools.deps DEBUG 2020-01-23 10:47:55,705 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Dependency biom-format not found.
galaxy.tools.deps DEBUG 2020-01-23 10:47:55,705 [p:111907,w:1,m:0] [uWSGIWorker1Core1] Dependency
4

This error suggests that Conda is not installed. The run.sh script should have done this for you before Galaxy started and placed it in /apps/galaxy/galaxy_app/database/dependencies/_conda. You may notice an error of this at the beginning of your startup log, or you can try the process manually with:

$ cd /apps/galaxy/galaxy_app
$ . galaxy_app/.venv/bin/activate
$ python ./scripts/manage_tool_dependencies.py init_if_needed
5

@Nate: thanks. I will try this out in the morning, however I did verify conda is installed at the location you mention.

galaxy@PB-DEV-01:~$ ls -altr /apps/galaxy/galaxy_app/database/dependencies/_conda
total 32
-rw-r--r--  1 galaxy galaxy 4134 Apr 17  2019 LICENSE.txt
drwxr-xr-x  2 galaxy galaxy   42 Aug 23 16:07 compiler_compat
drwxr-xr-x  3 galaxy galaxy   29 Aug 23 16:07 x86_64-conda_cos6-linux-gnu
drwxr-xr-x 10 galaxy galaxy  152 Aug 23 16:07 share
drwxr-xr-x  9 galaxy galaxy 4096 Aug 23 16:07 include
drwxr-xr-x  3 galaxy galaxy   30 Aug 23 16:07 shell
drwxr-xr-x  4 galaxy galaxy   47 Aug 23 16:07 etc
drwxr-xr-x  2 galaxy galaxy   10 Aug 23 16:07 envs
drwxr-xr-x  3 galaxy galaxy  178 Aug 23 16:07 ssl
drwxr-xr-x 39 galaxy galaxy 4096 Aug 23 16:07 pkgs
drwxr-xr-x 15 galaxy galaxy 4096 Aug 23 16:07 lib
drwxr-xr-x  2 galaxy galaxy 4096 Aug 23 16:07 conda-meta
drwxr-xr-x  2 galaxy galaxy   27 Aug 23 16:07 condabin
drwxr-xr-x  2 galaxy galaxy 4096 Aug 23 16:07 bin
drwxr-xr-x 15 galaxy galaxy  286 Aug 23 16:07 .
drwxr-xr-x  7 galaxy galaxy   98 Jan 23 14:20 ..


galaxy@PB-DEV-01:/apps/galaxy/galaxy_app$ find . -type f -name conda.sh -print
./database/dependencies/_conda/etc/profile.d/conda.sh
./database/dependencies/_conda/lib/python3.7/site-packages/conda/shell/etc/profile.d/conda.sh
./database/dependencies/_conda/pkgs/conda-4.6.14-py37_0/etc/profile.d/conda.sh
./database/dependencies/_conda/pkgs/conda-4.6.14-py37_0/lib/python3.7/site-packages/conda/shell/etc/profile.d/conda.sh
6

Ok, please also verify that /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda is present, and check the output of /apps/galaxy/galaxy_app/database/dependencies/_conda/bin/conda info.

1 Like
7

Thank you @Nate! The package was installed in the correct location but the conda script was failing because the very first line in the script was pointing to the wrong python interpreter. This script is obviously configured at Galaxy installation time so I’ll have to make a note of this dependency. You pointed right to the problem! Here is the site I found the fix on: https://stackoverflow.com/questions/39884499/conda-command-will-prompt-error-bad-interpreter-no-such-file-or-directory

8

Interesting, that script is installed automatically when conda is installed, which we do in a fairly standard manner. I’m not sure how it would’ve been corrupted, but I’m glad that it’s fixed.

9

We just ran into another potential pathing error with rsync (see below) so I think it is time to do a fresh install so any new pathing issues are proactively avoided.

In the future if I do a backup of the entire galaxy directory and restore it on another server but in a different location is there a parameter I can set to for Galaxy to do a relative path resolution or should I use unix hard/soft links to simulate the same base installation directory?

galaxy.jobs DEBUG 2020-01-24 10:17:14,458 [p:13951,w:1,m:0] [JobHandlerQueue.monitor_thread] (1) Persisting job destination (destination id: local)
galaxy.jobs DEBUG 2020-01-24 10:17:14,459 [p:13951,w:1,m:0] [JobHandlerQueue.monitor_thread] (1) Working directory for job is: /apps/galaxy/galaxy_app/database/jobs_directory/000/1
galaxy.jobs.runners DEBUG 2020-01-24 10:17:14,470 [p:13951,w:1,m:0] [JobHandlerQueue.monitor_thread] Job [1] queued (34.158 ms)
galaxy.jobs.handler INFO 2020-01-24 10:17:14,475 [p:13951,w:1,m:0] [JobHandlerQueue.monitor_thread] (1) Job dispatched
galaxy.jobs.runners ERROR 2020-01-24 10:17:14,534 [p:13951,w:1,m:0] [LocalRunner.work_thread-0] (1) Failure preparing job
Traceback (most recent call last):
  File "lib/galaxy/jobs/runners/__init__.py", line 224, in prepare_job
    job_wrapper.prepare()
  File "lib/galaxy/jobs/__init__.py", line 863, in prepare
    tool_evaluator.set_compute_environment(compute_environment, get_special=get_special)
  File "lib/galaxy/tools/evaluation.py", line 118, in set_compute_environment
    out_data=out_data, tool=self.tool, param_dict=incoming)
  File "lib/galaxy/tools/__init__.py", line 1726, in call_hook
    return code(*args, **kwargs)
  File "database/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_rsync_g2/e0329ab30f6d/data_manager_rsync_g2/data_manager/data_manager_rsync.py", line 88, in exec_before_job
    target_dir, tool_path, relative_target_dir = tdtm.get_target_install_dir(tool_shed_repository)
  File "lib/tool_shed/tools/data_table_manager.py", line 111, in get_target_install_dir
    tool_path, relative_target_dir = tool_shed_repository.get_tool_relative_path(self.app)
AttributeError: Error in 'Rsync with g2' hook 'exec_before_job', original message: 'NoneType' object has no attribute 'get_tool_relative_path'
galaxy.tools.error_reports DEBUG 2020-01-24 10:17:14,598 [p:13951,w:1,m:0] [LocalRunner.work_thread-0] Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7ff148a06690> generated response None
10.10.10.87 - - [24/Jan/2020:10:17:17 -0400] "GET /api/histories/ebfb8f50c6abde6d/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=1970-01-01T00%3A00%3A00.000Z&qv=False&qv=False HTTP/1.1" 200 - "http://10.10.200.140:8080/root?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2Fdevteam%2Fdata_manager_rsync_g2%2Fdata_manager_rsync_g2%2F0.0.3" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"