Galaxy local python Module Not Found Error

[Andrea_Furlani]

Hi everyone,
I’m trying to make a new tool on my local Galaxy using python.
I managed to create the python script and the xml file for the tool:

<tool id="get_pheno" name="Get phenotypes" version="0.1.0">
  <description>for a gene from IMPC</description>
  <requirements>
    <requirement type="package">tabulate</requirement>
	<requirement type="package">IPython</requirement>
  </requirements>
  <command>python3 '${__tool_directory__}/get_pheno.py' '$input' '$output'</command>
  <inputs>
	  <param name="input" type='text' label='Input gene:'/>
  </inputs>	  
  <outputs>
    <data format="tabular" name="output" />
  </outputs>

  <help>
This tool retrieves a list of phenotypes related to a gene entered by the user. The input must be an MGI id.
  </help>

</tool>

And this is the code I’m using:

import sys
import subprocess
import requests
import tabulate
from IPython.display import HTML, display

impc_api_url = "https://www.gentar.org/impc-dev-api/"
impc_api_search_url = f"{impc_api_url}/genes"
impc_api_gene_bundle_url = f"{impc_api_url}/geneBundles"

# 1-Given a gene id, retrieve all the phenotypes related to it (id and name)

mgi_accession_id = str(sys.argv[1])

 gene_url = f"{impc_api_search_url}/{mgi_accession_id}"
gene_data = requests.get(gene_url).json()

p_list = gene_data['significantMpTermNames']
display(HTML(tabulate.tabulate([i for i in p_list], headers="Phenotype Name", tablefmt='html')))

When running the tool, Galaxy is retrieving me this error:

Traceback (most recent call last):
  File "/Users/.../galaxy/tools/personal_tools/get_pheno.py", line 4, in <module>
    import tabulate
ModuleNotFoundError: No module named 'tabulate'

Now, looking in the admin section for the dependencies, I can see that the two packages are signed as managed by conda.
How can I solve this error?
Thank you very much for the help

EDIT: By adding the version number to the packages it seems that everything it’s fine from the dependencies manager, but now I’m getting this error:

Conda dependency seemingly installed but failed to build job environment.

[gbbio]

@Andrea_Furlani

Galaxy is showing the tool correctly, but it is not asking me for the input (which I think it’s because I need to put it also inside the xml file, but I need a little help to understand how to pass a string to the code)

Think this can help: Galaxy Tool XML File — Galaxy Project 23.3.dev0 documentation
For example a file as input:

<param name="input" type="data" format="fasta" label="fasta file"/>

Or text:

<param name="my_text_param" type="text" label="Genotype param value" />

And in the command it will look like:

<command>python3 '${__tool_directory__}/get_pheno.py' $input $my_text_param $output</command>

Debugging the error is not so easy (for me) but for now you can try this and maybe someone else gives a better answer:
Try to activate the environment (with the same conda that galaxy is using) from the command line (it starts with ‘mulled-v1-’) and try to execute your script. Also inside that evironment check the python version. I have the feeling you need to add python as a requirement.

<requirement type="package" version="3.8">python</requirement>

Or whatever python version.

After adding a specific python version as requirement your command should be:

<command>python '${__tool_directory__}/get_pheno.py' '$input' '$output'</command>

I think with python3 you are now using the python version of the system instead of the env.

There are also not many cases where you want text as a input parameter. See this for alternative:

<param name="paramfoo" type="select" multiple="false" label="Parameter 1">
          <option value="param1" selected="true">Some param 1</option>
          <option value="param2" selected="true">Some param 2</option>
          <option value="param3" selected="true">Some param 3</option>
</param>

[Andrea_Furlani]

Thank you for your reply. For the input part I found a solution on my own and it’s working as intended.
Unfortunately I’m still having issues with the packages. I have the exact same error by using python as additional requirement. And again in the dependencies manager it says that all the packages are correctly installed.

[gbbio]

(I see your edit now and removed my previous answer)
Did you let galaxy install conda or you pointed to an existing installation? And which version of galaxy are you using? And do you have other tools that do work?

Not 100% sure if it was the case with this specific error (I am not a galaxy developer). But removing the environment or packages and re-execute the tool can help. But this is a quick trick and dependend on the version I thought.

[Andrea_Furlani]

I’m using the last version and I let Galaxy to install Conda. Moreover, I tried to delete and reinstall Galaxy but again in the manager it says that all dependencies are installed but when I run the tool it says that “tabulate” was not found.
I tried other tools but they are working normally.

[gbbio]

@Andrea_Furlani

but when I run the tool it says that “tabulate” was not found

I thought this was solved (based on your latest edit). And your new problem was:

Conda dependency seemingly installed but failed to build job environment.

EDIT: By adding the version number to the packages it seems that everything it’s fine from the dependencies manager, but now I’m getting this error:

Can you edit your original question with the latest xml?

And did you already tried:

<command><![CDATA[which python; python '${__tool_directory__}/get_pheno.py' '$input' '$output']]></command>

The result of which you can see inside the history ‘block’ after folding it open (clicking on it).

[Andrea_Furlani]

I don’t know why but without changing anything the old error just come back…
Using the last command you suggested me I got this:

/Users/.../Desktop/Galaxy/galaxy/.venv/bin/python
Traceback (most recent call last):
  File "/Users/andrea/Desktop/Galaxy/galaxy/tools/personal_tools/get_pheno.py", line 3, in <module>
    import tabulate
ModuleNotFoundError: No module named 'tabulate

I’m adding the xml code here since I can’t edit the original post:

<tool id="get_pheno" name="Get phenotypes" version="0.1.0">
  <description>for a gene from IMPC</description>
  <requirements>
    <requirement type="package" version="3.8">python</requirement>
	<requirement type="package" version="0.8.9">tabulate</requirement>
	<requirement type="package" version='7.12.0'>IPython</requirement>
  </requirements>
  <command><![CDATA[which python; python '${__tool_directory__}/get_pheno.py' '$input' '$output']]></command>
  <inputs>
	  <param name="input" type='text' label='Input gene:'/>
  </inputs>	  
  <outputs>
    <data format="tabular" name="output" />
  </outputs>

  <help>
This tool retrieves a list of phenotypes related to a gene entered by the user. The input must be an MGI id.
  </help>

</tool>

[gbbio]

This is the output of which python:

@Andrea_Furlani

/Users/.../Desktop/Galaxy/galaxy/.venv/bin/python

So the tool is using the python installation of the virtualenv, instead of the conda package. So now we know the problem. I do not know a solution on top of my head now. But will think about it, my very first hunch was a problem with conda but you said everything was installed and you see the packages in the manage dependencies page.

If you start galaxy with the command sh run.sh --daemon by default a galaxy.log file will be made. Maybe if you start galaxy like that and execute the tool again this log file will give more hints.

[Andrea_Furlani]

I’m not sure about this because I’m new to Galaxy too. From the dependencies screen I can see that the packages are installed and they are managed by Conda.
In the Galaxy yml file I also setted conda_auto_init and conda_auto_install on true.

EDIT: By changing the default settings in the galaxy.yml file I’m no more able to install new dependencies and also Galaxy cannot find the already installed ones

[gbbio]

It is hard to debug because I dont know what you changed. Did you already checked the log?

[Andrea_Furlani]

I don’t know where to look for a log, I looked for the terminal output and the only interesting part is this one preceding the error:

[ "$GALAXY_VIRTUAL_ENV" = "None" ] && GALAXY_VIRTUAL_ENV="$_GALAXY_VIRTUAL_ENV"; _galaxy_setup_environment True
python "metadata/set.py"; sh -c "exit $return_code"

[gbbio]

You can start galaxy with the command sh run.sh --daemon this will start galaxy in the background and by default it creates the log file galaxy.log. To stop galaxy you can use the command sh run.sh --stop-daemon.

[Andrea_Furlani]

Ok, here’s the log file after launching Galaxy and run the tool:

galaxy.log

[gbbio]

Are you sure this is the command in your xml?

<command><![CDATA[python '${__tool_directory__}/get_pheno.py' '$input' '$output']]></command>

Because in the log I see python3 instead of python

[Andrea_Furlani]

I used both “python” and “python3” but the result is the same unfortunately

[gbbio]

oke, there is one last thing I could think off (considering you mostly use default settings).
When you start galaxy from the commandline do you see something like (base) at the beginning of the line where you type in your command? If yes, first use command"

conda deactivate

and then:

sh run.sh --daemon

Ofcourse first stop galaxy before you start it.

[Andrea_Furlani]

Yes I have (base) at the beginning of the line. I used deactivate and then tried again to launch Galaxy but the result didn’t change

[gbbio]

oke, well starting galaxy while you are in the base environment mostly causes the error you having. So you completely stopped galaxy, made sure you are not in base, started galaxy again. What is the output of which python?

[Andrea_Furlani]

This time I’m getting only this lines and the error it’s not completely displayed:

/Users/andrea/Desktop/Galaxy/galaxy/database/dependencies/_conda/envs/mulled-v1-397f595702d6efb72b51fc839fb61094a6e80efe699cb29e7bc02f178c4e80ea/bin/python
Traceback (most recent call last):
  File "/Users/andrea/Desktop/Galaxy/galaxy/tools/personal_tool

Looking in the log file this time the package not recognized it’s “requests” which is the one before “tabulate” in the code. By adding it to the requirements, if I go to the “manage dependencies” tab here I have:

Python 3.8 handled by conda
requirements 2.7.0 unresolved
tabulate 0.8.9 unresolved
IPython 7.12.0 unresolved

And if I check the tool to install the dependencies nothing happen.
Looking the log file I think it’s because I need to change some settings:

FileNotFoundError: [Errno 2] No such file or directory: '<conda_prefix>/bin/conda': '<conda_prefix>/bin/conda'

EDIT:
Ok by adjusting the settings now I have again this error:

Conda dependency seemingly installed but failed to build job environment.

[gbbio]

For now I would recommend to turn off galaxy, remove the galaxy.log file and start it again (run.sh --daemon)

Then you can try to remove the environment or/and packages from the admin menu (uninstall). It is possible that uninstallation does not work but we will see that later. And after that you can try to execute the tool again. If it does not work there should be something usefull in the log file.

EDIT:
This may also help

Per-job conda environment creation errors

We’re in the middle of setting up a new Galaxy server (18.09) and currently installing tools - mixed ToolShed and locally developed code. Quite a lot of our tools are currently failing with error: Conda dependency seemingly installed but failed to build job environment. This includes BCFTools, GATK (3.x), and others. Looking through the manage dependencies panel, the ones with issues often have multiple dependency requirements, with Conda as the resolver and separate environment paths listed…