put() and get() in jupyterlab notebooks in galaxy

,
1

Hi, I have a very basic notebook running in our local galaxy portal with interactive tools.

As you see, put() and get() functions are not recognized:

Screen Shot 2021-08-18 at 13.30.25
Screen Shot 2021-08-18 at 13.30.25859×789 63 KB

The tool-wrapper is the following taken from some official galaxy jupyterlab/jupyternotebook tutorial page.

What might I be missing to be able to have get() and put() working?

Tool wrapper:

<tool id="interactive_tool_jupyter_notebook" tool_type="interactive" name="General Jupyter Notebook" version="0.1">
  <description>Default empty notebook, reuse a previous one, or upload a new</description>
  <requirements>
    <container type="docker">quay.io/bgruening/docker-jupyter-notebook:2021-03-05</container>
  </requirements>
    <entry_points>
        <entry_point name="Jupyter Interactive Tool" requires_domain="True">
            <port>8888</port>
            <url>lab</url>
        </entry_point>
    </entry_points>
    <environment_variables>
        <environment_variable name="HISTORY_ID">$__history_id__</environment_variable>
        <environment_variable name="REMOTE_HOST">$__galaxy_url__</environment_variable>
        <environment_variable name="GALAXY_WEB_PORT">8080</environment_variable>
        <environment_variable name="GALAXY_URL">$__galaxy_url__</environment_variable>
        <environment_variable name="API_KEY" inject="api_key" />
    </environment_variables>
    <command detect_errors="aggressive"><![CDATA[
        #import re
        export GALAXY_WORKING_DIR=`pwd` &&
        mkdir -p ./jupyter/outputs/ &&
        mkdir -p ./jupyter/data &&

        #set $cleaned_name = re.sub('[^\w\-\.]', '_', str($input.element_identifier))
        ln -sf '$input' './jupyter/data/${cleaned_name}' &&

        ## change into the directory where the notebooks are located
        cd ./jupyter/ &&
        export PATH=/home/jovyan/.local/bin:\$PATH &&

        #if $mode.mode_select == 'scratch':
            ## copy default notebook
            cp '$__tool_directory__/default_notebook.ipynb' ./ipython_galaxy_notebook.ipynb &&
            jupyter trust ./ipython_galaxy_notebook.ipynb &&
            jupyter lab --allow-root --no-browser  --ServerApp.token=''  &&
            cp ./ipython_galaxy_notebook.ipynb '$jupyter_notebook'

        #else:
            #set $cleaned_name = re.sub('[^\w\-\.]', '_', str($input.element_identifier))
            cp '$mode.ipynb' ./${cleaned_name}.ipynb &&
            jupyter trust ./${cleaned_name}.ipynb &&

            #if $mode.run_it:
                jupyter nbconvert --to notebook --execute --output ./ipython_galaxy_notebook.ipynb --allow-errors  ./*.ipynb &&
            #else:
                jupyter lab --allow-root --no-browser  --ServerApp.token='' &&
            #end if
            cp ./ipython_galaxy_notebook.ipynb '$jupyter_notebook' 

        #end if
    ]]>
    </command>
    <inputs>

        <conditional name="mode">
            <param name="mode_select" type="select" label="Do you already have a notebook?" help="If not, no problem we will provide you with a default one.">
                <option value="scratch">Start with a fresh notebook</option>
                <option value="previous">Load a previous notebook</option>
            </param>
            <when value="scratch"/>
            <when value="previous">
                <param name="ipynb" type="data" format="ipynb" label="IPython Notebook"/>
                <param name="run_it" type="boolean" truevalue="true" falsevalue="false" label="Execute notebook and return a new one."
                    help="This option is useful in workflows when you just want to execute a notebook and not dive into the webfrontend."/>
            </when>
        </conditional>
        <param name="input" type="data" optional="true" label="Include data into the environment"/>

    </inputs>
    <outputs>
        <data name="jupyter_notebook" format="ipynb" label="Executed Notebook"></data>
    </outputs>
    <tests>
        <test expect_num_outputs="1">
            <param name="mode" value="previous" />
            <param name="ipynb" value="test.ipynb" />
            <param name="run_it" value="true" />
            <output name="jupyter_notebook" file="test.ipynb" ftype="ipynb"/>
        </test>
    </tests>
    <help>
    The Jupyter Notebook is an open-source web applicati...
1 Like
2

Cross-referenced post: galaxyproject/admins - Gitter

More feedback once this is looked into more (soon!).

1 Like
3

Any ideas yet what is going on with put and get not working? As I asked in the gitter post, are these python methods? Do I maybe need to import some module?

1 Like
4

Hi @maikenp

Sorry for the delay. The application was in the middle of an update at the usegalaxy.* servers and I wanted to cross-test. The functions put and get are now working at those. I’m not exactly sure what is needed for configuration at other places. The GTN tutorials that cover the usage are not quite current – the tools are now in the tool panel, not under the Visualization masthead menu.

It looks like you are getting some help at Gitter – please see the latest message from last week: https://gitter.im/galaxyproject/admins?at=6139ff9263dca818914e1056

5

There are lots of tutorials covering the usage of ITs, but none covering building ITs.

get and put are some special magic in the jupyter notebook we build. We configure a special environment for the notebook which loads those commands: docker-jupyter-notebook/ipython-profile.py at a48b767cd7d030b743605bb7984eda5eae76eb69 · bgruening/docker-jupyter-notebook · GitHub

So your tool XML looks fine, but it could be something about the specific container image you’re using. In that container it SHOULD be loaded by default in the default user’s home dir.

I see that you’re missing this statement:

export HOME=/home/jovyan/ &&

there’s a strong change that could be causing it. Generally looking at your XML it has diverged quite far from the in-galaxy jupyter notebook, you might try diffing those files and seeing if some of those changes you’ve made are relevant to the issues.

If it’s ok for everyone, maybe let’s keep the conversation here, it’s a bit easier to see what’s going on than scrolling through hundreds of messages to tease apart the thread.

1 Like
6

Thanks, yes I much prefer disucssing here, the gitter is not ideal for discussions!

Let’s see thanks, I will add the export.

Could you provide a link to the in-galaxy jupyter notebook?

For our notebook docker image, we used as a basis: GitHub - bgruening/docker-jupyter-notebook: Jupyter running in a docker container. This image can be used to integrate Jupyter into Galaxy but added some more software.

7

For our notebook docker image, we used as a basis: GitHub - bgruening/docker-jupyter-notebook: Jupyter running in a docker container. This image can be used to integrate Jupyter into Galaxy but added some more software.

The xml copy-pasted in this post is the one from the installation found in /srv/galaxy/server/tools/interactive/interactivetool_jupyter_notebook.xml - with one change - the <url>ipython/lab</url> which had to be changed to <url>lab</url> and some outdated ServerApp replacement to NotebookApp - see some earlier discussion on gitter.

[root@galaxy-hepp interactive]# diff /srv/galaxy/server/tools/interactive/interactivetool_jupyter_notebook.xml.orig interactivetool_jupyter_notebook.xml
1,4c1,5
< <tool id="interactive_tool_jupyter_notebook" tool_type="interactive" name="Interactive Jupyter Notebook" version="0.1">
<     <requirements>
<         <container type="docker">quay.io/bgruening/docker-jupyter-notebook:2021-03-05</container>
<     </requirements>
---
> <tool id="interactive_tool_jupyter_notebook" tool_type="interactive" name="General Jupyter Notebook" version="0.1">
>   <description>Default empty notebook, reuse a previous one, or upload a new</description>
>   <requirements>
>     <container type="docker">quay.io/bgruening/docker-jupyter-notebook:2021-03-05</container>
>   </requirements>
8c9
<             <url>ipython/lab</url>
---
>             <url>lab</url>
35c37
<             jupyter lab --allow-root --no-browser  &&
---
>             jupyter lab --allow-root --no-browser  --ServerApp.token=''  &&
46c48
<                 jupyter lab --allow-root --no-browser --NotebookApp.shutdown_button=True &&
---
>                 jupyter lab --allow-root --no-browser  --ServerApp.token='' &&
8

Answering myself: I can find the most recent jupyter_notebook here: galaxy/tools/interactive at dev · galaxyproject/galaxy · GitHub

9

Update, using the newest toolwrapper ( ) the put function is now recognized. Still having some issues using it, will get back with details.

Screen Shot 2021-09-14 at 11.51.45
Screen Shot 2021-09-14 at 11.51.451184×224 34.1 KB

10

Would you like me to create a new help issue - since now put() is indeed recognized, but I get this:

Screen Shot 2021-09-14 at 11.57.47
Screen Shot 2021-09-14 at 11.57.471259×1167 142 KB

11

The key point being I think the use of the quay.io/bgruening/docker-jupyter-notebook:ie2 image.

12

Ok! Progress :slight_smile: Did you set galaxy_infrastructure_url, this can be important for the commands to connect to the galaxy server.

13

Dont think so. Where would I set that, and what should the value be?

14

Ah, configuration option, right, lets see…

15

So, I set
galaxy_infrastructure_url: http://localhost:8000
since I see in our nginx setup:

[root@galaxy-hepp config]# cat /etc/nginx/conf.d/galaxy-gie-proxy.conf
server {
    # Listen on port 443
    listen       443 ssl;
(.... various ssl settings)
    client_max_body_size 2G;
    proxy_intercept_errors on;

    fastcgi_buffers 16 16k;
    fastcgi_buffer_size 32k;

    #Proxy all requests to the GIE Proxy application
    location / {
       proxy_redirect off;
       proxy_http_version 1.1;
       proxy_set_header Host $host;
       #proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_pass http://localhost:8000;
    }
}

But in the galaxy.conf I see:

[root@galaxy-hepp config]# cat /etc/nginx/conf.d/galaxy.conf 
server {
  listen 443 ssl;
  server_name  galaxy-hepp.hpc.uio.no;
  
.....(various settings)

  location / {
    uwsgi_pass 127.0.0.1:8080;
    uwsgi_param UWSGI_SCHEME $scheme;
    include uwsgi_params;
  }
}

Which would be the correct to use?

I read from the config doc pages that the default is http://localhost:8080

With the http://localhost:8000 the same error message occurs as the one in the screenshot above.

16

@hexylena do you have some advice on what would be the expected galaxy_infrastructure_url based on our nginx setup?

In the tool wrapper for the interactive tool there is Screen Shot 2021-09-16 at 09.49.05

entry point port 8888 and galaxy_web_port 8080. Should I be using any of these for galaxy_infrastructure_url?

Thanks!
Maiken

17

Hi @maikenp sorry for the delay, too many things going on.

Infrastructure url should really be the address you access galaxy at, I see listen 443 in your nginx settings, so for example EU sets it to

galaxy_infrastructure_url: 'https://usegalaxy.eu'

Then the container will use the external address to contact galaxy’s API which is a lot more reliable than trying to figure out the internal networking path to reach galaxy (what it initially tries within the container)

18

Thanks a lot for getting back to me, and yes, understand completely the delay.

Ok, I understand, thanks. Will give that a try once we have our system up and going again (going through some reinstallations!)

19

Ok, unfortunately that did not help at all. Exact same error now using
galaxy_infrastructure_url: https://galaxy-hepp.hpc.uio.no

Screen Shot 2021-09-16 at 13.38.35
Screen Shot 2021-09-16 at 13.38.351143×880 121 KB

What else could be wrong?

20

Could there be some

  • selinux issues?
  • firewall issues?
  • docker argument issues?

I have tried looking for hints in various log files, but not really finding anything of interest. So not at all sure where I should be looking.

If it is relevant: the galaxy server sends the job via slurm to a compute node, and jupterlab runs inside the docker container on this compute node.

Also not that (not sure if it is a hint or not) that in the docker container I see:

root@d8087e912320:/storage/galaxy/jobs_directory/001/1251/working/jupyter# curl https://galaxy-hepp.hpc.uio.no/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.