put() and get() in jupyterlab notebooks in galaxy

,
#15

So, I set
galaxy_infrastructure_url: http://localhost:8000
since I see in our nginx setup:

[root@galaxy-hepp config]# cat /etc/nginx/conf.d/galaxy-gie-proxy.conf
server {
    # Listen on port 443
    listen       443 ssl;
(.... various ssl settings)
    client_max_body_size 2G;
    proxy_intercept_errors on;

    fastcgi_buffers 16 16k;
    fastcgi_buffer_size 32k;

    #Proxy all requests to the GIE Proxy application
    location / {
       proxy_redirect off;
       proxy_http_version 1.1;
       proxy_set_header Host $host;
       #proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_pass http://localhost:8000;
    }
}

But in the galaxy.conf I see:

[root@galaxy-hepp config]# cat /etc/nginx/conf.d/galaxy.conf 
server {
  listen 443 ssl;
  server_name  galaxy-hepp.hpc.uio.no;
  
.....(various settings)

  location / {
    uwsgi_pass 127.0.0.1:8080;
    uwsgi_param UWSGI_SCHEME $scheme;
    include uwsgi_params;
  }
}

Which would be the correct to use?

I read from the config doc pages that the default is http://localhost:8080

With the http://localhost:8000 the same error message occurs as the one in the screenshot above.

#16

@hxr do you have some advice on what would be the expected galaxy_infrastructure_url based on our nginx setup?

In the tool wrapper for the interactive tool there is Screen Shot 2021-09-16 at 09.49.05

entry point port 8888 and galaxy_web_port 8080. Should I be using any of these for galaxy_infrastructure_url?

Thanks!
Maiken

#17

Hi @maikenp sorry for the delay, too many things going on.

Infrastructure url should really be the address you access galaxy at, I see listen 443 in your nginx settings, so for example EU sets it to

galaxy_infrastructure_url: 'https://usegalaxy.eu'

Then the container will use the external address to contact galaxy’s API which is a lot more reliable than trying to figure out the internal networking path to reach galaxy (what it initially tries within the container)

#18

Thanks a lot for getting back to me, and yes, understand completely the delay.

Ok, I understand, thanks. Will give that a try once we have our system up and going again (going through some reinstallations!)

#19

Ok, unfortunately that did not help at all. Exact same error now using
galaxy_infrastructure_url: https://galaxy-hepp.hpc.uio.no

Screen Shot 2021-09-16 at 13.38.35
Screen Shot 2021-09-16 at 13.38.351143×880 121 KB

What else could be wrong?

#20

Could there be some

  • selinux issues?
  • firewall issues?
  • docker argument issues?

I have tried looking for hints in various log files, but not really finding anything of interest. So not at all sure where I should be looking.

If it is relevant: the galaxy server sends the job via slurm to a compute node, and jupterlab runs inside the docker container on this compute node.

Also not that (not sure if it is a hint or not) that in the docker container I see:

root@d8087e912320:/storage/galaxy/jobs_directory/001/1251/working/jupyter# curl https://galaxy-hepp.hpc.uio.no/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
#21

Hi again, sorry to use the ping, but we would love to get the put() and get() working @hxr

Could the issue be related to some missing docker options? We have this currently in our job_conf.xml

Screen Shot 2021-09-17 at 11.27.34
Screen Shot 2021-09-17 at 11.27.34907×183 26.5 KB

#22

The container has a DEBUG option, I believe, I should have thought of that sooner. If you set that environment variable, then the get/put commands will print out additional logging information that might help you diagnose the issue.

I think you’ll need to set that in the tool xml.

#23

Try adding something like <environment_variable name="DEBUG">True</environment_variable>

#24

xref galaxy_ie_helpers/galaxy_ie_helpers at master · bgruening/galaxy_ie_helpers · GitHubinit.py#L12

#25

Yes thanks, did that actually manually in the container. But where is actually this logged to? I.e. where should I be able to find the log file?

#26

it should log inside the notebook, whenever you run the command. If not, you can try running the get command in a terminal inside jupyter

#27

Right, I see no logging output at all in the notebook, only the exception.

In the running container I did:

root@f0557899417c:/storage/galaxy/jobs_directory/001/1252/working/jupyter# echo $DEBUG
true

In addition I manually edited the init.py :

#!/usr/bin/env python                                                                                                                                                                                              
from bioblend.galaxy import objects
from bioblend.galaxy import GalaxyInstance
from bioblend.galaxy.histories import HistoryClient
from bioblend.galaxy.datasets import DatasetClient
import subprocess
import argparse
import re
import os
from string import Template
import logging
DEBUG = os.environ.get('DEBUG', "False").lower() == 'true'
#if DEBUG:                                                                                                                                                                                                         
#    logging.basicConfig(level=logging.DEBUG)                                                                                                                                                                      
#logging.getLogger("bioblend").setLevel(logging.CRITICAL)                                                                                                                                                          
logging.basicConfig(level=logging.DEBUG)
log = logging.getLogger()

And I have added some more log.debug messages in the code. I see no trace of the logging output in the notebook though.

#28

This is how it looks, you see the printout of the code where I added log.debug message, but the logging message is not actually shown in the notebook:

Screen Shot 2021-09-17 at 12.22.46
Screen Shot 2021-09-17 at 12.22.461834×983 147 KB

#29

Also, in a terminal inside jupyter I get this (put taken as bash command)

Screen Shot 2021-09-17 at 12.26.45
Screen Shot 2021-09-17 at 12.26.451222×406 43.9 KB

#30

try put -h in the terminal

1 Like
#31

yes, sorry, realized. We get debug output now in the terminal, thanks!

1 Like
#32

So this is the reason:

root@f0557899417c:/storage/galaxy/jobs_directory/001/1252/working/jupyter# put --history-id 63cd3858d057a6d1  -p /storage/galaxy/jobs_directory/001/1252/working/jupyter/testfile.txt 
DEBUG:root:here I am
DEBUG:root:Host IP determined to be b'172.17.0.1\n'
DEBUG:root:TestURL url=https://galaxy-hepp.hpc.uio.no obj=True
DEBUG:bioblend:GET - attempts left: 1; retry delay: 10
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): galaxy-hepp.hpc.uio.no:443
ERROR:bioblend:HTTPSConnectionPool(host='galaxy-hepp.hpc.uio.no', port=443): Max retries exceeded with url: /api/histories/63cd3858d057a6d1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)'))), 0 attempts left
DEBUG:root:TestURL url=https://galaxy-hepp.hpc.uio.no state=failure
DEBUG:root:TestURL url=http://b'172.17.0.1':8080 obj=True
DEBUG:bioblend:GET - attempts left: 1; retry delay: 10
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): b'172.17.0.1':8080
ERROR:bioblend:HTTPConnectionPool(host="b'172.17.0.1'", port=8080): Max retries exceeded with url: /api/histories/63cd3858d057a6d1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f90c336bbe0>: Failed to establish a new connection: [Errno -2] Name or service not known')), 0 attempts left
DEBUG:root:TestURL url=http://b'172.17.0.1':8080 state=failure
DEBUG:root:do I see this?
Traceback (most recent call last):
  File "/opt/conda/bin/put", line 17, in <module>
    put( args.filepath, file_type=args.filetype, history_id=args.history_id )
  File "/opt/conda/lib/python3.8/site-packages/galaxy_ie_helpers/__init__.py", line 115, in put
    gi = get_galaxy_connection(history_id=history_id)
  File "/opt/conda/lib/python3.8/site-packages/galaxy_ie_helpers/__init__.py", line 101, in get_galaxy_connection
    raise Exception(msg)
Exception: Could not connect to a galaxy instance. Please contact your SysAdmin for help with this error

So then to figure out how to fix that.

#33

Could it be that the CA we use:
issuer=C = NL, O = GEANT Vereniging, CN = GEANT OV RSA CA 4
needs to be included in /etc/ssl/certs in the container?

I do not see it there.

#34

We have solved the put() error - the problem was due to the lack of certificate cocatination on our galaxy-hepp.uio.no server, so the CA certificate was not available for requests.

Now in place and put() does not give that error anymore.

(Another error is being dealt with related to permissions in /storage/galaxy/tmp - but that is another issue).

2 Likes