I have developed an IT (it happens to use PyQt5 for the GUI) and I’d like to be able to save a file that it generates directly to my local disk (not into the Galaxy workspace). Is this possible? What I’ve read so far seems to indicate that I need to run the Docker container using additional params, perhaps (Networking using the host network | Docker Docs).
Later, I’d like the tool to access/upload a file from my local disk too.
thanks, Randy
Hi @rheiland
To be clear, the IT is being launched from inside of Galaxy, but you want to be able to use that IT to read and write files to/from outside of Galaxy. Applications like RStudio can do this when run as an IT, so I don’t think it has to do with Galaxy necessarily, and the focus is more the network everything is hosted on.
And now I read your link 
– maybe compare what the RStudio Galaxy IT wrapper is doing since that is the example I can think of off hand. We also have the tutorial here which I am sure you have seen but someone else may read this later and be not sure sure where to look.
Let’s also cross-post this to the Dev chat to see if anyone has instructions that may be written up somewhere outside of the tutorials. → You're invited to talk on Matrix
Good questions!
@rheiland I think this is independent of Galaxy. Your webtool would need to support this. However, you seem to have a Desktop app (that we fake to be a web-application). I assume that Desktop app can only write to “a local” directory. “local” here means local to where the Docker container is running.
If you run the container on your Desktop (without Galaxy) you can indeed add a Docker parameter -v /home/foo/transfer_folder/:/output/ to the Docker start command. This parameter will mount a host (your Desktop) into the container (/output/). When you now save a file in your App to /output it will also appear in /home/foo/transfer_folder/ on your Desktop.
However, this does not work if the Docker container is running somewhere else and not on your Desktop.
I hope that makes sense.
Thanks @bjoern.gruening . After re-reading the Docker link I shared, I have to agree with you. So, I’m still stuck at not knowing how I can download/export a file, e.g., foo.dat, from my IT so that it can be saved to my Desktop. Optionally, if I can make foo.dat appear in the Galaxy History, but “on-demand” and not wait for the IT to exit, that would be acceptable for now.
I began looking at the BioBlend Python API, in part because my IT is written in Python, but so far it’s not obvious how that could do what I want. I don’t even understand its initial “GalaxyInstance” method and why an API key is necessary.
Optionally, if I can make foo.dat appear in the Galaxy History, but “on-demand” and not wait for the IT to exit
For example, you can use this small helper library. This will provide you a put() and get() function based on bioblend. We use that for example in the Jupyter IT to retrieve and export data to Galaxy.
The API key is needed, because the IT run completely isolated from Galaxy. It’s really an external service just started by Galaxy. Galaxy can insert an API key on the fly, so you should not worry about that.
Thanks @bjoern.gruening . I’ve been trying to figure out how I might use your put() in my IT. So, I pip installed your helpers in my Dockerfile:
and I copy/pasted the env vars (that you linked to) into my IT’s .xml:
and I attempt a File -> Download -> PhysiCell_settings.xml in my Studio IT:
and
It did not work - it killed my IT. I can take a closer look at my Galaxy log to see if I can learn anything from the errors, but before that, would you be able to tell me if I’m invoking the put with the correct/sufficient args? That file, PhysiCell_settings.xml, does indeed exist in the /config dir of the Docker image.
And if it’s not obvious, I’m just testing this on my local Galaxy server. Does that cause problems with this?
Can you just raise the error and see why it crashes?
Thanks - and sorry, I should have done that earlier. I’m getting a KeyError('HISTORY_ID'). So perhaps I’m misunderstanding what I need to do to make the <environment_variables> do their magic in my interactivetool_pcstudio.xml. I am not explicitly setting any env vars on my own.
That looks fine. Your XML also looks fine? Can you check you are really running this version.
This is working for the Juypter IT galaxy/tools/interactive/interactivetool_ml_jupyter_notebook.xml at release_24.2_europe · usegalaxy-eu/galaxy · GitHub
Can you maybe see which ENVs are set inside the container when Galaxy starts it?
Thanks for continuing to build up my confidence in trying to test all this.
Long story short: I had been attempting to test using a “myTools” (local) version of my IT and (previously) a version in the testtoolshed. I just now realized that even though I was selecting my IT in the MyTools section of the Galaxy Tools column, what was being loaded in the URL was the testtoolshed version:
http://127.0.0.1:8080/?tool_id=testtoolshed.g2.bx.psu.edu%2Frepos%2Frheiland%2Fphysicell_studio%2Finteractive_tool_pcstudio%2F0.1&version=latest
When I renamed
yoga:~/dev/galaxy-24.2.3/config$ mv tool_sheds_conf.xml tool_sheds_conf.xml-argh
and re-ran the server (sh run.sh), I now got the desired myTools version:
http://127.0.0.1:8080/?tool_id=interactive_tool_pcstudio&version=latest
and when I ran my IT and tested its File -> download feature, I at least got past the KeyError, but now I get Exception("Could not connect to a galaxy instance. Please contact your SysAdmin for help..." which is not good since I am the SysAdmin 
. Do you know what this might mean?
Then I tried the opposite approach - remove the local myTools version, update the IT in the testtoolshed, reinstall it in my local Galaxy as Admin, confirm that it is using that version due to a suffixed-name change (“TTS”), and re-run. Same error.
I’m probably doing this in a very crude fashion, but I added a menu item to my IT to show relevant(?) env vars and it dumps:
{"HISTORY ID': f597429621d6eb2b', 'REMOTE HOST': 'http://
localhost:8080", 'HOME': 'home/heiland/dev/galaxy-24.2.3/database/
jobs_directory/000/38/home', GALAXY_URL': 'http://localhost:8080',
'GALAXY MEMORY MB': ". 'GALAXY MEMORY MB PER SLOT: ".
'GALAXY SLOTS':'1, 'API KEY':
'7c53-delete-for-privacy', 'GALAXY WEB PORT':
"8080'}
yet when I attempt a put, still get Could not connect to a galaxy instance
Just to make sure your setup works, can you please try the Juypter IT?
So far this looks all good to me. I assume localhost:8080 brings you to the Galaxy? Or you running OSX maybe? In OSX Docker is running in a VM I think, maybe localhost is resolving to the VM and not your host OSX?
Yes, localhost:8080 takes me to my local Galaxy. I’m testing on Ubuntu. I start a Jupyter IT and run a Python notebook, create a hello.txt and try to “put” it:
Exception Traceback (most recent call last)
<ipython-input-7-f641031f7f42> in <module>
----> 1 put("hello.txt")
/opt/conda/lib/python3.8/site-packages/galaxy_ie_helpers/__init__.py in put(filenames, file_type, history_id)
111
112 history_id = history_id or os.environ['HISTORY_ID']
--> 113 gi = get_galaxy_connection(history_id=history_id)
114 for filename in filenames:
115 log.debug('Uploading gx=%s history=%s localpath=%s ft=%s', gi, history_id, filename, file_type)
/opt/conda/lib/python3.8/site-packages/galaxy_ie_helpers/__init__.py in get_galaxy_connection(history_id, obj)
98 ### Fail ###
99 msg = "Could not connect to a galaxy instance. Please contact your SysAdmin for help with this error"
--> 100 raise Exception(msg)
101
Do I need to provide more args, e.g., a history_id, to the put?.. apparently not.
I remember now that I’d tested earlier on usegalaxy.eu and it worked
This all looks fine. The put() is also fine. The history is obtained with
history_id or os.environ['HISTORY_ID']
The problem is your setup. Inside the Docker container localhost is not getting you to your Galaxy host, but inside the container.
Have a look at Hands-on: Galaxy Interactive Tools / Galaxy Interactive Tools / Development in Galaxy
Especially <param id="docker_run_extra_arguments">--add-host localhost:host-gateway</param> where we teach the Docker Container what localhost means.
You can also push your changes and we test on EU - as you like.
I definitely want to be able to test locally. I’ve got a lot of changes to make to our IT before we go “live” at a workshop in July and it seems logical to shorten the dev time by testing locally.
Thanks for pointing out the Galaxy host/Docker container issue. This makes me feel like maybe I’m really close to resolving this. When I visit the link you provided, it discusses job_conf.xml, yet in my server, I’m seeing job_conf.yml. Copy/pasting that here. It sure sounds like what’s being done in there now would resolve this. What am I missing?
(base) heiland@yoga:~/dev/galaxy-24.2.3/config$ cat job_conf.yml
runners:
local:
load: galaxy.jobs.runners.local:LocalJobRunner
workers: 4
execution:
default: docker_dispatch
environments:
local:
runner: local
docker_local:
runner: local
docker_enabled: true
docker_set_user:
# InteractiveTools do need real hostnames or URLs to work - simply specifying IPs will not work.
# If you develop interactive tools on your 'localhost' and don't have a proper domain name
# you need to tell all Docker containers a hostname where Galaxy is running.
# This can be done via the add-host parameter during the `docker run` command.
# 'localhost' here is an arbitrary hostname that matches the IP address of your
# Galaxy host. Make sure this hostname ('localhost') is also set in your galaxy.yml file, e.g.
# `galaxy_infrastructure_url: http://localhost:8080`.
docker_run_extra_arguments: --add-host localhost:host-gateway --platform linux/amd64
docker_dispatch:
runner: dynamic
type: docker_dispatch
docker_destination_id: docker_local
default_destination_id: local
And fwiw, I also have:
(base) heiland@yoga:~/dev/galaxy-24.2.3/config$ cat galaxy.yml
gravity:
app_server: gunicorn
gunicorn:
bind: 'localhost:8080'
gx_it_proxy:
enable: true
port: 4002
#handlers:
# handler:
# processes: 3
# pools:
# - job-handlers
# - workflow-schedulers
galaxy:
admin_users: randy.heiland@gmail.com
interactivetools_enable: true
interactivetools_map: database/interactivetools_map.sqlite
# outputs_to_working_directory will provide you with a better level of isolation. It is highly recommended to set
# this parameter with InteractiveTools.
outputs_to_working_directory: true
# `galaxy_infrastructure_url` needs to be reachable from IT containers.
# For local development you can map arbitrary hostnames. See `job_conf.yml.interactivetools`
# for an example.
# In the local development case you should use the `http` protocol (e.g http://localhost:8080) to access
# your Galaxy, so saving notebooks doesn't fail due to invalid certificates.
galaxy_infrastructure_url: http://localhost:8080
# Do not set the following 2 options if you are using an upstream proxy server like nginx
interactivetools_upstream_proxy: false
interactivetools_proxy_host: localhost:4002
job_config_file: config/job_conf.yml
tool_config_file: config/tool_conf.xml