With the amazing help from @bjoern.gruening and others, our IT is now working at Galaxy . I thought it would be straightforward to also get it working on usegalaxy.org. However, I’ve been told that it cannot run there because its Dockerfile runs as root user, which is not allowed. I’m not a Dockerfile expert, so my first question: how can I make it run as rootless (which presumably would still allow it to run on the .eu instance). Second question: where, in the Galaxy training materials for ITs, is this root vs. rootless Docker policy and documentation discussed?
The Dockerfile (docker image build) does not run rootless, just the container itself (e.g. docker run --user ...). If your IT can run in that scenario then it should be fine on usegalaxy.org.
There’s no explicit policy, just operating from the principle of least privilege, so far we have been able to get all ITs to run as unprivileged users. This does not mean they cannot start as root - both the Jupyter and RStudio ITs start as root and drop privileges, which is fine.
I do not know the details of how Galaxy runs the container, if that’s what you are suggesting with the docker run --user ... comment. If someone can educate me on that, I could test on my local Galaxy server, presumably.
In our case it’s docker run --user $(id -u):$(id -g) ..., so the container runs with Galaxy’s uid and gid. Locally this will run the container’s processes as your local user’s uid/gid (if you are not running docker as root).
What I know is that my IT runs when testing locally (not as root, unless it’s something hidden to me) using Release v24.2.3 · galaxyproject/galaxy · GitHub . Are you saying that therefore it should run on the production server? Can you point me to where in the server code the docker run ... is being invoked? Not sure it will help, but it sounds like these details are relevant.
Docker is a daemon that runs as root, and by default, containers also run as root - they can change the user e.g. using USER in the Dockerfile, but by default they run as root. Passing --user to docker run causes the container to run as the specified user rather than as root. If you can test your container locally by passing the --user flag as described above and it still works, then it should work on usegalaxy.org. If you have a job destination/environment defined for your interactive tools, you can specify --user with docker_set_user in job_conf.yml like so:
execution:
environments:
docker:
runner: local
docker_enabled: true
docker_set_user: $UID
This is an example, presumably your existing execution environment will look different, but the key here is docker_set_user: $UID.
I am not sure that the code will help as it’s not very straightforward. The docker command line is assembled here but depends on all the arguments passed in. Here is an example of a command line generated for another interactive tool:
restart my local server (release 24.2.3), select my IT, it tells me I can “open” it, but then I just see Proxy target missing instead of my IT. I made no changes to my Dockerfile, i.e., I never attempted to set USER.
If I leave docker_set_user blank and do not append --user $UID:$GID to the extra args, the IT runs without error.
In my log, I see the following after the failed test (attempting to use $UID):
(base) heiland@yoga:~/dev/galaxy-24.2.3/database/gravity/log$ grep UID *.log
gunicorn.log:docker run -e “GALAXY_SLOTS=$GALAXY_SLOTS” -e “GALAXY_MEMORY_MB=$GALAXY_MEMORY_MB” -e “GALAXY_MEMORY_MB_PER_SLOT=$GALAXY_MEMORY_MB_PER_SLOT” -e “HOME=$HOME” -e “_GALAXY_JOB_HOME_DIR=$_GALAXY_JOB_HOME_DIR” -e “_GALAXY_JOB_TMP_DIR=$_GALAXY_JOB_TMP_DIR” -e “TMPDIR=$TMPDIR” -e “TMP=$TMP” -e “TEMP=$TEMP” -e “HISTORY_ID=$HISTORY_ID” -e “REMOTE_HOST=$REMOTE_HOST” -e “GALAXY_WEB_PORT=$GALAXY_WEB_PORT” -e “GALAXY_URL=$GALAXY_URL” -e “API_KEY=$API_KEY” -p 5800 --name ec940f1a40b94e2e9268304688546bcd -v “$_GALAXY_JOB_TMP_DIR:$_GALAXY_JOB_TMP_DIR:rw” -v “$TMPDIR:$TMPDIR:rw” -v “$TMP:$TMP:rw” -v “$TEMP:$TEMP:rw” -v /home/heiland/dev/galaxy-24.2.3:/home/heiland/dev/galaxy-24.2.3:ro -v /home/heiland/dev/galaxy-24.2.3/tools/interactive:/home/heiland/dev/galaxy-24.2.3/tools/interactive:ro -v /home/heiland/dev/galaxy-24.2.3/database/jobs_directory/000/66:/home/heiland/dev/galaxy-24.2.3/database/jobs_directory/000/66:ro -v /home/heiland/dev/galaxy-24.2.3/database/jobs_directory/000/66/outputs:/home/heiland/dev/galaxy-24.2.3/database/jobs_directory/000/66/outputs:rw -v “$_GALAXY_JOB_HOME_DIR:$_GALAXY_JOB_HOME_DIR:rw” -v /home/heiland/dev/galaxy-24.2.3/database/jobs_directory/000/66/working:/home/heiland/dev/galaxy-24.2.3/database/jobs_directory/000/66/working:rw -v /home/heiland/dev/galaxy-24.2.3/database/objects:/home/heiland/dev/galaxy-24.2.3/database/objects:ro -v /home/heiland/dev/galaxy-24.2.3/tool-data:/home/heiland/dev/galaxy-24.2.3/tool-data:ro --cpus ${GALAXY_SLOTS:-1} -w /home/heiland/dev/galaxy-24.2.3/database/jobs_directory/000/66/working --rm --add-host localhost:host-gateway --platform linux/amd64 --user $UID:$GID --user 1000:1000 quay.io/physicell/pcstudio:0.3 /bin/sh /home/heiland/dev/galaxy-24.2.3/database/jobs_directory/000/66/tool_script.sh > ‘../outputs/tool_stdout’ 2> ‘../outputs/tool_stderr’; return_code=$?; echo $return_code > /home/heiland/dev/galaxy-24.2.3/database/jobs_directory/000/66/galaxy_66.ec;
Is the job running or does it fail (red)? If it fails there should be additional details about the failure in the job stderr.
You have --user $UID:$GID in docker_run_extra_arguments, which is fine, but causes a second --user to be passed to docker run, so you want either that or docker_set_user: $UID but not both. I don’t know if this is responsible (locally for me it seems not to cause problems).
OK, ran again, this time just using the docker_run_extra_arguments . Job fails - History entries for the IT turn red. I click on “info” for one of them and see this in the “Tool Standard Error”:
cp: cannot stat '/opt/pcstudio/output/final.xml': No such file or directory
cp: cannot stat '/opt/pcstudio/output/PhysiCell_settings.xml': No such file or directory
cp: cannot stat '/opt/pcstudio/output/detailed_rules.html': No such file or directory
Repeating what I said before, I’m not attempting to set a USER in the Dockerfile.
and then my local server lets me run my IT successfully. So I feel like I’m closer to running on the main server, but still unsure what steps need to happen next.