I’m following the Scaling and Load Balancing — Galaxy Project 23.3.dev0 documentation and cannot get gunicorn to bind to the socket file.
– gunicorn-galaxy.socket –
[Unit]
Description=gunicorn socket for galaxy
[Socket]
ListenStream=/run/gunicorn-galaxy.sock
[Install]
WantedBy=sockets.target
– galaxy.yml gunicorn section –
gunicorn:
enable: true
bind: 'unix:/run/gunicorn-galaxy.sock'
workers: 2
timeout: 300
extra_args: '--forwarded-allow-ips="*"'
preload: true
– gunicorn-galaxy.service –
[Unit]
Description=gunicorn daemon for galaxy
After=network.target
After=time-sync.target
[Service]
PermissionsStartOnly=true
Type=simple
User=galaxy
Group=www-data
Restart=on-abort
WorkingDirectory=/home/galaxy/galaxy
TimeoutStartSec=10
ExecStart=/home/galaxy/galaxy/.venv/bin/galaxy \
--state-dir /home/galaxy/galaxy/database/gravity
Environment=VIRTUAL_ENV=/home/galaxy/galaxy/.venv PATH=/home/galaxy/galaxy/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Install]
WantedBy=multi-user.target
This is the serviced error:● gunicorn-galaxy.service - gunicorn daemon for galaxyactive (running) since Mon 2022-10-17 21:20:23 UTC; 14min ago
Oct 17 21:34:07 ip-10-0-1-180 galaxy[5063]: [2022-10-17 21:34:07 +0000] [5310] [ERROR] Can’t connect to /run/gunicorn-galaxy.sock
The file exists and is rw to all:/run/gunicorn-galaxy.sock
jennaj
October 18, 2022, 4:38pm
2
Hi @isaphan
I’ve cross posted your question to our Admin chat to bring in some expert help. They may reply here or there, and feel free to join the chat: Gitter galaxyproject/admins - Gitter or at Matrix You're invited to talk on Matrix
Tutorials https://training.galaxyproject.org/training-material/topics/admin/tutorials/
Hello @jennaj I’m getting no response on the chat and the links that you provided redirected me to an error page: Oops!
Perhaps you are here due to a mis-typed URL
For future reference: admin suggested downgrading to galaxy version 22.01.
I am getting exactly the same error as with version 22.05.
Socket is up and listening:
$ sudo systemctl status gunicorn-galaxy.socket
● gunicorn-galaxy.socket - gunicorn socket for galaxy
Loaded: loaded (/etc/systemd/system/gunicorn-galaxy.socket; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-10-20 18:31:51 UTC; 19s ago
Triggers: ● gunicorn-galaxy.service
Listen: /run/gunicorn-galaxy.sock (Stream)
CGroup: /system.slice/gunicorn-galaxy.socket
Oct 20 18:31:51 ip-10-0-1-180 systemd[1]: Listening on gunicorn socket for galaxy.
Systemd service file (Group changed to ‘galaxy’ as recommended in release_22.01 docs):
[Unit]
Description=gunicorn daemon for galaxy
Requires=gunicorn-galaxy.socket
After=network.target
After=time-sync.target
[Service]
PermissionsStartOnly=true
Type=simple
User=galaxy
Group=galaxy
WorkingDirectory=/home/galaxy/galaxy
TimeoutStartSec=60
ExecStart=/home/galaxy/galaxy/.venv/bin/galaxy \
--state-dir /home/galaxy/galaxy/database/gravity
Environment=VIRTUAL_ENV=/home/galaxy/galaxy/.venv PATH=/home/galaxy/galaxy/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Install]
WantedBy=multi-user.target
Systemd fails to run gunicorn:
$ sudo systemctl status gunicorn-galaxy
● gunicorn-galaxy.service - gunicorn daemon for galaxy
Loaded: loaded (/etc/systemd/system/gunicorn-galaxy.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2022-10-20 18:31:58 UTC; 1min 16s ago
TriggeredBy: ● gunicorn-galaxy.socket
Main PID: 68088 (galaxy)
Tasks: 11 (limit: 18962)
Memory: 316.9M
CPU: 55.129s
CGroup: /system.slice/gunicorn-galaxy.service
├─68088 /home/galaxy/galaxy/.venv/bin/python3.7 /home/galaxy/galaxy/.venv/bin/galaxy --state-dir /home/galaxy/galaxy/database/gravity
├─68090 /home/galaxy/galaxy/.venv/bin/python3.7 /home/galaxy/galaxy/.venv/bin/supervisord -c /home/galaxy/galaxy/database/gravity/supervisor/supervisord.conf --nodaemon
├─68093 /home/galaxy/galaxy/.venv/bin/python3.7 /home/galaxy/galaxy/.venv/bin/celery --app galaxy.celery worker --concurrency 2 --loglevel DEBUG --pool threads --queues celery,galaxy.internal,galaxy.external
├─68094 /home/galaxy/galaxy/.venv/bin/python3.7 /home/galaxy/galaxy/.venv/bin/celery --app galaxy.celery beat --loglevel DEBUG --schedule /home/galaxy/galaxy/database/gravity/celery-beat-schedule
└─68096 /usr/bin/tail -f /home/galaxy/galaxy/database/gravity/log/gunicorn.log /home/galaxy/galaxy/database/gravity/log/celery.log /home/galaxy/galaxy/database/gravity/log/celery-beat.log
Oct 20 18:33:05 ip-10-0-1-180 galaxy[68096]: galaxy.webapps.galaxy.buildapp DEBUG 2022-10-20 18:33:05,661 [pN:main,p:68136,tN:MainThread] Prior to webapp return, Galaxy thread <_MainThread(MainThread, started 140058733199872)> is alive.
Oct 20 18:33:06 ip-10-0-1-180 galaxy[68096]: [2022-10-20 18:33:06 +0000] [68136] [INFO] Starting gunicorn 20.1.0
Oct 20 18:33:06 ip-10-0-1-180 galaxy[68096]: [2022-10-20 18:33:06 +0000] [68136] [ERROR] Retrying in 1 second.
Oct 20 18:33:07 ip-10-0-1-180 galaxy[68096]: [2022-10-20 18:33:07 +0000] [68136] [ERROR] Retrying in 1 second.
Oct 20 18:33:08 ip-10-0-1-180 galaxy[68096]: [2022-10-20 18:33:08 +0000] [68136] [ERROR] Retrying in 1 second.
Oct 20 18:33:09 ip-10-0-1-180 galaxy[68096]: [2022-10-20 18:33:09 +0000] [68136] [ERROR] Retrying in 1 second.
Oct 20 18:33:10 ip-10-0-1-180 galaxy[68096]: [2022-10-20 18:33:10 +0000] [68136] [ERROR] Retrying in 1 second.
Oct 20 18:33:12 ip-10-0-1-180 galaxy[68090]: 2022-10-20 18:33:12,000 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
Oct 20 18:33:12 ip-10-0-1-180 galaxy[68096]: [2022-10-20 18:33:12 +0000] [68136] [ERROR] Can't connect to /run/gunicorn-galaxy.sock
Oct 20 18:33:14 ip-10-0-1-180 galaxy[68090]: 2022-10-20 18:33:14,546 INFO exited: gunicorn (exit status 1; not expected)
Fixed: moved sock file from /run to <galaxy_project_location>/varmust be located inside the project root?
jennaj
October 20, 2022, 9:15pm
6
Glad you discovered the problem and that you posted that back here (thank you!).
Going further and adding a note to the docs sounds like a great idea. Please suggest the edit at Github in a PR.
The solution allows me to start galaxy… but it is unusable as I cannot upload any files. I am now fighting with tusd and not getting anywhere… I’m not a sysadmin, yet I must have installed galaxy a dozen times since 2007 and this is by far the very worst experience I’ve had with it. Very disappointing.
jennaj
October 26, 2022, 9:35pm
8
Hi @isaphan
I’m sorry that this has been difficult.
Do you think the instructions here are not working correctly? It was updated recently, about a month ago, as part of the overall update of the admin docs/tutorials that is mostly done by now. Some tutorials were updated as recently as today.
If you want to start up a new topic with the details, we can get some more community help again.
isaphan
October 26, 2022, 11:06pm
9
Hello @jennaj ,
Thank you for the instructions. If I had the help of a unix sysadmin, I would certainly ask them to look into ansible.
We finally got galaxy to run with a working file upload on our system:
tusd settings for external tusd server: we could not figure it out, due to the “Mixed Content” error that masked tusd configuration issues (see below). We are now using the tusd that comes already shipped with galaxy. I was very confused by the fact that the tusd config has to be disabled in galaxy.yml in order for tusd to work by default ?!
Scaling and Load Balancing — Galaxy Project 22.05.1 documentation : on our system, the group has to be set to www-data and not galaxy, even though the galaxy user is in the galaxy group.
the NGINX config statement in the docs: “proxy_set_header X-Forwarded-Proto $scheme;” causes uploads to fail on Chrome and Firefox with error: “Mixed Content: The page was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint”. Replacing $scheme with https fixed this.
I searched the project tickets in GitHub and since nobody reported the same issues, I assume the problems are due to our setup, where an external load balancer translates http requests to https -an institutional constraint over which we have no control.