After git pull and run.sh the client build part errors on npm trying to install yarn …
So yarn is never installed into the .venv/lib/node_modules directory. ENOENT: no such file or directory, chmod ‘/home/galaxy/.venv/lib/node_modules/yarn/bin/yarn.js’
Is there a config missing here for npm??
Thanks, Gerhard
Hi Gerhard,
I’m not an expert of these issues, but I’d remove the Galaxy virtualenv directory .venv, then start Galaxy and if the error is still present, copy&paste the relevant output to a gist on https://gist.github.com/ and link that here.
[https://gist.github.com/gerhard1234/443938447dca0e66d8c76d60697bd5c7]
Conda will not uninstall certifi so the process halts. I am looking around for a way to do this manually - perhaps you can recommend a way. The process you suggested seems to run fine until this point.
Thanks for your suggestion!
Galaxy is using pip to install its own dependencies, not conda. Moreover all the dependencies should be installed in the virtual environment so if you delete the .venv folder all should be gone. Are you using a non-default virtual environment?
This galaxy installation uses supervisord to start. So there is a .venv stanza there and has been from the start - 18.05 was a clean install. I do not quite understand … there seems to be an either/or architecture with conda being incompatible with venv. Also the devs seem to be moving more towards conda.
Then searches indicate problems with individual packages like devutils when they do not uninstall cleanly. Also in the .venv world yarn does not play nicely with npm. It looks like my previous galaxy installation is not upgrading in either the .venv paradigm or the conda paradigm because of some package that cant make way to be reinstalled by the upgrade.
So to answer your question the virtual environment is the one that was created from the original clean install of 18.05.
Would it be OK to remove the package that is causing the installation to fail by file deletion? I dont want to just try this without some reflection.
Galaxy uses pip to install its own dependencies, it manages its own conda and it uses own conda to install dependencies for any Galaxy Tools you choose (but that only happens at tool installation time). You can override most of these settings and set up your own environments but that is generally not recommended unless you have a good reason.
If you remove the whole .venv folder and execute run.sh it should recreate the folder and install all Galaxy dependencies using pip.
In case you want to see what happens most of the dep-related startup stuff happens here: https://github.com/galaxyproject/galaxy/blob/dev/scripts/common_startup.sh
note that if you are using systemd configuration like the one here: https://docs.galaxyproject.org/en/master/admin/scaling.html#supervisord you are not running these startup scripts and have to maintain your dependencies yourself.
Removing .venv results in …
[galaxy@storage1 ~]$ sh run.sh
Found conda at: /home/.galaxy/database/dependencies/_conda/bin/conda
Found Conda, virtualenv will not be used.
To use a virtualenv instead, create one with a non-Conda Python 2.7 at .venv
Activating Conda environment: _galaxy_18.09
And it then tries to establish a non-venv/conda installation which fails because of the certifi clash.
The installation readme says to run the run.sh after the git pull to finalise the upgrade - so I dont see how starting the Galaxy installation from supervisord is involved in the maintenance of deps.
The .venv version of the installation will not install yarn due to a similar issue with packages not agreeing on how they should be uninstalled. Seems to only happen on systems where the package scertifi and already exist and requires removal and reinstallation, or yarn must be installed by npm into a pre-existing .venv …
Anyhow thanks for thinking about all this - hope we are not boring you 
The issue probably is that you are using the conda-built Python, which is non-standard to some extent. I think the best approach would be to use a different python 2.7.x to create the .venv but you can read details about this situation here: https://docs.galaxyproject.org/en/master/admin/framework_dependencies.html#conda
So we now have a working Galaxy 18.09. The workaround was to prevent the installation script from throwing its exception: by doing a clean install of 18.09 in a different location and dropping the yarn libs into the troubled installation’s .venv/lib/node_modules
The 18.09 upgrade proceeded smoothly from there on.
Thanks all for your thoughts and comments.
Gerhard
It sounds like you have figured it out. I’m a newbie and having trouble at the sh run.sh step which gives me the error kws-imac:~ kw$ sh run.sh > sh: run.sh: No such file or directory. I’m trying to figure out how you changed the install but its not clear to me since the installation was installed automatically when installing by $ git clone -b release_19.01 https://github.com/galaxyproject/galaxy.git . If you can point me in the right direction I would really appreciate it.
Please try the help in this reply to your original question: Error when trying to run Galaxy on MacOs
It looks like you are not running the sh run.sh command from inside of the galaxy directory, plus might need the python virtualenv created first.