I wrote a short blog post on my upgrade to Galaxy 21.05 and how I use a cluster_venv setup - i.e. a virtualenv built on one of my cluster worker nodes which is used for running Galaxy jobs and is independent of the main Galaxy virtualenv.
I don’t know if this is the best approach to use but I thought it might be useful for others to look at: Galaxy 21.05 upgrade and cluster_venv | low order magic