Expected speed / responsiveness with cvmfs mount

Hi!

I have mounted the singularity containers from the galaxyproject with cvmfs and am wondering what the expected speed and responsiveness should be. Currently listing the files takes a while the first time, but I think that is expected. But using a container is also a bit slow. I am in the EU and I thought it would use the Freiburg mirror (cvmfs1-ufr0), also because I enabled the GEOAPI function. But looking at the network traffic it seems to connect the penn state / psu server, which is off course far away. Is this expected? Or should I change something in the setup.

Part of the config

CVMFS_SERVER_URL='http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org;http://cvmfs1-iu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org;http://cvmfs1-tacc0.galaxyproject.org/cvmfs/singularity.galaxyproject.org;http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/singularity.galaxyproject.org;http://cvmfs1-mel0.gvl.org.au/cvmfs/singularity.galaxyproject.org'    # from /cvmfs/cvmfs-config.galaxyproject.org/etc/cvmfs/domain.d/galaxyproject.org.conf
CVMFS_SHARED_CACHE=yes    # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no    # from /etc/cvmfs/default.conf
CVMFS_SYSLOG_FACILITY=
CVMFS_SYSLOG_LEVEL=
CVMFS_SYSTEMD_NOKILL=
CVMFS_TIMEOUT=5    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10    # from /etc/cvmfs/default.conf
CVMFS_TRACEFILE=
CVMFS_TRUSTED_CERTS=
CVMFS_USER=cvmfs    # from /etc/cvmfs/default.conf
CVMFS_USE_CDN=yes    # from /etc/cvmfs/default.local
CVMFS_USE_GEOAPI=yes    # from /etc/cvmfs/default.local

Output from iftop and nethogs:

# iftop
server     => cvmfs1-psu0.galaxyproject.org             23.1Kb  32.0Kb  39.1Kb
      <=                                                      1.74Mb  2.56Mb  3.31Mb
# nethogs
  13093 cvmfs    /usr/bin/cvmfs2      ens2f0      5.861     456.206 KB/sec

Thanks for sharing experience and insights!

Welcome, @mattias

Good questions. I’ve cross-posted over to the Admin chat to hopefully get more eyes on this. They may reply here or there, and feel free to join! You're invited to talk on Matrix

Thanks for cross-posting. Re-enabling a matrix account so I can follow over there as well :stuck_out_tongue:

1 Like

@mattias Did you try swapping the order of servers in CVMFS_SERVER_URL and check whether that has any effect? Something I’ve encountered in the past is that if, say the Freiburg server is down, the cvmfs client automatically fails over to the next closest one. It could be the case that a restart of the client could kick off a GEOAPI lookup again.

You might also be able to use this: ansible-cvmfs/main.yml at main · galaxyproject/ansible-cvmfs · GitHub, which is the playbook use by the usegalaxy.* federation, to crosscheck your configuration.

Thanks for your answer. I did try switching some urls, but it did not have effect. I did run wipecache and now I notice the geo api is active in the syslog:

Jun  7 15:11:20 nioo0009 cvmfs2: (cvmfs-config.galaxyproject.org) switching proxy from (none) to DIRECT (set proxies)
Jun  7 15:11:20 nioo0009 cvmfs2: (cvmfs-config.galaxyproject.org) geographic order of servers retrieved from cvmfs1-iu0.galaxyproject.org
Jun  7 15:11:20 nioo0009 cvmfs2: (cvmfs-config.galaxyproject.org) switching proxy from (none) to DIRECT (cloned)
Jun  7 15:11:20 nioo0009 cvmfs2: (cvmfs-config.galaxyproject.org) switching host from http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/cvmfs-config.galaxyproject.org to http://cvmfs1-psu0.galaxyproject.org/cvmfs/cvmfs-config.galaxyproject.org (host returned HTTP error)
Jun  7 15:11:21 nioo0009 cvmfs2: (cvmfs-config.galaxyproject.org) CernVM-FS: linking /cvmfs/cvmfs-config.galaxyproject.org to repository cvmfs-config.galaxyproject.org
Jun  7 15:11:40 nioo0009 mount.cvmfs: configuration repository directory does not exist: /cvmfs/cvmfs-config.galaxyproject.org/etc/cvmfs/config.d
Jun  7 15:11:40 nioo0009 cvmfs2: configuration repository directory does not exist: /cvmfs/cvmfs-config.galaxyproject.org/etc/cvmfs/config.d
Jun  7 15:11:40 nioo0009 cvmfs2: configuration repository directory does not exist: /cvmfs/cvmfs-config.galaxyproject.org/etc/cvmfs/config.d
Jun  7 15:11:41 nioo0009 cvmfs2: (singularity.galaxyproject.org) switching proxy from (none) to DIRECT (set proxies)
Jun  7 15:11:42 nioo0009 cvmfs2: (singularity.galaxyproject.org) geographic order of servers retrieved from cvmfs1-mel0.gvl.org.au
Jun  7 15:11:42 nioo0009 cvmfs2: (singularity.galaxyproject.org) switching proxy from (none) to DIRECT (cloned)
Jun  7 15:11:42 nioo0009 cvmfs2: (singularity.galaxyproject.org) whitelist lifetime verification failed, expired
Jun  7 15:11:42 nioo0009 cvmfs2: (singularity.galaxyproject.org) whitelist verification failed (5): expired whitelist
Jun  7 15:11:42 nioo0009 cvmfs2: (singularity.galaxyproject.org) failed to fetch manifest (8 - bad whitelist), trying another stratum 1
Jun  7 15:11:42 nioo0009 cvmfs2: (singularity.galaxyproject.org) switching host from http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/singularity.galaxyproject.org to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org (manually triggered)

What I notice is 1) a config folder on the mount does not exist 2) the fr0 host is tried but switched to psu0 because of a host returned HTTP error.

It is not the case that the freiburg stratum1 is down at the moment?

I have this setup:

/etc/cvmfs/default.local

CVMFS_HTTP_PROXY=DIRECT
CVMFS_CLIENT_PROFILE=single
CVMFS_QUOTA_LIMIT=5000
CVMFS_USE_GEOAPI=yes

/etc/cvmfs/config.d/cvmfs-config.galaxyproject.org.conf

CVMFS_USE_GEOAPI=yes
CVMFS_SERVER_URL="http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/@fqrn@;http://cvmfs1-psu0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-iu0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-tacc0.galaxyproject.org/cvmfs/@fqrn@"
CVMFS_KEYS_DIR="/etc/cvmfs/keys/galaxyproject.org/

I did already look at the ansible configs but went for directly creating the config files. Maybe I have something not correctly set, but I don’t see it myself yet.

@bjoern.gruening Is singularity hosted on the Freiburg cvmfs servers? I see a 404 when I visit the above link in the browser, but a 403 when I try the same with psu and mel servers.

I am using the cvmfs-config to get the keys etc, but it seems with my config it tries to get this form the freiburg server but gets a 404 and therefore goes to the pennstate one. But the singularity folder is there at freiburg, but since the host switched maybe it is never tried? Should I manually configure things witthout cvmfs-config?

Here is a list of status codes I get:

link | status |
---- |----
http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/cmfs-config.galaxyproject.org/ | 404
http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/singularity.galaxyproject.org/ | 403
http://cvmfs1-psu0.galaxyproject.org/cvmfs/cvmfs-config.galaxyproject.org/ | 403
http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/ | 403

I think it’s worth a try. The last time I tried, I didn’t have much luck with cvmfs-config. @hexylena may be able to point to the issue.

I removed the cvmfs-config repo from the setup, but I still have issues accessing the freiburg mirror. If I remove the psu mirror it first tries the freiburg one, fails, and then goes to the Indiana one. That one is already much faster in response time, but still I think the freiburg one should be better since it is much closer. Here the logs:

Jun 23 14:22:55 server cvmfs2: (singularity.galaxyproject.org) switching proxy from (none) to DIRECT (set proxies)
Jun 23 14:22:55 server cvmfs2: (singularity.galaxyproject.org) switching proxy from (none) to DIRECT (cloned)
Jun 23 14:22:55 server cvmfs2: (singularity.galaxyproject.org) whitelist lifetime verification failed, expired
Jun 23 14:22:55 server cvmfs2: (singularity.galaxyproject.org) whitelist verification failed (5): expired whitelist
Jun 23 14:22:55 server cvmfs2: (singularity.galaxyproject.org) failed to fetch manifest (8 - bad whitelist), trying another stratum 1
Jun 23 14:22:55 server cvmfs2: (singularity.galaxyproject.org) switching host from http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/singularity.galaxyproject.org to http://cvmfs1-iu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org (manually triggered)
Jun 23 14:23:12 server cvmfs2: (singularity.galaxyproject.org) CernVM-FS: linking /cvmfs/singularity.galaxyproject.org to repository singularity.galaxyproject.org

Error seems to be whitelist lifetime verification and according to this thread ([CVM-1828] .ch) whitelist verification failed (6) - SFTJIRA) is because of an expired key. Is there a different key for the freiburg one? Seems not if I look at the ansible configs. Or is there something not up to date at the freiburg mirror, since the other mirrors work? @bjoern.gruening, can you maybe help here?

We will have a look at this next week. The monitoring looks like it should work Grafana

Thanks for the dashboard link. For me this confirms that:

  • There is no cvmfs-config mirror at fr0
  • The singularity mirror is up at fr0, or at least pings :slight_smile:

Maybe because there is no cvmfs-config I misconfigured something? But it seems the other mirrors work with the same config. @bjoern.gruening, can you confirm that the public key for fr0 is the same as for the others, so like this:

# /etc/cvmfs/keys/galaxyproject.org/singularity.galaxyproject.org.pub
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuJZTWTY3/dBfspFKifv8
TWuuT2Zzoo1cAskKpKu5gsUAyDFbZfYBEy91qbLPC3TuUm2zdPNsjCQbbq1Liufk
uNPZJ8Ubn5PR6kndwrdD13NVHZpXVml1+ooTSF5CL3x/KUkYiyRz94sAr9trVoSx
THW2buV7ADUYivX7ofCvBu5T6YngbPZNIxDB4mh7cEal/UDtxV683A/5RL4wIYvt
S5SVemmu6Yb8GkGwLGmMVLYXutuaHdMFyKzWm+qFlG5JRz4okUWERvtJ2QAJPOzL
mAG1ceyBFowj/r3iJTa+Jcif2uAmZxg+cHkZG5KzATykF82UH1ojUzREMMDcPJi2
dQIDAQAB
-----END PUBLIC KEY-----

This is also the same public key as configured for the ansible role: ansible-cvmfs/defaults/main.yml at main · galaxyproject/ansible-cvmfs · GitHub

Hi!

The public key that you pasted is correct. There was a problem with our server. At the moment, I expect data.galaxyproject.org, main.galaxyproject.org, sandbox.galaxyproject.org, singularity.galaxyproject.org, refgenomes-databio.galaxyproject.org and usegalaxy.galaxyproject.org to work on the Freiburg server. Can you try again?

1 Like

Hi José,

I can confirm it works. I configured it with only the freiburg mirror to be sure. And the response time for me is now much faster! Thanks a lot! I am going to explore this setup to see if it works for us. But loading tools within a few seconds is very exciting!

1 Like