bioblend gateway time-out library.get_folders()

usegalaxy.org support
, ,
#1

Hi all!
I was wondering if there is a way to increase the waiting time for bioblend when performing a get-request (in this case, specifically when using the get_folders() function from the bioblend.galaxy.folders; FoldersClient).
I have a library that is very large and wrote a wrapper to automatically add new files from our filesystem, for which I get gateway time-outs (504, I will paste errorlog below) when performing a get_folders() on that library. I want to eliminate the chance that I get the gateway time-outs, because bioblend doesn’t have enough time to finish the request.
Thanks!!!

library.get_folders(“2a56795cad3c7db3”)
/opt/fs_galaxy/galaxy_app/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host ‘galaxy-prd.intra.local’. Adding certificate verification is strongly advised. See: Advanced Usage - urllib3 2.0.0.dev0 documentation
InsecureRequestWarning,
Traceback (most recent call last):
File “”, line 1, in
File “/opt/fs_galaxy/galaxy_app/venv/lib/python3.6/site-packages/bioblend/galaxy/libraries/init.py”, line 269, in get_folders
library_contents = self.show_library(library_id=library_id, contents=True)
File “/opt/fs_galaxy/galaxy_app/venv/lib/python3.6/site-packages/bioblend/galaxy/libraries/init.py”, line 324, in show_library
return self._get(id=library_id, contents=contents)
File “/opt/fs_galaxy/galaxy_app/venv/lib/python3.6/site-packages/bioblend/galaxy/client.py”, line 128, in _get
status_code=r.status_code)
bioblend.ConnectionError: GET: error 504: b’\r\n504 Gateway Time-out\r\n\r\n

504 Gateway Time-out

\r\n
nginx/1.14.1\r\n\r\n\r\n’, 0 attempts left:

#2

You can increase the number of attempts using gi.libraries.set_max_get_retries(5). (You might also want to adjust the time between attempts with gi.libraries.set_get_retry_delay().)

Hope this helps,

Simon

#3

Hi Simon! Thanks for your fast reply!
True, I tried increasing both of them already (max_get_retries(20) and get_retry_delay(2000)), but I got the feeling that the retry delay only adds time in between tries. In my case, I have the feeling that the time of the try itself is too short (I have no idea for how long it tries to get a response or that maybe this is system based…?). For this I couldn’t find any settings in bioblend…
Hope there are more options or workarounds to adjust this.
Annabel

#4

You can see the code here: bioblend/client.py at main ¡ galaxyproject/bioblend ¡ GitHub It just tries the GET request repeatedly (with the delay between attempts) until it runs out of attempts.

A delay of 2000 is really long, this is more than half an hour… maybe try reducing it?

#5

Hi Simon!
True, that test was a bit over enthusiastic :smiley: . But in that case, I think after seeing the code from your link, I can conclude that the 504 is not necessarily caused by an too early retry, but a failed try? Because (in my eyes) with a delay that long, it gets all the time to finish, right?
Or… did it fail because it didn’t have enough time to finish the get and then the new try takes too long because of my 2000 setting? But in that case, a second try will also not have enough time to finish (that relates to the setting I couldn’t find, so this shouldn’t change).
I hope I state my thoughts clear enough, I really appreciate all the help :). It’s just really difficult to pinpoint why the get request on my properly functioning library is getting the 504.
Thanks a lot,
Annabel

#6

On second thoughts I don’t think this is a bioblend error. Do you have access to the logs of the Galaxy server?

#7

You could add something like this to the config of nginx:

  proxy_connect_timeout       800;
  proxy_send_timeout          800;
  proxy_read_timeout          800;
  send_timeout                800;

Besides that keep in mind that for large amounts of data ftp is recommended.

If files are greater than 2GB in size, they will need to be uploaded via FTP. Importing files from the user’s FTP folder can be done via running the upload tool again:
Usage documentation — BioBlend 0.15.0 documentation

Hope this leads you to the right direction.

1 Like
#8

Thanks a lot, I will try these suggestions and let you know!
I will also check the galaxy logs indeed to see if that gives some extra hints :).
Thanks again!