Hi all!
I was wondering if there is a way to increase the waiting time for bioblend when performing a get-request (in this case, specifically when using the get_folders() function from the bioblend.galaxy.folders; FoldersClient).
I have a library that is very large and wrote a wrapper to automatically add new files from our filesystem, for which I get gateway time-outs (504, I will paste errorlog below) when performing a get_folders() on that library. I want to eliminate the chance that I get the gateway time-outs, because bioblend doesnât have enough time to finish the request.
Thanks!!!
library.get_folders(â2a56795cad3c7db3â)
/opt/fs_galaxy/galaxy_app/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host âgalaxy-prd.intra.localâ. Adding certificate verification is strongly advised. See: Advanced Usage - urllib3 2.1.0 documentation
InsecureRequestWarning,
Traceback (most recent call last):
File ââ, line 1, in
File â/opt/fs_galaxy/galaxy_app/venv/lib/python3.6/site-packages/bioblend/galaxy/libraries/init.pyâ, line 269, in get_folders
library_contents = self.show_library(library_id=library_id, contents=True)
File â/opt/fs_galaxy/galaxy_app/venv/lib/python3.6/site-packages/bioblend/galaxy/libraries/init.pyâ, line 324, in show_library
return self._get(id=library_id, contents=contents)
File â/opt/fs_galaxy/galaxy_app/venv/lib/python3.6/site-packages/bioblend/galaxy/client.pyâ, line 128, in _get
status_code=r.status_code)
bioblend.ConnectionError: GET: error 504: bâ\r\n504 Gateway Time-out\r\n\r\n
You can increase the number of attempts using gi.libraries.set_max_get_retries(5). (You might also want to adjust the time between attempts with gi.libraries.set_get_retry_delay().)
Hi Simon! Thanks for your fast reply!
True, I tried increasing both of them already (max_get_retries(20) and get_retry_delay(2000)), but I got the feeling that the retry delay only adds time in between tries. In my case, I have the feeling that the time of the try itself is too short (I have no idea for how long it tries to get a response or that maybe this is system basedâŚ?). For this I couldnât find any settings in bioblendâŚ
Hope there are more options or workarounds to adjust this.
Annabel
Hi Simon!
True, that test was a bit over enthusiastic . But in that case, I think after seeing the code from your link, I can conclude that the 504 is not necessarily caused by an too early retry, but a failed try? Because (in my eyes) with a delay that long, it gets all the time to finish, right?
Or⌠did it fail because it didnât have enough time to finish the get and then the new try takes too long because of my 2000 setting? But in that case, a second try will also not have enough time to finish (that relates to the setting I couldnât find, so this shouldnât change).
I hope I state my thoughts clear enough, I really appreciate all the help :). Itâs just really difficult to pinpoint why the get request on my properly functioning library is getting the 504.
Thanks a lot,
Annabel
Besides that keep in mind that for large amounts of data ftp is recommended.
If files are greater than 2GB in size, they will need to be uploaded via FTP. Importing files from the userâs FTP folder can be done via running the upload tool again: Usage documentation â BioBlend 1.2.0 documentation
Thanks a lot, I will try these suggestions and let you know!
I will also check the galaxy logs indeed to see if that gives some extra hints :).
Thanks again!