Local instance of galaxy UnicodeEncodeError: 'latin-1'

Hello everyone,

I have installed a local instance of Galaxy (release_16.07) on my CentOS (7.3.1611) server and added a custom tool for taxonomic analysis using qiime2.

When I run the tool I get the error:

UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2018' in position 408: ordinal not in range(256)

but the job command line produced by the tool actually works when typed in the terminal.

I had noticed this type of behaviour in other different tools as well and noticed that when they produced an output like a warning or any additional output, Galaxy mistakes it as an error and produces the same error.

None of the tools have anything to do with MySQL, I think it’s some kind of internal error from Galaxy’s database or something.

So far I managed to get around that by redirecting stderr ( 2>/dev/null ) but in this case nothing works.

Could you help me out?

Thank you in advance,

The full error is the following:

Traceback (most recent call last):
  File "/home/user/galaxy/lib/galaxy/jobs/runners/local.py", line 128, in queue_job
    job_wrapper.finish( stdout, stderr, exit_code )
  File "/home/user/galaxy/lib/galaxy/jobs/__init__.py", line 1332, in finish
    self.sa_session.flush()
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 150, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush
    self._flush(objects)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush
    flush_context.execute()
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
    rec.execute(self)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
    uow
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
    mapper, table, update)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements
    execute(statement, multiparams)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1344, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 187, in execute
    query = query % tuple([db.literal(item) for item in args])
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 278, in literal
    return self.escape(o, self.encoders)
  File "/home/user/galaxy/.venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 208, in unicode_literal
    return db.literal(u.encode(unicode_literal.charset))
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2018' in position 408: ordinal not in range(256)

I recommend using 19.05 instead. You are missing 3 years of bugfixes and 16.07 is not really supported anymore.

At this point it is not really feasible to upgrade as I am currently using it for a research project… Is this a bug related to the specific version?

Hard to say, you can check the release notes for encoding fixes: https://docs.galaxyproject.org/en/master/

My guess is you are using some (date affecting) locale that is being stored in your mysql database but fails due to encoding.

That said upgrading Galaxy version should not affect reproducibility of your analyses.

Well that idea certainly backfired… I tried to upgrade my galaxy instance but initialization failed as it got stuck trying to upgrade the database (it stuck around version 143). I reverted the installation to each previous setup but the upgraded version of the database was incopatible and when I tried

sh manage_db.sh downgrade 132

I still got an error:

sh manage_db.sh downgrade 132
Activating virtualenv at .venv
Traceback (most recent call last):
  File "./scripts/manage_db.py", line 22, in <module>
    invoke_migrate_main()
  File "./scripts/manage_db.py", line 19, in invoke_migrate_main
    main( repository=repo, url=db_url )
  File "/home/omic-engine/galaxy/.venv/lib/python2.7/site-packages/migrate/versi                                                                                                             oning/shell.py", line 209, in main
    ret = command_func(**kwargs)
  File "/home/omic-engine/galaxy/.venv/lib/python2.7/site-packages/migrate/versi                                                                                                             oning/api.py", line 202, in downgrade
    return _migrate(url, repository, version, upgrade=False, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/home/omic-engine/galaxy/.venv/lib/python2.7/site-packages/migrate/versi                                                                                                             oning/util/__init__.py", line 160, in with_engine
    return f(*a, **kw)
  File "/home/omic-engine/galaxy/.venv/lib/python2.7/site-packages/migrate/versi                                                                                                             oning/api.py", line 345, in _migrate
    changeset = schema.changeset(version)
  File "/home/omic-engine/galaxy/.venv/lib/python2.7/site-packages/migrate/versi                                                                                                             oning/schema.py", line 82, in changeset
    changeset = self.repository.changeset(database, start_ver, version)
  File "/home/omic-engine/galaxy/.venv/lib/python2.7/site-packages/migrate/versi                                                                                                             oning/repository.py", line 225, in changeset
    changes = [self.version(v).script(database, op) for v in versions]
  File "/home/omic-engine/galaxy/.venv/lib/python2.7/site-packages/migrate/versi                                                                                                             oning/repository.py", line 189, in version
    return self.versions.version(*p, **k)
  File "/home/omic-engine/galaxy/.venv/lib/python2.7/site-packages/migrate/versi                                                                                                             oning/version.py", line 163, in version
    return self.versions[VerNum(vernum)]
KeyError: <VerNum(144)>

Any ideas?

If you did not make backups before, make backups now.

To be able to downgrade db you need to have the newer codebase (which has the migration script). I think you have the older code but are trying to downgrade a newer database.

What version were you trying to upgrade to?

I am not sure which database the 19.05 release was trying to upgrade to I just know it stopped at 143 and then I couldn’t get back at 132 which was needed for 16.07. Anyway I installed the 16.07 from scratch and moved all my tools there again. I will try to bypass my initial error some other way.

You can always see what version your database is at by checking the migrate_version table in the database.

for the unicode error this could be a lead: https://github.com/galaxyproject/galaxy/issues/7421

1 Like

So I figured out what the error was. I had been using my mysql as a default galaxy database for everything and its default collation is latin-1 which caused all kinds of problems. I created a fresh database with utf-8 collation to bypass that. @marten yeah you were right: always back up your database. It could have saved me some trouble with the migration failing. Thanks a lot.

Glad you solved it. Word of caution: mysql is not going to be a supported database in the newest release, only postgres.