Hi I’m trying to get data using the local galaxy while the error keeping occurs. Wondering what should I do with that??
Well, are you sure your local Galaxy was not restarted when you were running the job – as the error indicates? How does your local Galaxy run jobs?
So I’m using Mac terminnal so after launch the sh run.sh for galaxy then I went to the local host page http://localhost:8080/ to uploading the data via pasting the weblink. Meanwhile I had no operation on the terminal.
We are getting the same error with version 24.2 when uploading/importing a 43GB bam file that is in a user’s ftp directory which is on the same cluster as our Galaxy instance. The galaxy.log shows the galaxy.tool_util.provided_metadata line but instead of the expected next line “Collecting metrics for Job”, it restarts with the error “This job was killed when Galaxy was restarted. Please retry the job”. The galaxy.log does show all the restart initialization messages. I checked the .dat file and even though the job failed, the .dat file was uploaded successfully. The file upload works fine for smaller files.
Hi @cmdickens
Let’s see if any of our developers recognize the issue. Maybe there was a small change in the later releases. I’ve cross-posted over to our Admin Matrix channel but let’s keep the conversation here.
If you would like to post back the galaxy.log content from right before, during, and after that would be helpful (redact anything private and try to keep the format intact by quoting, or just post text with line breaks and we will reformat). 
If there is anything special about your server, you can include those details too.
For example, more details your proxy configuration would be helpful. Your database choice may matter too. I’m not sure if you mean the FTP directory is on the webserver’s head node, or the same node as the database, or just another cluster node in your local job execution environment (as opposed to a remote cluster?).
Then, was your server restarted during this time frame? Or is this error showing up spontaneously, but only for larger files, not smaller ones. And, only when both large and small are using the FTP directory transfer (during restart or not?) or do you mean regular file Upload (browsing, URL) works Ok. I’m just trying to pinpoint the series of events and observations for the troubleshooting.
Let’s start there, thanks!
XRef → Private Galaxy Servers
The installation, ftp, Galaxy and database, are all on the same server. The error shows up after the large 42GB file upload is complete. We are not restarting Galaxy manually. Small files are uploaded successfully by selecting “Choose local file”. I also tested “Paste/Fetch data” pasting in some text and the pasted text uploaded successfully. We haven’t had other users mention this error. This is the first time and it is for a 42GB file.
=== STARTED the upload job selecting a 42GB file in my ftp directory
galaxy.tools INFO 2025-10-15 10:13:07,358 [pN:main.1,p:2274142,tN:AnyIO worker thread] Validated and populated state for tool request (0.353 ms)
galaxy.tools.actions.upload DEBUG 2025-10-15 10:13:07,374 [pN:main.1,p:2274142,tN:AnyIO worker thread] Persisted uploads (0.008 ms)
galaxy.tools.actions.upload DEBUG 2025-10-15 10:13:07,414 [pN:main.1,p:2274142,tN:AnyIO worker thread] Created upload job (5.180 ms)
galaxy.tools.execute DEBUG 2025-10-15 10:13:07,414 [pN:main.1,p:2274142,tN:AnyIO worker thread] Tool DATA_FETCH created job None (40.007 ms)
galaxy.web_stack.handlers INFO 2025-10-15 10:13:07,535 [pN:main.1,p:2274142,tN:AnyIO worker thread] (Job[id=45043,tool_id=DATA_FETCH]) Handler ‘default’ assigned using ‘HANDLER_ASSIGNMENT_METHODS.DB_SKIP_LOCKED’ assignment method
galaxy.tools.execute DEBUG 2025-10-15 10:13:07,538 [pN:main.1,p:2274142,tN:AnyIO worker thread] Created 1 job(s) for tool DATA_FETCH request (177.610 ms)
galaxy.jobs.handler DEBUG 2025-10-15 10:13:07,852 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor_thread] Grabbed Job(s): 45043
galaxy.jobs.mapper DEBUG 2025-10-15 10:13:07,866 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor_thread] (45043) Mapped job to destination id: local
galaxy.jobs.handler DEBUG 2025-10-15 10:13:07,874 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor_thread] (45043) Dispatching to local runner
galaxy.jobs DEBUG 2025-10-15 10:13:08,015 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor_thread] (45043) Persisting job destination (destination id: local)
galaxy.jobs DEBUG 2025-10-15 10:13:08,196 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor_thread] (45043) Working directory for job is: /path/to/my_galaxy/jobs_directory/045/45043
galaxy.jobs.runners DEBUG 2025-10-15 10:13:08,209 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor_thread] Job [45043] queued (334.623 ms)
galaxy.jobs.handler INFO 2025-10-15 10:13:08,211 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor_thread] (45043) Job dispatched
galaxy.jobs DEBUG 2025-10-15 10:13:08,510 [pN:main.1,p:2274142,tN:LocalRunner.work_thread-3] Job wrapper for Job [45043] prepared (295.267 ms)
galaxy.jobs.command_factory INFO 2025-10-15 10:13:08,574 [pN:main.1,p:2274142,tN:LocalRunner.work_thread-3] Built script [/path/to/my_galaxy/jobs_directory/045/45043/tool_script.sh] for tool command [python ‘/path/to/my_galaxy/galaxy/lib/galaxy/tools/data_fetch.py’ --galaxy-root ‘/path/to/my_galaxy/galaxy’ --datatypes-registry ‘/path/to/my_galaxy/jobs_directory/045/45043/registry.xml’ --request-version ‘1’ --request ‘/path/to/my_galaxy/jobs_directory/045/45043/configs/tmphhlsa66y’]
galaxy.jobs.runners DEBUG 2025-10-15 10:13:08,774 [pN:main.1,p:2274142,tN:LocalRunner.work_thread-3] (45043) command is: mkdir -p working outputs configs
if [ -d _working ]; then
rm -rf working/ outputs/ configs/; cp -R _working working; cp -R _outputs outputs; cp -R _configs configs
else
cp -R working _working; cp -R outputs _outputs; cp -R configs _configs
fi
cd working; /bin/bash /path/to/my_galaxy/jobs_directory/045/45043/tool_script.sh > ‘../outputs/tool_stdout’ 2> ‘../outputs/tool_stderr’; return_code=$?; echo $return_code > /path/to/my_galaxy/jobs_directory/045/45043/galaxy_45043.ec; cd ‘/path/to/my_galaxy/jobs_directory/045/45043’;
[ “$GALAXY_VIRTUAL_ENV” = “None” ] && GALAXY_VIRTUAL_ENV=“$_GALAXY_VIRTUAL_ENV”; _galaxy_setup_environment True; python metadata/set.py; sh -c “exit $return_code”
galaxy.jobs.runners.local DEBUG 2025-10-15 10:13:08,791 [pN:main.1,p:2274142,tN:LocalRunner.work_thread-3] (45043) executing job script: /path/to/my_galaxy/jobs_directory/045/45043/galaxy_45043.sh=== JOB running in GUI in yellow for about 20 minutes while uploading the 42GB file
galaxy.jobs.runners.util.process_groups DEBUG 2025-10-15 10:34:42,303 [pN:main.1,p:2274142,tN:LocalRunner.work_thread-3] check_pg(): No process found in process group 2444433
galaxy.jobs.runners.local DEBUG 2025-10-15 10:34:42,328 [pN:main.1,p:2274142,tN:LocalRunner.work_thread-3] execution finished: /path/to/my_galaxy/jobs_directory/045/45043/galaxy_45043.sh
galaxy.jobs DEBUG 2025-10-15 10:34:42,857 [pN:main.1,p:2274142,tN:LocalRunner.work_thread-3] finish(): Moved /path/to/my_galaxy/jobs_directory/045/45043/outputs/dataset_577b1f54-bc07-4746-8591-880aa035d206.dat to /path/to/my_galaxy/objects/5/7/7/dataset_577b1f54-bc07-4746-8591-880aa035d206.dat
galaxy.tool_util.provided_metadata DEBUG 2025-10-15 10:34:42,864 [pN:main.1,p:2274142,tN:LocalRunner.work_thread-3] unnamed outputs [{‘__unnamed_outputs’: [{‘destination’: {‘type’: ‘hdas’}, ‘elements’: [{‘name’: ‘genome.GRCh38p13_primary_assembly.dedup.bam’, ‘dbkey’: ‘?’, ‘ext’: ‘bam’, ‘link_data_only’: False, ‘sources’: [{‘source_uri’: ‘gxftp://genome.GRCh38p13_primary_assembly.dedup.bam’}], ‘hashes’: , ‘info’: ‘uploaded bam file’, ‘state’: ‘ok’, ‘filename’: ‘/path/to/my_galaxy/jobs_directory/045/45043/working/gx_file_streamwuqpe415’, ‘object_id’: 105168}]}]}]=== JOB still running in GUI in yellow for several minutes then history refresh button turns red for a few minutes then galaxy restarts
=== AFTER the restart, the checksum for the dataset_577b1f54-bc07-4746-8591-880aa035d206.dat file is correct but the job is in redgalaxy.web_stack DEBUG 2025-10-15 10:41:02,689 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] server_name set to: main.1
galaxy.jobs.handler INFO 2025-10-15 10:41:02,693 [pN:main.1,p:2467629,tN:WorkflowRequestMonitor.monitor_thread] Handler job grabber initialized with ‘HANDLER_ASSIGNMENT_METHODS.DB_SKIP_LOCKED’ assignment method for handler ‘main.1’, tag(s): None, default
galaxy.jobs.manager DEBUG 2025-10-15 10:41:02,693 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Initializing job handler
galaxy.jobs INFO 2025-10-15 10:41:02,694 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Handler ‘main.1’ will load all configured runner plugins
galaxy.jobs.runners.state_handler_factory DEBUG 2025-10-15 10:41:02,748 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Loaded ‘failure’ state handler from module galaxy.jobs.runners.state_handlers.resubmit
galaxy.jobs DEBUG 2025-10-15 10:41:02,749 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Loaded job runner ‘galaxy.jobs.runners.local:LocalJobRunner’ as ‘local’
galaxy.jobs.runners DEBUG 2025-10-15 10:41:02,755 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Loading SlurmRunner with params: {‘drmaa_library_path’: ‘/path/to/slurm-drmaa/lib/libdrmaa.so’}
galaxy.jobs.runners.state_handler_factory DEBUG 2025-10-15 10:41:02,756 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Loaded ‘failure’ state handler from module galaxy.jobs.runners.state_handlers.resubmit
galaxy.jobs.runners.drmaa INFO 2025-10-15 10:41:02,756 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Overriding DRMAA_LIBRARY_PATH due to runner plugin parameter: /path/to/slurm-drmaa/lib/libdrmaa.so
pulsar.managers.util.drmaa DEBUG 2025-10-15 10:41:32,300 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Initializing DRMAA session from thread Thread-2 (run_postfork)
galaxy.jobs DEBUG 2025-10-15 10:41:32,307 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Loaded job runner ‘galaxy.jobs.runners.slurm:SlurmJobRunner’ as ‘slurm’
galaxy.jobs.handler DEBUG 2025-10-15 10:41:32,307 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Loaded job runners plugins: local:slurm
galaxy.jobs.runners DEBUG 2025-10-15 10:41:32,307 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Starting 4 LocalRunner workers
galaxy.jobs.runners DEBUG 2025-10-15 10:41:32,309 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Starting 10 SlurmRunner workers
galaxy.jobs.handler DEBUG 2025-10-15 10:41:32,312 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Handler queue starting for jobs assigned to handler: main.1
galaxy.jobs.handler DEBUG 2025-10-15 10:41:32,371 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] recovering job 44997 in slurm runner
galaxy.jobs.runners.drmaa DEBUG 2025-10-15 10:41:32,379 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] (44997/16762962) is still in running state, adding to the DRM queue
galaxy.jobs.handler DEBUG 2025-10-15 10:41:32,379 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] recovering job 45043 in local runner
galaxy.jobs.handler INFO 2025-10-15 10:42:04,468 [pN:main.1,p:2467629,tN:JobHandlerQueue.monitor_thread] Handler job grabber initialized with ‘HANDLER_ASSIGNMENT_METHODS.DB_SKIP_LOCKED’ assignment method for handler ‘main.1’, tag(s):
galaxy.jobs.handler INFO 2025-10-15 10:42:04,594 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] job handler queue started
galaxy.jobs.handler INFO 2025-10-15 10:42:04,599 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] job handler stop queue started
galaxy.queue_worker INFO 2025-10-15 10:42:04,672 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Binding and starting galaxy control worker for main.1
galaxy.queue_worker DEBUG 2025-10-15 10:42:04,680 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Executing toolbox reload on ‘main.1’
galaxy.model.database_heartbeat DEBUG 2025-10-15 10:42:04,715 [pN:main.1,p:2467629,tN:database_heartbeart_main.1.thread] main.1 is config watcher
galaxy.util WARNING 2025-10-15 10:42:04,806 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] directory not found: /path/to/my_galaxy/config/plugins/activities
galaxy.tool_util.toolbox.base INFO 2025-10-15 10:42:23,684 [pN:main.1,p:2467629,tN:Thread-2 (run_postfork)] Parsing the tool configuration /path/to/my_galaxy/galaxy/config/tool_conf.xml=== lots of startup messages
Serving on http://127.0.0.1:4001
Thanks for including the extra details!
To clarify about this a bit more, this is how you set up the job?
Data upload to your FTP service (successful), then when the user is moving the data into the history (via Upload: Choose from repository), the process keeps running (yellow “executing” dataset in the history) and that leads to the quoted situation above. Uploading via the file browsing method and URL are Ok (tested with smaller data). This is all happening in Galaxy release 24.2.
- admin configures this → Hands-on: Enable upload via FTP / Enable upload via FTP / Galaxy Server administration also here Galaxy FTP Uploads — Galaxy Project 24.2.5.dev0 documentation
- then the user does this → FAQ: Upload many files (>10) via FTP
My question would next be, are you able to load up any BAM files? Ever? Or, problems started just now? You can try by URL with a very small test file from our Github → https://github.com/galaxyproject/galaxy/raw/refs/heads/dev/test-data/1.bam (raw link that can be pasted directly in the Upload tool, using all defaults).
And, are you able to load any other types of data via the “FTP → Upload: Choose from repository” method? Ever? Or, problems started just now? You could grab another small file (not a BAM), maybe one that you loaded into a history already (download it, then FTP to your server, then move into a history again).
I’ve also asked again at our Admin chat to see if someone knows more about the older release’s database changes since 24.2.
The 1.bam uploaded successfully pasting the URL and I uploaded a smaller fasta file from my ftp directory. We have not had any issues with uploading from ftp directory with any type of file until the recent 42GB bam file. We have had our Galaxy successfully uploading via ftp for over 5 years and have not seen this issue until now. Maybe this is the first time a user has tried uploading a 42GB file. Our last Galaxy update was April of this year from version 21.01 to 24.2.
Thanks for the extra details @cmdickens
My guess is that this is overwhelming the node. It fails while trying to load the completed transfer into the history at the same time that the Galaxy UI is rebuilding.
The final warning is about the new Activity bar and updated Toolbox. Both had changes between 21.01 and 24.2. See → Releases — Galaxy Project 25.0.2.dev0 documentation and Tool Panel Administration — Galaxy Project 24.2.5.dev0 documentation.
Did you upgrade/migrate one release at a time (recommended), or all at once? I wonder if there are unresolved issues and the large data upload is revealing those.
The admins can comment (I pinged them again) but you can also double check your configuration. Are these set correctly?
24.2 version
21.01 version (no activities!) – again, search with the file name to see all references
Hi @cmdickens
More feedback – the job is probably not ending gracefully due to reasons related to this issue (closed but consider it moved, but pending the final issue home) → Replace json.load() with json_stream.load() by ksuderman · Pull Request #20802 · galaxyproject/galaxy · GitHub
If that is actually what is going on, offloading the job to another cluster node will protect your base Galaxy instance, plus avoid resource competition. And while scaling up memory itself may not be enough (depends on what the BAM looks like in terms of content), you might be able to see the actual memory utilization in your server logs. This will help to get better idea if memory is the root issue or not.
So, the good news is that this probably isn’t related to the upgrade through releases or your configuration files, so that guess was wrong, and the issue is more how certain large data scales versus resources. My original hunch all the services on the same node is more likely to be correct. The “overwhelming job” forced the server to reboot instead.
All of the extra metadata, data format screening, plus sorting on BAMs is really nice 99% of the time. We are actively thinking about how to improve the performance for that 1%, since it will soon be 5% and more. Streaming some of the functionality better is probably where this will go.
For practical use: if a server wasn’t able to load up and sort a BAM file, it might not be able to use that same BAM file for other job/tool purposes either, so reviewing the metrics and thinking about off-loading larger jobs is still something we would recommend, even if we can manage to make the loading part go easier.
More next week – I mostly wanted to get you something before the weekend!
If the header contains a huge number of sequence names this can cause OOM. This might be fixed in [25.0] Skip sam metadata if we have too many references by mvdbeek · Pull Request #20820 · galaxyproject/galaxy · GitHub .. if you like you can cherry-pick the two commits and try.
I upgraded in steps from 21.01 → 22.01 → 23.2 → 24.2.3
The RHEL 8.10 server has an uptime of 61 days. Is there a way to see the Galaxy uptime?
I watched the memory usage and it started at 75GB free and stayed there for most of the processing and at most dropped to 53GB at the end of the processing but there are other applications running on the same server.
There were two pysam index processes that ran during this time. Galaxy restarted a few minutes after the second pysam index process ended.
1630862 galaxy 20 0 166184 39980 13892 R 99.7 0.0 5:41.99 python -c import pysam; pysam.set_verbosity(0); pysam.index(‘-o’, ‘/path/to/my_galaxy/tmp/bam_index3lw5bbig’, '/path/to/galaxy/my_galaxy/jobs_directory/045/45058/working/gx_+
1648906 galaxy 20 0 168144 41268 13792 R 97.4 0.0 6:17.77 python -c import pysam; pysam.set_verbosity(0); pysam.index(‘-o’, ‘/path/to/galaxy/my_galaxy/tmp/bam_indexmo4f3pre’, '/path/to/galaxy/my_galaxy/jobs_directory/045/45058/working/gx_f+
the header of the sorted bam file has 200 lines
there are no file/disk quotas set for our ‘galaxy’ user that processes uploads