# This job was killed when Galaxy was restarted. Please retry the job **URL:** \ **Category:** Uncategorized\ **Tags:** devops-administration, galaxy-local\ **Created:** [August 21, 2023, 4:57am UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650 "2023-08-21T04:57:40Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![H\_Z](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/h_z/32/4634_2.png) [@H\_Z](https://help.galaxyproject.org/u/H_Z)\ **Post date:** [August 21, 2023, 4:57am UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/1 "2023-08-21T04:57:40Z")
Hi I’m trying to get data using the local galaxy while the error keeping occurs. Wondering what should I do with that?? ![截屏2023-08-21 下午12.56.34](https://us1.discourse-cdn.com/flex020/uploads/galaxy/original/2X/d/dc343a8386269db97edcc14d895b6e28ba0d2bc5.png) ---
**Author:** ![marten](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/marten/32/6_2.png) [@marten](https://help.galaxyproject.org/u/marten)\ **Post date:** [August 21, 2023, 4:46pm UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/2 "2023-08-21T16:46:32Z")
> [@H\_Z](#): > > ing to get data using the local galaxy while the error keeping occurs. Wondering what should I do with that?? Well, are you sure your local Galaxy was not restarted when you were running the job – as the error indicates? How does your local Galaxy run jobs? ---
**Author:** ![H\_Z](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/h_z/32/4634_2.png) [@H\_Z](https://help.galaxyproject.org/u/H_Z)\ **Post date:** [August 22, 2023, 6:12am UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/4 "2023-08-22T06:12:04Z")
So I’m using Mac terminnal so after launch the _sh run.sh_ for galaxy then I went to the local host page [http://localhost:8080/](http://localhost:8080/) to uploading the data via pasting the weblink. Meanwhile I had no operation on the terminal. ---
**Author:** ![cmdickens](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/cmdickens/32/7154_2.png) [@cmdickens](https://help.galaxyproject.org/u/cmdickens)\ **Post date:** [October 14, 2025, 3:43pm UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/5 "2025-10-14T15:43:13Z")
We are getting the same error with version 24.2 when uploading/importing a 43GB bam file that is in a user’s ftp directory which is on the same cluster as our Galaxy instance. The galaxy.log shows the galaxy.tool\_util.provided\_metadata line but instead of the expected next line “Collecting metrics for Job”, it restarts with the error “This job was killed when Galaxy was restarted. Please retry the job”. The galaxy.log does show all the restart initialization messages. I checked the .dat file and even though the job failed, the .dat file was uploaded successfully. The file upload works fine for smaller files. ---
**Author:** ![jennaj](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/jennaj/32/27_2.png) [@jennaj](https://help.galaxyproject.org/u/jennaj)\ **Post date:** [October 14, 2025, 6:27pm UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/6 "2025-10-14T18:27:25Z")
Hi @cmdickens Let’s see if any of our developers recognize the issue. Maybe there was a small change in the later releases. I’ve [cross-posted over to our Admin Matrix](https://matrix.to/#/!fVQwotRzniGGJbeUUi:gitter.im/$XmOITyD3ay6p2vDPinSGnV1bNQg1IUFIGeKFntiXv0U?via=gitter.im&via=matrix.org&via=surf.nl) channel but let’s keep the conversation here. If you would like to post back the galaxy.log content from right before, during, and after that would be helpful (redact anything private and try to keep the format intact by quoting, or just post text with line breaks and we will reformat). 🙂 If there is anything special about your server, you can include those details too. For example, more details your proxy configuration would be helpful. Your database choice may matter too. I’m not sure if you mean the FTP directory is on the webserver’s head node, or the same node as the database, or just another cluster node in your local job execution environment (as opposed to a remote cluster?). > [@cmdickens](#): > > user’s ftp directory which is on the same cluster as our Galaxy instance Then, was your server restarted during this time frame? Or is this error showing up spontaneously, but only for larger files, not smaller ones. And, only when both large and small are using the FTP directory transfer (during restart or not?) or do you mean regular file Upload (browsing, URL) works Ok. I’m just trying to pinpoint the series of events and observations for the troubleshooting. > [@cmdickens](#): > > The file upload works fine for smaller files. * * * Let’s start there, thanks! XRef → [Private Galaxy Servers](https://help.galaxyproject.org/t/private-galaxy-servers/10861) ---
**Author:** ![cmdickens](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/cmdickens/32/7154_2.png) [@cmdickens](https://help.galaxyproject.org/u/cmdickens)\ **Post date:** [October 15, 2025, 4:19pm UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/7 "2025-10-15T16:19:02Z")
The installation, ftp, Galaxy and database, are all on the same server. The error shows up after the large 42GB file upload is complete. We are not restarting Galaxy manually. Small files are uploaded successfully by selecting “Choose local file”. I also tested “Paste/Fetch data” pasting in some text and the pasted text uploaded successfully. We haven’t had other users mention this error. This is the first time and it is for a 42GB file. > === STARTED the upload job selecting a 42GB file in my ftp directory > > galaxy.tools INFO 2025-10-15 10:13:07,358 [pN:main.1,p:2274142,tN:AnyIO worker thread] Validated and populated state for tool request (0.353 ms) > galaxy.tools.actions.upload DEBUG 2025-10-15 10:13:07,374 [pN:main.1,p:2274142,tN:AnyIO worker thread] Persisted uploads (0.008 ms) > galaxy.tools.actions.upload DEBUG 2025-10-15 10:13:07,414 [pN:main.1,p:2274142,tN:AnyIO worker thread] Created upload job (5.180 ms) > galaxy.tools.execute DEBUG 2025-10-15 10:13:07,414 [pN:main.1,p:2274142,tN:AnyIO worker thread] Tool **DATA\_FETCH** created job None (40.007 ms) > galaxy.web\_stack.handlers INFO 2025-10-15 10:13:07,535 [pN:main.1,p:2274142,tN:AnyIO worker thread] (Job[id=45043,tool\_id= **DATA\_FETCH**]) Handler ‘_default_’ assigned using ‘HANDLER\_ASSIGNMENT\_METHODS.DB\_SKIP\_LOCKED’ assignment method > galaxy.tools.execute DEBUG 2025-10-15 10:13:07,538 [pN:main.1,p:2274142,tN:AnyIO worker thread] Created 1 job(s) for tool **DATA\_FETCH** request (177.610 ms) > galaxy.jobs.handler DEBUG 2025-10-15 10:13:07,852 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor\_thread] Grabbed Job(s): 45043 > galaxy.jobs.mapper DEBUG 2025-10-15 10:13:07,866 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor\_thread] (45043) Mapped job to destination id: local > galaxy.jobs.handler DEBUG 2025-10-15 10:13:07,874 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor\_thread] (45043) Dispatching to local runner > galaxy.jobs DEBUG 2025-10-15 10:13:08,015 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor\_thread] (45043) Persisting job destination (destination id: local) > galaxy.jobs DEBUG 2025-10-15 10:13:08,196 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor\_thread] (45043) Working directory for job is: /path/to/my\_galaxy/jobs\_directory/045/45043 > galaxy.jobs.runners DEBUG 2025-10-15 10:13:08,209 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor\_thread] Job [45043] queued (334.623 ms) > galaxy.jobs.handler INFO 2025-10-15 10:13:08,211 [pN:main.1,p:2274142,tN:JobHandlerQueue.monitor\_thread] (45043) Job dispatched > galaxy.jobs DEBUG 2025-10-15 10:13:08,510 [pN:main.1,p:2274142,tN:LocalRunner.work\_thread-3] Job wrapper for Job [45043] prepared (295.267 ms) > galaxy.jobs.command\_factory INFO 2025-10-15 10:13:08,574 [pN:main.1,p:2274142,tN:LocalRunner.work\_thread-3] Built script [/path/to/my\_galaxy/jobs\_directory/045/45043/tool\_script.sh] for tool command [python ‘/path/to/my\_galaxy/galaxy/lib/galaxy/tools/data\_fetch.py’ --galaxy-root ‘/path/to/my\_galaxy/galaxy’ --datatypes-registry ‘/path/to/my\_galaxy/jobs\_directory/045/45043/registry.xml’ --request-version ‘1’ --request ‘/path/to/my\_galaxy/jobs\_directory/045/45043/configs/tmphhlsa66y’] > galaxy.jobs.runners DEBUG 2025-10-15 10:13:08,774 [pN:main.1,p:2274142,tN:LocalRunner.work\_thread-3] (45043) command is: mkdir -p working outputs configs > if [-d \_working]; then > rm -rf working/ outputs/ configs/; cp -R \_working working; cp -R \_outputs outputs; cp -R \_configs configs > else > cp -R working \_working; cp -R outputs \_outputs; cp -R configs \_configs > fi > cd working; /bin/bash /path/to/my\_galaxy/jobs\_directory/045/45043/tool\_script.sh \> ‘../outputs/tool\_stdout’ 2\> ‘../outputs/tool\_stderr’; return\_code=$?; echo $return\_code \> /path/to/my\_galaxy/jobs\_directory/045/45043/galaxy\_45043.ec; cd ‘/path/to/my\_galaxy/jobs\_directory/045/45043’; > [“$GALAXY\_VIRTUAL\_ENV” = “None”] && GALAXY\_VIRTUAL\_ENV=“$\_GALAXY\_VIRTUAL\_ENV”; \_galaxy\_setup\_environment True; python metadata/set.py; sh -c “exit $return\_code” > galaxy.jobs.runners.local DEBUG 2025-10-15 10:13:08,791 [pN:main.1,p:2274142,tN:LocalRunner.work\_thread-3] (45043) executing job script: /path/to/my\_galaxy/jobs\_directory/045/45043/galaxy\_45043.sh > > === JOB running in GUI in yellow for about 20 minutes while uploading the 42GB file > > galaxy.jobs.runners.util.process\_groups DEBUG 2025-10-15 10:34:42,303 [pN:main.1,p:2274142,tN:LocalRunner.work\_thread-3] check\_pg(): No process found in process group 2444433 > galaxy.jobs.runners.local DEBUG 2025-10-15 10:34:42,328 [pN:main.1,p:2274142,tN:LocalRunner.work\_thread-3] execution finished: /path/to/my\_galaxy/jobs\_directory/045/45043/galaxy\_45043.sh > galaxy.jobs DEBUG 2025-10-15 10:34:42,857 [pN:main.1,p:2274142,tN:LocalRunner.work\_thread-3] finish(): Moved /path/to/my\_galaxy/jobs\_directory/045/45043/outputs/dataset\_577b1f54-bc07-4746-8591-880aa035d206.dat to /path/to/my\_galaxy/objects/5/7/7/dataset\_577b1f54-bc07-4746-8591-880aa035d206.dat > galaxy.tool\_util.provided\_metadata DEBUG 2025-10-15 10:34:42,864 [pN:main.1,p:2274142,tN:LocalRunner.work\_thread-3] unnamed outputs [{‘\_\_unnamed\_outputs’: [{‘destination’: {‘type’: ‘hdas’}, ‘elements’: [{‘name’: ‘genome.GRCh38p13\_primary\_assembly.dedup.bam’, ‘dbkey’: ‘?’, ‘ext’: ‘bam’, ‘link\_data\_only’: False, ‘sources’: [{‘source\_uri’: ‘gxftp://genome.GRCh38p13\_primary\_assembly.dedup.bam’}], ‘hashes’: , ‘info’: ‘uploaded bam file’, ‘state’: ‘ok’, ‘filename’: ‘/path/to/my\_galaxy/jobs\_directory/045/45043/working/gx\_file\_streamwuqpe415’, ‘object\_id’: 105168}]}]}] > > === JOB still running in GUI in yellow for several minutes then history refresh button turns red for a few minutes then galaxy restarts > === AFTER the restart, the checksum for the dataset\_577b1f54-bc07-4746-8591-880aa035d206.dat file is correct but the job is in red > > galaxy.web\_stack DEBUG 2025-10-15 10:41:02,689 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] server\_name set to: main.1 > galaxy.jobs.handler INFO 2025-10-15 10:41:02,693 [pN:main.1,p:2467629,tN:WorkflowRequestMonitor.monitor\_thread] Handler job grabber initialized with ‘HANDLER\_ASSIGNMENT\_METHODS.DB\_SKIP\_LOCKED’ assignment method for handler ‘main.1’, tag(s): None, _default_ > galaxy.jobs.manager DEBUG 2025-10-15 10:41:02,693 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Initializing job handler > galaxy.jobs INFO 2025-10-15 10:41:02,694 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Handler ‘main.1’ will load all configured runner plugins > galaxy.jobs.runners.state\_handler\_factory DEBUG 2025-10-15 10:41:02,748 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Loaded ‘failure’ state handler from module galaxy.jobs.runners.state\_handlers.resubmit > galaxy.jobs DEBUG 2025-10-15 10:41:02,749 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Loaded job runner ‘galaxy.jobs.runners.local:LocalJobRunner’ as ‘local’ > galaxy.jobs.runners DEBUG 2025-10-15 10:41:02,755 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Loading SlurmRunner with params: {‘drmaa\_library\_path’: ‘/path/to/slurm-drmaa/lib/libdrmaa.so’} > galaxy.jobs.runners.state\_handler\_factory DEBUG 2025-10-15 10:41:02,756 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Loaded ‘failure’ state handler from module galaxy.jobs.runners.state\_handlers.resubmit > galaxy.jobs.runners.drmaa INFO 2025-10-15 10:41:02,756 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Overriding DRMAA\_LIBRARY\_PATH due to runner plugin parameter: /path/to/slurm-drmaa/lib/libdrmaa.so > pulsar.managers.util.drmaa DEBUG 2025-10-15 10:41:32,300 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Initializing DRMAA session from thread Thread-2 (run\_postfork) > galaxy.jobs DEBUG 2025-10-15 10:41:32,307 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Loaded job runner ‘galaxy.jobs.runners.slurm:SlurmJobRunner’ as ‘slurm’ > galaxy.jobs.handler DEBUG 2025-10-15 10:41:32,307 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Loaded job runners plugins: local:slurm > galaxy.jobs.runners DEBUG 2025-10-15 10:41:32,307 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Starting 4 LocalRunner workers > galaxy.jobs.runners DEBUG 2025-10-15 10:41:32,309 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Starting 10 SlurmRunner workers > galaxy.jobs.handler DEBUG 2025-10-15 10:41:32,312 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Handler queue starting for jobs assigned to handler: main.1 > galaxy.jobs.handler DEBUG 2025-10-15 10:41:32,371 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] recovering job 44997 in slurm runner > galaxy.jobs.runners.drmaa DEBUG 2025-10-15 10:41:32,379 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] (44997/16762962) is still in running state, adding to the DRM queue > galaxy.jobs.handler DEBUG 2025-10-15 10:41:32,379 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] recovering job 45043 in local runner > galaxy.jobs.handler INFO 2025-10-15 10:42:04,468 [pN:main.1,p:2467629,tN:JobHandlerQueue.monitor\_thread] Handler job grabber initialized with ‘HANDLER\_ASSIGNMENT\_METHODS.DB\_SKIP\_LOCKED’ assignment method for handler ‘main.1’, tag(s): > galaxy.jobs.handler INFO 2025-10-15 10:42:04,594 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] job handler queue started > galaxy.jobs.handler INFO 2025-10-15 10:42:04,599 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] job handler stop queue started > galaxy.queue\_worker INFO 2025-10-15 10:42:04,672 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Binding and starting galaxy control worker for main.1 > galaxy.queue\_worker DEBUG 2025-10-15 10:42:04,680 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Executing toolbox reload on ‘main.1’ > galaxy.model.database\_heartbeat DEBUG 2025-10-15 10:42:04,715 [pN:main.1,p:2467629,tN:database\_heartbeart\_main.1.thread] main.1 is config watcher > galaxy.util WARNING 2025-10-15 10:42:04,806 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] directory not found: /path/to/my\_galaxy/config/plugins/activities > galaxy.tool\_util.toolbox.base INFO 2025-10-15 10:42:23,684 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Parsing the tool configuration /path/to/my\_galaxy/galaxy/config/tool\_conf.xml > > === lots of startup messages > > Serving on [http://127.0.0.1:4001](http://127.0.0.1:4001) ---
**Author:** ![jennaj](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/jennaj/32/27_2.png) [@jennaj](https://help.galaxyproject.org/u/jennaj)\ **Post date:** [October 15, 2025, 5:47pm UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/8 "2025-10-15T17:47:49Z")
> [@cmdickens](#): > > === JOB still running in GUI in yellow for several minutes then history refresh button turns red for a few minutes then galaxy restarts Thanks for including the extra details! To clarify about this a bit more, this is how you set up the job? Data upload to your FTP service (successful), then when the user is moving the data into the history (via Upload: Choose from repository), the process keeps running (yellow “executing” dataset in the history) and that leads to the quoted situation above. Uploading via the file browsing method and URL are Ok (tested with smaller data). This is all happening in [Galaxy release 24.2](https://docs.galaxyproject.org/en/master/releases/24.2_announce_user.html). - admin configures this → [Hands-on: Enable upload via FTP / Enable upload via FTP / Galaxy Server administration](https://training.galaxyproject.org/training-material/topics/admin/tutorials/ftp/tutorial.html) also here [Galaxy FTP Uploads — Galaxy Project 24.2.5.dev0 documentation](https://docs.galaxyproject.org/en/release_24.2/admin/special_topics/ftp.html) - then the user does this → [FAQ: Upload many files (\>10) via FTP](https://training.galaxyproject.org/training-material/faqs/galaxy/datasets_upload_ftp.html) My question would next be, are you able to load up any BAM files? Ever? Or, problems started just now? You can try by URL with a very small test file from our Github → [https://github.com/galaxyproject/galaxy/raw/refs/heads/dev/test-data/1.bam](https://github.com/galaxyproject/galaxy/raw/refs/heads/dev/test-data/1.bam) (raw link that can be pasted directly in the Upload tool, using all defaults). And, are you able to load any other types of data via the “FTP → Upload: Choose from repository” method? Ever? Or, problems started just now? You could grab another small file (not a BAM), maybe one that you loaded into a history already (download it, then FTP to your server, then move into a history again). I’ve also asked again at our Admin chat to see if someone knows more about the older release’s database changes since 24.2. ---
**Author:** ![cmdickens](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/cmdickens/32/7154_2.png) [@cmdickens](https://help.galaxyproject.org/u/cmdickens)\ **Post date:** [October 15, 2025, 7:08pm UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/9 "2025-10-15T19:08:30Z")
The 1.bam uploaded successfully pasting the URL and I uploaded a smaller fasta file from my ftp directory. We have not had any issues with uploading from ftp directory with any type of file until the recent 42GB bam file. We have had our Galaxy successfully uploading via ftp for over 5 years and have not seen this issue until now. Maybe this is the first time a user has tried uploading a 42GB file. Our last Galaxy update was April of this year from version 21.01 to 24.2. ---
**Author:** ![jennaj](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/jennaj/32/27_2.png) [@jennaj](https://help.galaxyproject.org/u/jennaj)\ **Post date:** [October 17, 2025, 9:10pm UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/10 "2025-10-17T21:10:15Z")
Thanks for the extra details @cmdickens My guess is that this is overwhelming the node. It fails while trying to load the completed transfer into the history at the same time that the Galaxy UI is rebuilding. > [@cmdickens](#): > > galaxy.queue\_worker INFO 2025-10-15 10:42:04,672 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Binding and starting galaxy control worker for main.1 > galaxy.queue\_worker DEBUG 2025-10-15 10:42:04,680 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Executing toolbox reload on ‘main.1’ > galaxy.model.database\_heartbeat DEBUG 2025-10-15 10:42:04,715 [pN:main.1,p:2467629,tN:database\_heartbeart\_main.1.thread] main.1 is config watcher > galaxy.util **WARNING 2025-10-15 10:42:04,806 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] directory not found: /path/to/my\_galaxy/config/plugins/activities** > **galaxy.tool\_util.toolbox.base INFO 2025-10-15 10:42:23,684 [pN:main.1,p:2467629,tN:Thread-2 (run\_postfork)] Parsing the tool configuration /path/to/my\_galaxy/galaxy/config/tool\_conf.xml** The final warning is about the new **Activity** bar and updated **Toolbox**. Both had changes between 21.01 and 24.2. See → [Releases — Galaxy Project 25.0.2.dev0 documentation](https://docs.galaxyproject.org/en/master/releases/index.html) and [Tool Panel Administration — Galaxy Project 24.2.5.dev0 documentation](https://docs.galaxyproject.org/en/release_24.2/admin/tool_panel.html). Did you upgrade/migrate one release at a time (recommended), or all at once? I wonder if there are unresolved issues and the large data upload is revealing those. The admins can comment (I pinged them again) but you can also double check your configuration. Are these set correctly? **24.2 version** > > **21.01 version** (no activities!) – again, search with the file name to see all references > ---
**Author:** ![jennaj](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/jennaj/32/27_2.png) [@jennaj](https://help.galaxyproject.org/u/jennaj)\ **Post date:** [October 18, 2025, 2:06am UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/11 "2025-10-18T02:06:21Z")
Hi @cmdickens More feedback – the job is probably not ending gracefully due to reasons related to this issue (closed but consider it moved, but pending the final issue home) → [Replace json.load() with json\_stream.load() by ksuderman · Pull Request #20802 · galaxyproject/galaxy · GitHub](https://github.com/galaxyproject/galaxy/pull/20802) If that is actually what is going on, offloading the job to another cluster node will protect your base Galaxy instance, plus avoid resource competition. And while scaling up memory itself may not be enough (depends on what the BAM looks like in terms of content), you might be able to see the actual memory utilization in your server logs. This will help to get better idea if memory is the root issue or not. So, the good news is that this probably isn’t related to the upgrade through releases or your configuration files, so that guess was wrong, and the issue is more how certain large data scales versus resources. My original hunch all the services on the same node is more likely to be correct. The “overwhelming job” forced the server to reboot instead. All of the extra metadata, data format screening, plus sorting on BAMs is really nice 99% of the time. We are actively thinking about how to improve the performance for that 1%, since it will soon be 5% and more. Streaming some of the functionality better is probably where this will go. For practical use: if a server wasn’t able to load up and sort a BAM file, it might not be able to use that same BAM file for other job/tool purposes either, so reviewing the metrics and thinking about off-loading larger jobs is still something we would recommend, even if we can manage to make the loading part go easier. More next week – I mostly wanted to get you something before the weekend! 🙂 ---
**Author:** ![bernt-matthias](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/bernt-matthias/32/502_2.png) [@bernt-matthias](https://help.galaxyproject.org/u/bernt-matthias)\ **Post date:** [October 20, 2025, 7:48am UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/12 "2025-10-20T07:48:08Z")
If the header contains a huge number of sequence names this can cause OOM. This might be fixed in [[25.0] Skip sam metadata if we have too many references by mvdbeek · Pull Request #20820 · galaxyproject/galaxy · GitHub](https://github.com/galaxyproject/galaxy/pull/20820) .. if you like you can cherry-pick the two commits and try. ---
**Author:** ![cmdickens](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/cmdickens/32/7154_2.png) [@cmdickens](https://help.galaxyproject.org/u/cmdickens)\ **Post date:** [October 20, 2025, 6:00pm UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/13 "2025-10-20T18:00:39Z")
I upgraded in steps from 21.01 → 22.01 → 23.2 → 24.2.3 The RHEL 8.10 server has an uptime of 61 days. Is there a way to see the Galaxy uptime? I watched the memory usage and it started at 75GB free and stayed there for most of the processing and at most dropped to 53GB at the end of the processing but there are other applications running on the same server. There were two pysam index processes that ran during this time. Galaxy restarted a few minutes after the second pysam index process ended. > 1630862 galaxy 20 0 166184 39980 13892 R 99.7 0.0 5:41.99 python -c import pysam; pysam.set\_verbosity(0); pysam.index(‘-o’, ‘/path/to/my\_galaxy/tmp/bam\_index3lw5bbig’, '/path/to/galaxy/my\_galaxy/jobs\_directory/045/45058/working/gx\_+ > 1648906 galaxy 20 0 168144 41268 13792 R 97.4 0.0 6:17.77 python -c import pysam; pysam.set\_verbosity(0); pysam.index(‘-o’, ‘/path/to/galaxy/my\_galaxy/tmp/bam\_indexmo4f3pre’, '/path/to/galaxy/my\_galaxy/jobs\_directory/045/45058/working/gx\_f+ the header of the sorted bam file has 200 lines there are no file/disk quotas set for our ‘galaxy’ user that processes uploads ---
**Author:** ![jennaj](https://sea2.discourse-cdn.com/flex020/user_avatar/help.galaxyproject.org/jennaj/32/27_2.png) [@jennaj](https://help.galaxyproject.org/u/jennaj)\ **Post date:** [March 17, 2026, 4:25am UTC](https://help.galaxyproject.org/t/this-job-was-killed-when-galaxy-was-restarted-please-retry-the-job/10650/14 "2026-03-17T04:25:25Z")