How to allow 4GB memory when building with webpack

,
1

During the installation of a local Galaxy instance, I noticed that the step GXY_BUILD_SOURCEMAPS=1 NODE_ENV=production webpack consumes excessive memory. On servers with only 4GB of memory, this step frequently causes the system to freeze and receive a SIGKILL signal. Is there a method to configure the memory allocation for webpack?

I observed a PR on ansible-galaxy titled “Allow 4GB memory when building with webpack”. For installations performed using the sh run.sh method, how can we impose memory limits on webpack?

Thanks.

2

Hi @istevenshen

I’ve cross posted this over to the Admin chat to bring in some help. :slight_smile: You're invited to talk on Matrix

3

Before running the Galaxy installation, you can simply set the NODE_OPTIONS environment variable in your shell session:

export NODE_OPTIONS="--max_old_space_size=4096"
./run.sh

This will apply the memory limit to all Node.js processes launched during the execution of run.sh, including webpack. The 4GB limit should prevent the out-of-memory issues you’re experiencing on systems with limited RAM.

2 Likes
4

Thank you for your reply, Danno. I tried release_24.2 on a 2-core 4G server and found that the parameter --max_old_space_size=2048 did not seem to work. The actual memory used was larger than the value set by this parameter.

5

Might not be this, but in the online results it seemed to be “–max-old-space-size” (with “-” instead of “_” between the words)

6

Thank you @vlad.visan, I see the same.

If you try that version of the variable with the dashes instead of underscores, what happens @istevenshen ?

And @dannon can you clarify how to format this? Thanks!

7

I set export NODE_OPTIONS="--max-old-space-size=2048" in my ~/.bashrc, and after source ~/.bashrc, I found that the memory used to execute sh run.sh still exceeded 2GB.

$ echo $NODE_OPTIONS
--max-old-space-size=2048
$ sh run.sh
...
[09:14:19] Finished 'installPlugins' after 41 s
[09:14:19] Starting 'cleanPlugins'...
[09:14:19] Finished 'cleanPlugins' after 78 ms
[09:14:19] Starting 'stagePlugins'...
[09:14:20] Finished 'stagePlugins' after 1.22 s
[09:14:20] Finished 'default' after 42 s
$ GXY_BUILD_SOURCEMAPS=1 NODE_ENV=production webpack
Browserslist: caniuse-lite is outdated. Please run:
  npx update-browserslist-db@latest
  Why you should do it regularly: https://github.com/browserslist/update-db#readme
error Command failed with signal "SIGKILL".
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
ERROR: Galaxy client build failed. See ./client/README.md for more information, including how to get help.
8

I set it to export NODE_OPTIONS="-max-old-space-size=2048", but during the installation process, other issues appeared as shown below.

$ echo $NODE_OPTIONS
-max-old-space-size=2048
$ sh run.sh
...
Requirement already satisfied: zipp==3.20.2 in ./.venv/lib/python3.9/site-packages (from -r requirements.txt (line 217)) (3.20.2)
Requirement already satisfied: zipstream-new==1.1.8 in ./.venv/lib/python3.9/site-packages (from -r requirements.txt (line 218)) (1.1.8)
Requirement already satisfied: zstandard==0.23.0 in ./.venv/lib/python3.9/site-packages (from -r requirements.txt (line 219)) (0.23.0)
WARNING: You are using pip version 21.3.1; however, version 25.0.1 is available.
You should consider upgrading via the '/galaxy-dist/release_24.2/.venv/bin/python3 -m pip install --upgrade pip' command.
node: -max-old-space-size=2048 is not allowed in NODE_OPTIONS
Installing node into /galaxy-dist/release_24.2/.venv with nodeenv.
Removing old /galaxy-dist/release_24.2/.venv/lib/node_modules directory.
 * Install prebuilt node (18.12.1) ../usr/lib64/python3.9/tarfile.py:2268: RuntimeWarning: The default behavior of tarfile extraction has been changed to disallow common exploits (including CVE-2007-4559). By default, absolute/parent paths are disallowed and some mode bits are cleared. See https://access.redhat.com/articles/7004769 for more details.
  warnings.warn(
... done.
The Galaxy client has not yet been built and will be built now.
Installing yarn into /galaxy-dist/release_24.2/.venv with npm.
node: -max-old-space-size=2048 is not allowed in NODE_OPTIONS
9

Sorry for the continued trouble @istevenshen! One minor note – I see a single dash instead of a double dash for NODE_OPTIONS="--max-old-space-size=2048"

This is likely why that option isn’t recognized?