MultiQC issue creating flat plots

usegalaxy.org support
, ,
1

Hi all,

I was wondering if anyone has had issues using MultiQC– it works perfectly fine when I run it to create interactive plots but whenever I re-run to create flat plots, I get a disk quota exceeded error (even though I have about 153GB of data in my disk so I still have a little under 100GB of remaining quota). Any ideas on why the quota is going so high while running the tool to generate flat plots?

Standard error output:

/// MultiQC 🔍 v1.27

     update_config | Report title: RNA-Seq in vitro mapped reads
     version_check | MultiQC Version v1.32 now available!
       file_search | Search path: /corral4/main/jobs/072/965/72965894/working/multiqc_WDir

     featurecounts | Found 12 reports
     write_results | Rendering plots. Plots are requested to be rendered flat with `--flat` or `plots_force_flat: true` in config, and rendering might take a while. To disable, remove or override that flag

     write_results | Data        : report_data
Traceback (most recent call last):
  File "/usr/local/bin/multiqc", line 10, in <module>
    sys.exit(run_multiqc())
             ~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/multiqc/__main__.py", line 21, in run_multiqc
    multiqc.run_cli(prog_name="multiqc")
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.13/site-packages/multiqc/multiqc.py", line 498, in run_cli
    result = run(*analysis_dir, clean_up=clean_up, cfg=cfg, interactive=False)
  File "/usr/local/lib/python3.13/site-packages/multiqc/multiqc.py", line 565, in run
    write_results()
    ~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/multiqc/core/write_results.py", line 97, in write_results
    _write_html_report(paths.to_stdout, paths.report_path)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/multiqc/core/write_results.py", line 487, in _write_html_report
    shutil.copytree(template_mod.template_dir, tmp_dir.get_tmp_dir(), dirs_exist_ok=True)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/shutil.py", line 593, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
                     ignore=ignore, copy_function=copy_function,
                     ignore_dangling_symlinks=ignore_dangling_symlinks,
                     dirs_exist_ok=dirs_exist_ok)
  File "/usr/local/lib/python3.13/shutil.py", line 547, in _copytree
    raise Error(errors)
shutil.Error: [('/usr/local/lib/python3.13/site-packages/multiqc/templates/default/assets/js/packages/plotly-2.27.0.js', '/corral4/main/jobs/072/965/72965894/tmp/tmpuoqvxgui/assets/js/packages/plotly-2.27.0.js', '[Errno 122] Disk quota exceeded'), ('/usr/local/lib/python3.13/site-packages/multiqc/templates/default/assets/js/packages/plotly-2.27.0.min.js', '/corral4/main/jobs/072/965/72965894/tmp/tmpuoqvxgui/assets/js/packages/plotly-2.27.0.min.js', '[Errno 122] Disk quota exceeded')]
2

Welcome @grwtz

Memory error messages from tools are referring to the tool execution itself running out of resources. This is usually due to characteristics of the input dataset (problematic formatting) or parameter choices (mismatched to the data) but can also be due to legitimately large/complex data! The first case is much more common when working with the clusters at the UseGalaxy servers. Examples → exceeds-memory-error.

From the part of the your message posted, we can’t see which data reports are being graphed but this is where we would suggest reviewing first. If these were the same reports that graphed interactively, then the basic format is likely Ok and the content is what to focus on. Is there anything special about this data? If you try to graph just 2 of the 12 reports with flat plots, what happens?

If you need more help, we can try to help more! Off hand, I’m guessing unusually dense data for the graph type. Maybe we can assist in modifying the job to avoid the excessive memory use.

Then, if these jobs seem like they should graph Ok but for some reason are not, we can help to review the technical details and possibly make adjustments to how the job is executing technically on our side. Learning which server you are working at would be part of this.

For this closer review: please capture more screenshots (the job details view’s input/parameter table with at least one expanded peek view of an input report plus all MultiQC parameters) or share the exact example error job in a shared history link (we love these details!). How to → How to get faster help with your question

Let’s start there and please let us know what happens! :slight_smile:

3

Hi Jenna,

Thank you so much for your prompt and kind reply.

It is the same reports that graphed interactively– there’s nothing really special about this data. I was using multiQC to aggregate some RNA-Seq QC. So, maybe a density issue.

I’ve temporarily gotten around the memory use issue for MultiQC (I switched the storage location to the 30 day storage over the permanent storage and that seems to have solved it for the time being). I managed to get the flat plots to generate, thankfully.

But in case you’re curious, here’s the full history (error points were at 1595-1600): Galaxy

Sorry my organizational skills are slop. But I’ve been having issues with running tools that I typically have no issue with (like limma, as you’ll see) all day, getting an “unable to finish job” error. I’ll go back and check on my input files since the issues across multiple tools points to something fishy with my inputs.

Thank you again!

4

Ah, thanks for explaining, and I have more input after everyone was back in our (virtual) office today.

A few of the clusters nodes attached to the UseGalaxy.org server where themselves running out of disk space! That is where your error was coming from. For you and anyone else with the same error again from the last few days: a rerun will correct this across tools (no matter which account level storage is being used).

But if this is ever seen again inside of a job log message

‘[Errno 122] Disk quota exceeded’)

please let us know here or send in a bug report from the red error job so we can review and confirm what is exactly going on. We would always rather help, even if the issue is data, rather than miss something important.

So .. while all of that advice about data content didn’t actually apply this time it maybe helps someone later on. It can be pretty easy to run a massive analysis project through a web form. :slight_smile:

Glad to know you have this going now!