MultiQC: summarize analysis results for multiple tools and samples in a single report

MultiQC v1.24

Mostly a maintenance release, containing several bug fixes, performance improvements, plus 6 new modules, along with improvements of the existing modules.

The most significant performance boost got the Kraken and Mosdepth modules, that now don't take way more memory and CPU than any other typical module:

| Tool | Data Set | Memory - Before | CPU - Before | Memory - After | CPU - After | | -------- | -------------- | --------------- | ------------ | -------------- | ----------- | | mosdepth | 1 set of files | 196 Mb | 3.04s | 129 Mb | 2.27s | | | 10 | 464 Mb | 8.48s | 131 Mb | 6.11s | | | 100 | 3,719 Mb | 63.19s | 172 Mb | 43.12s | | kraken | 1 set of files | 155 Mb | 2.07s | 132 Mb | 2.20s | | | 10 | 606 Mb | 8.39s | 180 Mb | 3.47s | | | 100 | 4,970 Mb | 71.89s | 809 Mb | 14.53s |

Large plots that may hang browser are now not loaded by default, and the user can click a button to load, so the heavy plots don't slow down the initial report rendering. This is controlled by the config.plots_defer_loading_numseries: 100 option.

Updates

• Search patterns: allow multiple values for contents (#2696)
• Custom content:
  • Allow dict input to heatmap (#2761)
  • Allow multiple custom content to general stats table (#2727)
• Plots:
  • Defer render of plots if number of samples > config.plots_defer_loading_numseries (#2759, #2777, #2773, #2774)
  • Line plot: show markers when num of data points < config.lineplot_number_of_points_to_hide_markers (=50) (#2760). As a nice consequence, trivial lines of a single data point become visible.
  • Line plot: smooth by default to 500 points on the X axis to avoid inflating the report file size (#2776)
  • Allow to configure the scale of the exported plot fonts through the config option config.plots_export_font_scale (#2758)
  • Improve the performance of loading large tables in browser (#2737)
  • Fix the toolbox highlight of the line plots (#2724)
• The function that returns built plots in an interactive session now uses the module anchor (or lowercase module name) to key the results (#2741)
• More helpful config validation error: print the parent model name, if applicable (#2709)

New modules

• VG (#2690), a toolkit to manipulate graphical genomes. The module parses vg-stats reports that summarize alignment stats from a GAM file.
• ngs-bits (#2231). A tool that calculating statistics from FASTQ, BAM, and VCF files. The module parses XML output generated for two tools in the ngs-bits collection:
  • ReadQC for FastQ file stats
  • MappingQC for BAM file stats
• Pairtools (#1148). A toolkit for Chromatin Conformation Capture experiments. Handles short-reads paired reference alignments, extracts 3C-specific information, and perform common tasks such as sorting, filtering, and deduplication. The module parses summary statistics generated by pairtools's dedup and stats tools.
• nanoq (#2723). A tool that reports read quality and length from nanopore sequencing data.
• Ganon (#1935). A tool for metagenomics classification: quickly assigns sequence fragments to their closest reference among thousands of references via Interleaved Bloom Filters of k-mer/minimizers

Fixes

• Fix --pdf option to generate multiqc_report.pdf (#2733)
• Fix saving table plots to file (#2735)
• Fix adding software versions when config.run_modules is set (#2755)
• Fix toolbox highlight in line plots (#2724)
• Refactor write_results to avoid dynamically overriding config, fixes module.write_data_file (#2722)
• Search stats: do not double-count ignored files (#2708)
• Escape values passed to HTML properties (e.g. val in tables) (#2706)
• Fix re-loading explicit user configs in interactive sessions (#2704)
• Fix file search performance regression (#2762)
• Fix handling module href string (#2739)
• Custom content:
  • Cast names to strings instead of asserting (allows numerical sample names) (#2769)
  • Fix parsing custom content tables with numerical samples (#2705)
  • Section order and custom content order: allow skip the *-module suffix from anchors (#2770)

Module updates

• Kraken: optimize memory and runtime (#2756)
• Mosdepth: optimize memory and runtime (#2748, #2749)
• Abstract config.get_cov_thresholds function for mosdepth and qualimap (#2707)
• Anglerfish: adjust for version 0.6.1 (#2757)
• Umitools: prefer output for sample name, handle the <stdin>/<stdout> placeholders (#2698)
• Kraken: fix top % calculation, more efficient total read count calculation (#2744)
• Bracken: when printing number of found samples, indicate that running Bracken not Kraken (#2743)
• Nonpareil: Update docs about new version that generates the JSON file (#2734)
• STAR: add all alignment summary metrics into a new separate table (#1828)
• Pairtools: fix typos and grammar, remove redundancies (#2711)
• Peddy sex plot: color predicted sex (#2778)

Module fixes

• Cellranger: fix for missing analysis_tab data (#2771)
• Fix setting coverage thresholds for mosdepth and qualimap (#2754)
• Nonpareil: fix running with >12 samples (#2752)
• Bracken: fix bug when direct reads not classified (#2738)
• ngsderive: fix ValueError in the encoding submodule (#2740)
• RSeQC: fix duplicated namespace (#2732)
• Glimpse: fix parsing data, add proper type hints (#2721)
• Fix ignoring samples in spaceranger, ngsbits, isoseq, dragen coverage (#2717)
• Glimpse: clean and fix filtering samples (#2716)
• Spaceranger: fix ignoring samples (#2714)

Refactoring

• Refactor write_results to avoid dynamically overriding config, fixes module.write_data_file (#2722)
• Modules:
  • isoseq and odgi: fix module warnings and error handling (#2718)
  • Qualimap: refactor and add type hints (#2707)
  • FastQC: refactor and add type hints (#2763)
  • Kraken: refactor and add type hints (#2744)
  • Cell Ranger: refactor, add type hints, get rid of module mixins and fields (#2775)
  • Spaceranger: refactor, add type hints, get rid of module mixins and fields (#2714)
  • RSeQC: refactor, add type hints, and remove multiqc_rseqc.js (#2710)

Infrastructure

• Split up the tests for sample versions discovery (#2751)
• Move integration test variations into unit tests, actually test them (#2713)
• Move module docs to the docstrings & generate docs/modules/*.md from the docstrings using a separate script (#2703)
• Embed the search patterns into the module docs (#2765)
• Dockerfile: use COPY instead of ADD to copy only relevant files, update base image to Python 3.12 (#2700)