nextstrain/nextclade: 2.8.0

Nextclade Web 2.8.0, Nextclade CLI 2.8.0 (2022-10-20) Community datasets in Nextclade Web

This release adds support for fetching custom datasets from a remote location. This can be used for testing datasets introducing support for new pathogens, as well as for sharing these datasets with the community.

For that, we added the dataset-url URL query parameter, where you can specify either a direct URL to the directory of your custom dataset:

https://clades.nextstrain.org?dataset-url=http://example.com/path/to/dataset-dir

or a URL to a GitHub repository:

https://clades.nextstrain.org?dataset-url=https://github.com/my-name/my-repo/tree/my-branch/path/to/dataset-dir

or a special shortcut to a GitHub repository:

https://clades.nextstrain.org?dataset-url=github:my-name/my-repo@my-branch@/path/to/dataset-dir

If a branch name is not specified, the default branch name is queried from GitHub. If a path is omitted, then the files are fetched from the root of the repository.

When dataset-url parameter is specified, instead of loading a list of default datasets, a single custom dataset is loaded from the provided address. Note that this should be publicly accessible and have CORS enabled. GitHub public repositories already comply with these requirements, so if you are using a GitHub URL or a shortcut, then no additional action is needed.

For more information, refer to:

• PR #1026
• Nextclade documentation on datasets
• Nextclade data repo
• Nextclade developers guide

Compression of all input and output files in Nextclade CLI and Nextalign CLI

Previously, only FASTA files could be compressed and decompressed on the fly. Now Nextclade CLI and Nextalign CLI can read all input and write all output files in compressed formats. Simply add one of the supported file extensions: "gz", "bz2", "xz", "zstd", and the files will be compressed or decompressed transparently.

Decrease default number of threads in Nextclade Web

Some users have observed long startup times of the analysis in Nextclade Web. In this release we decreased the default number of processing threads from 8 to 3, such that startup time is now a little faster.

If you want to speedup the analysis of large batches of sequences, at the expense of longer startup time, you can tune the number of threads in the "Settings" dialog.

Improve readability of text fragments in Nextclade Web

We've made text paragraphs on main page and some other places a little prettier and hopefully more readable.

Fix crash when reading large, highly-nested tree files

We improved handling of Auspice JSON format, such that it no longer crashes when large trees and trees with large number of deep branches are provided.

<details> <summary><h3>Commit history</h3> (click to expand)</summary> - [[`7438805`](https://github.com/nextstrain/nextclade/commit/7438805b5254fec90c45fecbddb83fe9eda85107)] chore(deps): bump auspice in /packages_rs/nextclade-web Bumps [auspice](https://github.com/nextstrain/auspice) from 2.38.0 to 2.39.0. - [Release notes](https://github.com/nextstrain/auspice/releases) - [Changelog](https://github.com/nextstrain/auspice/blob/master/CHANGELOG.md) - [Commits](https://github.com/nextstrain/auspice/compare/v2.38.0...v2.39.0) --- updated-dependencies: - dependency-name: auspice dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> - [[`891af32`](https://github.com/nextstrain/nextclade/commit/891af32ab1da481e6107158de09311218777fd1e)] Update clades.svg - [[`0e41219`](https://github.com/nextstrain/nextclade/commit/0e4121918b2f02fe82567ce92e9c2ecb7a664e2f)] Merge pull request #1013 from nextstrain/update-clades-svg Update clades.svg - [[`77bcc03`](https://github.com/nextstrain/nextclade/commit/77bcc03b2df3b1892e241ee09c8bd9802f719d0e)] Merge pull request #1012 from nextstrain/dependabot/npm_and_yarn/packages_rs/nextclade-web/auspice-2.39.0 - [[`e6aab13`](https://github.com/nextstrain/nextclade/commit/e6aab13f39777d36355207a92a5b6e9a5ddff0e7)] chore: release web v2.7.1 - [[`3de3205`](https://github.com/nextstrain/nextclade/commit/3de3205a283258fbf3e3934d9e340d2e49b0c016)] chore: disable core dumps when running dev docker container [skip ci] - [[`6cb95a4`](https://github.com/nextstrain/nextclade/commit/6cb95a4c12c95d34dce8f9d8c1bd4dce5b6b1759)] refactor: minify clades.svg - [[`02ea73a`](https://github.com/nextstrain/nextclade/commit/02ea73a066e40d031235e2a1400e0c9a67e61d29)] feat(web): prettify markdown rendering, changelog window text Applies small visual fixes and tweaks to make markdown text a little more readable. Highlights headings in the changelog and fixes a few visual bugs in code snippets. Removes unnecessary and unused styles. - [[`63f6d21`](https://github.com/nextstrain/nextclade/commit/63f6d21324d2f488c8849e951d1f5780e6eb8386)] Merge pull request #1016 from nextstrain/refactor/web-minify-clades-svg - [[`82b93f2`](https://github.com/nextstrain/nextclade/commit/82b93f2f1ddc8bc033ecc30c162402f90dc35f6a)] Merge pull request #1017 from nextstrain/feat/web-prettier-markdown - [[`2539c25`](https://github.com/nextstrain/nextclade/commit/2539c25c0a91e1d21e59fa8ef33ba8b61d9a6cf6)] fix: ensure parsing of large, highly-nested tree JSONs Resolves https://github.com/nextstrain/nextclade/issues/1018 When tree JSON is sufficiently nested, `serde_json` fails to parse it into the `AuspiceTree` `struct` with: ``` recursion limit exceeded at line <line> column <column> ``` This problem has been reported before in `serde_json` in https://github.com/serde-rs/json/issues/334, and the solution allowing to remove the recursion limit was implemented in the form of [`Deserializer::disable_recursion_limit`](https://docs.rs/serde_json/latest/serde_json/struct.Deserializer.html#method.disable_recursion_limit) traded off to the increased risk of overflowing the call stack. So here I implemented the suggested solution, by calling the `disable_recursion_limit()` on deserializer. I am also mitigating the risk of stack overflow by using growing stack adapter from [`serde_stacker`](https://docs.rs/serde_stacker/latest/serde_stacker/). This allows to successfully parse the large tree from https://github.com/nextstrain/nextclade/issues/1018, although probably also makes the tree parsing a bit slower. But it is not critical for overall runtime, as it is a one-off operation. - [[`7c437b0`](https://github.com/nextstrain/nextclade/commit/7c437b0ab17587235ec1bc5bc8efa11f03771267)] chore: remove unrelated changes [skip ci] - [[`6ed348b`](https://github.com/nextstrain/nextclade/commit/6ed348b6ff2ecaa11f3424b8472f6845379a30ef)] refactor: deduplicate json parsing code - [[`33fd032`](https://github.com/nextstrain/nextclade/commit/33fd03274832846293d0578ba4b2cbb98489e5b6)] feat(web): dynamically set dataset server root using a URL param Allows to dynamically set dataset server root URL in Nextclade Web on runtime. It is sufficient to add URL parameter `dataset-server` with the value containing URL to the root of the dataset file server (containing `index_v2.json`), and Nextclade Web will fetch dataset index and all dataset files from this server instead of the default: ``` ?dataset-server=http://example.com ``` The localhost URLs should also work: ``` ?dataset-server=http://localhost:27722 ``` This is equivalent to `--server` flag in Nextclade CLI. This should facilitate testing of custom datasets in Nextclade Web. - [[`02b19a2`](https://github.com/nextstrain/nextclade/commit/02b19a256cdbc37c66bc856aa74f9800b80efd22)] chore: enable CORS for local dataset server - [[`064393e`](https://github.com/nextstrain/nextclade/commit/064393ea6fabd713d201b383d69c2093fc63aa2b)] docs: document dataset-server URL param - [[`ed4b775`](https://github.com/nextstrain/nextclade/commit/ed4b775da1bfe26d30b501590d5837d42b56d8b8)] feat(web): reduce default number of threads to speedup initialization Despite "thread" (actually webworker) initialization is supposed to be concurrent, there seem to be a clear trend of increased initialization time as number of threads increased. This is likely due to data transfer overhead between workers. Here I reduce number of threads that can be initialized with default settings. But these setting can always be altered in the "Settings" dialog. - [[`c4e7736`](https://github.com/nextstrain/nextclade/commit/c4e77367ec3a742f818299a7447ef3e57e282d83)] Merge pull request #1020 from nextstrain/feat/web-dataset-server-url-param - [[`7890180`](https://github.com/nextstrain/nextclade/commit/78901802745f3a18926337779aeb2a2a08341cde)] Merge branch 'master' into fix/tree-json-parsing-recursion-limit - [[`ecdff00`](https://github.com/nextstrain/nextclade/commit/ecdff0041fab4b9a77046e5c24ae6278e239b026)] Merge pull request #1021 from nextstrain/feat/web-decrease-num-threads - [[`d677b95`](https://github.com/nextstrain/nextclade/commit/d677b950dc85e21bd5a4c10ebed664fc67a24a71)] docs: add info about dataset-server URL param to dev guide [skip ci] - [[`e77141e`](https://github.com/nextstrain/nextclade/commit/e77141e115a672fd5688fe89ba8a566585b4648f)] Merge remote-tracking branch 'origin/master' into fix/tree-json-parsing-recursion-limit - [[`62008ff`](https://github.com/nextstrain/nextclade/commit/62008ff877b6378e5e8a5ca06200d2044e96da40)] feat: improve file io logic - [[`ddef52d`](https://github.com/nextstrain/nextclade/commit/ddef52d7707598e8f9a452127bc353ca932e3794)] feat(web): fetch custom dataset from github - [[`68bf7b8`](https://github.com/nextstrain/nextclade/commit/68bf7b8fd62f09eff6a2c0b4ee36bd2dbdb4a159)] feat(web): check existence of github dataset files, use tag data - [[`ed20825`](https://github.com/nextstrain/nextclade/commit/ed208252c6b3e80121d17a5bee410eef860401c0)] refactor: lint - [[`f0c460a`](https://github.com/nextstrain/nextclade/commit/f0c460a309c657fa762163580b5cd8ed4e7f7db2)] feat: ensure github urls to the root of a repo also work - [[`a79a6df`](https://github.com/nextstrain/nextclade/commit/a79a6df0022b22608219dfaa6e63e487590bf801)] refactor: split github function into a separate module - [[`cb6b6c1`](https://github.com/nextstrain/nextclade/commit/cb6b6c1d66b991905edb5a06c85503f3cb29fe6f)] refactor: treat dataset attributes as optional in tag.json - [[`c76e5dd`](https://github.com/nextstrain/nextclade/commit/c76e5ddcda979fd82755865c31d347b673ae531a)] feat(web): accept shortcut GitHub URLs Allows `?dataset-github=` URL param to accept shortcuts instead of full URLs for GitHub repos: ``` github:owner/org github:owner/org/path/to/file github:owner/org@branch@ github:owner/org@branch@/path/to/file ``` If a branch is not specified, the default branch is queried from GitHub REST API (subject to rate limits). Example: ``` ?dataset-github=github:nextstrain/nextclade_data@master@/data/datasets/flu_yam_ha/references/JN993010/versions/2022-07-27T12:00:00Z/files ``` - [[`144c6b4`](https://github.com/nextstrain/nextclade/commit/144c6b461044a503e175dd97b7c42de91224b940)] feat: make `sequences.fasta` optional - [[`6124f64`](https://github.com/nextstrain/nextclade/commit/6124f64f08220886d73f8a4977634e47d1d5a561)] feat(web): allow shorter prefix in github shortcuts Let's allow a shorter `gh:` prefix additionally to `github:` in shortcut GitHub URLs - [[`3858008`](https://github.com/nextstrain/nextclade/commit/3858008fa650b39d4c49317d408d8fec029ce3c8)] feat(web): remove `dataset-github`, detect `dataset-url` type automatically Let's have only one URL param and detect whether we need to treat special cases (GitHub) based on the value. - [[`6cd8f2e`](https://github.com/nextstrain/nextclade/commit/6cd8f2eee9b904b21d22849cc46ca99ea740bede)] fix: replace example GitHub url with a permalink - [[`d747e62`](https://github.com/nextstrain/nextclade/commit/d747e6295a8598c0babc34a98cc6d1a74e0522cc)] docs: document the new dataset-url URL param [skip ci] - [[`f5dfc40`](https://github.com/nextstrain/nextclade/commit/f5dfc4040d9fcc8b273746164c780ccfd18676b3)] Merge pull request #1026 from nextstrain/feat/datasets-from-github - [[`9913aef`](https://github.com/nextstrain/nextclade/commit/9913aef05f439545f360620d5dccf8b7194c31a6)] Merge pull request #1019 from nextstrain/fix/tree-json-parsing-recursion-limit - [[`f3591eb`](https://github.com/nextstrain/nextclade/commit/f3591ebbfae66268d502f8c808f203dd636e0899)] perf: remove mimalloc It was not configured correctly, due to mistake in `cfg` conditional and even when set up correctly does not make Nextclade any faster. See: https://github.com/nextstrain/nextclade/issues/1027 - [[`15fe4af`](https://github.com/nextstrain/nextclade/commit/15fe4af14d2ddabad36b52dd81c3671d718a03b0)] Merge pull request #1028 from nextstrain/perf/allocator-default - [[`6ee3450`](https://github.com/nextstrain/nextclade/commit/6ee34504c84f5bbefeb6675be4616053201c3a9a)] Merge remote-tracking branch 'origin/master' into feat/better-file-io - [[`338c0dc`](https://github.com/nextstrain/nextclade/commit/338c0dc268effb5163daf4aa24c28f983ee9fb42)] refactor: clarify function names, extract constant - [[`20fecc1`](https://github.com/nextstrain/nextclade/commit/20fecc16c45fc11f8caecf7ecbc7840ad3a2598f)] Merge pull request #1029 from nextstrain/feat/better-file-io - [[`a7610d5`](https://github.com/nextstrain/nextclade/commit/a7610d50338faa0026a2ab6d4227400d0cb7be93)] feat(cli): add info about compression to cli help Followup of https://github.com/nextstrain/nextclade/pull/1029 Extends CLI help, now that we can read and write any file in compressed formats. - [[`f4f54c0`](https://github.com/nextstrain/nextclade/commit/f4f54c0fd7273c7fd4bfc55b9f6140490c20ddc9)] Merge pull request #1030 from nextstrain/feat/cli-help-file-io - [[`e73f4a1`](https://github.com/nextstrain/nextclade/commit/e73f4a1ee8d321d1cf95a58aa1200c1539b476d5)] fet(web): increase spacing between headings in changelog - [[`4c82d30`](https://github.com/nextstrain/nextclade/commit/4c82d30462a31954b01bcf701919ed57af335f48)] Merge pull request #1031 from nextstrain/feat/web-md-headings - [[`ba80d9f`](https://github.com/nextstrain/nextclade/commit/ba80d9f39e9cf4c5bee579cc756024a4da84a535)] chore: release web and cli 2.8.0 </details>