There is a newer version of the record available.

Published September 24, 2024 | Version v0.20.0

It4innovations/hyperqueue: v0.20.0

  • 1. Rust Project
  • 2. it4i
  • 3. @microsoft

Description

HyperQueue 0.20.0

New features

  • It is now possible to dynamically submit new tasks into an existing job (we call this concept "Open jobs"). See Open jobs documentation

  • Worker streaming. Before, you could stream task stderr/stdout to the server over the network using the --log parameter of hq submit. This approach had various issues and was not scalable. Therefore, we have replaced this functionality with worker streaming, where the streaming of task output to a set of files on disk is performed by workers instead. This new streaming approach creates more files than original solution (where it was always one file per job), but the number of files stays small and independent on the number of executed tasks. The new architecture also allows parallel I/O writing and storing of multiple job streams in one stream handle. You can use worker streaming using the --stream parameter of hq submit. Check out the documentation for more information.

  • Optimization of journal size

  • Tasks' crash counters are not increased when worker is stopped by hq worker stop or by time limit.

Removed

  • Because worker streaming fully replaces original streaming, the original server streaming was removed. For most cases, you can rename --log to --stream and hq log to hq output-log. See the docs for more details.

Fixes

  • HQ should no longer crash while printing job info when a failed task does not have any workers attached (https://github.com/It4innovations/hyperqueue/issues/731).

Note

  • Dashboard still not enabled in this version

Artifact summary:

  • hq-v0.20.0-*: Main HyperQueue build containing the hq binary. Download this archive to use HyperQueue from the command line.
  • hyperqueue-0.20.0-*: Wheel containing the hyperqueue package with HyperQueue Python bindings.

Files

It4innovations/hyperqueue-v0.20.0.zip

Files (1.7 MB)

Name Size
md5:0316a31f643a060b58ab37b2ac385698
1.7 MB Preview Download

Additional details

Related works