It4innovations/hyperqueue: v0.20.0
Authors/Creators
- 1. Rust Project
- 2. it4i
- 3. @microsoft
Description
HyperQueue 0.20.0
New features
It is now possible to dynamically submit new tasks into an existing job (we call this concept "Open jobs"). See Open jobs documentation
Worker streaming. Before, you could stream task stderr/stdout to the server over the network using the
--logparameter ofhq submit. This approach had various issues and was not scalable. Therefore, we have replaced this functionality with worker streaming, where the streaming of task output to a set of files on disk is performed by workers instead. This new streaming approach creates more files than original solution (where it was always one file per job), but the number of files stays small and independent on the number of executed tasks. The new architecture also allows parallel I/O writing and storing of multiple job streams in one stream handle. You can use worker streaming using the--streamparameter ofhq submit. Check out the documentation for more information.Optimization of journal size
Tasks' crash counters are not increased when worker is stopped by
hq worker stopor by time limit.
Removed
- Because worker streaming fully replaces original streaming, the original server streaming was removed.
For most cases, you can rename
--logto--streamandhq logtohq output-log. See the docs for more details.
Fixes
- HQ should no longer crash while printing job info when a failed task does not have any workers attached (https://github.com/It4innovations/hyperqueue/issues/731).
Note
- Dashboard still not enabled in this version
Artifact summary:
- hq-v0.20.0-*: Main HyperQueue build containing the
hqbinary. Download this archive to use HyperQueue from the command line. - hyperqueue-0.20.0-*: Wheel containing the
hyperqueuepackage with HyperQueue Python bindings.
Files
It4innovations/hyperqueue-v0.20.0.zip
Additional details
Related works
- Is supplement to
- Software: https://github.com/It4innovations/hyperqueue/tree/v0.20.0 (URL)