Variable-Length Data in HDF5 Sketch Design

As demonstrated in the benchmarks described in the appendix, the existing implementation of variable-length data in HDF5 has significant performance problems.

In this paper,  we outline the current method of storing variable-length data, discuss the reasons for its poor performance, and offer a sketch of a proposed re-implementation.        

Scot    Breitenfeld wrote the above-mentioned benchmark to compare the performance of the current variable-length data implementation with a mockup of the proposed re-implementation. While the benchmark does not exactly mimic the structure  of the proposed re-implementation,  the  cases are sufficiently similar as to suggest significant performance gains