There is a newer version of the record available.

Published November 9, 2023 | Version 0.13.1
Software Open

LiberTEM/LiberTEM: 0.13.1

  • 1. Jülich Research Centre, Ernst Ruska Centre
  • 2. CEA-Leti
  • 3. RWTH Aachen University, Jülich Research Centre, Ernst Ruska Centre
  • 4. Government Engineering College Sreekrishnapuram
  • 5. Chandigarh University
  • 6. Siliguri Institute of Technology
  • 7. Jadavpur University
  • 8. University of Illinois at Urbana-Champaign
  • 9. Direct Electron
  • 10. University of Antwerp
  • 11. Lawrence Berkeley National Laboratory
  • 12. University of Szeged
  • 13. Monash University eResearch Centre
  • 14. ETH Zürich
  • 15. Columbia University
  • 16. Norwegian University of Science and Technology
  • 1. FZ Jülich
  • 2. X-Spectrum
  • 3. MPSD
  • 4. HZM
  • 5. Brigham and Women's Hospital
  • 6. Frederick National Laboratory for Cancer Research
  • 7. Quantum Detectors
  • 8. pyXem
  • 9. HZDR
  • 10. HU Berlin
  • 11. Berkeley Lab
  • 12. Microscopy Australia
  • 13. DESY
  • 14. Göttingen University
  • 15. Monash
  • 16. U Antwerp
  • 17. Weizmann
  • 18. Rosalind Franklin Institute
  • 19. PNDetector
  • 20. NeXus project
  • 21. Anaconda Inc.
  • 22. Gatan
  • 23. U Glasgow
  • 24. U Bremen
  • 25. Nion
  • 26. Point Electronic
  • 27. Python Software Foundation
  • 28. TVIPS
  • 29. Lille
  • 30. U Limerick
  • 31. Iambic Innovation
  • 32. U Wisconsin
  • 33. PSI
  • 34. U Queensland
  • 35. HZB

Description

Homepage: https://libertem.github.io/LiberTEM/
GitHub repository: https://github.com/LiberTEM/LiberTEM/
PyPI: https://pypi.org/project/libertem/

LiberTEM is an open source platform for high-throughput distributed processing of large-scale binary data sets and live data streams using a modified MapReduce programming model. The current focus is pixelated scanning transmission electron microscopy (STEM) and scanning electron beam diffraction data.

MapReduce-like processing allows to specify an algorithm through two functions: One function that is mapped on portions of the input data, and another function that merges (reduces) a partial result from this mapping step into the complete result. A wide range of TEM and 4D STEM processing tasks can be expressed in this fashion, see Applications.

The UDF interface of LiberTEM offers a standardized, versatile API to decouple the mathematical core of an algorithm from details of data source, parallelism, and use of results. Mapping and merging can be performed in any order and with different subdivisions of the input data, including running parts of the calculation concurrently. That means the same implementation can be used in a wide range of modalities, including massive scaling on clusters. Since each merge step produces an intermediate result, this style of processing is suitable for displaying live results from a running calculation in a GUI application and for processing live data streams. A closed-loop feedback between processing and instrument control can be realized as well. See User-defined functions for more details on the LiberTEM UDF interface.

The LiberTEM back-end offers high throughput and scalability on PCs, single server nodes, clusters and cloud services. On clusters it can use fast distributed local storage on high-performance SSDs. That way it achieves very high aggregate IO performance on a compact and cost-efficient system built from stock components. All CPU cores and CUDA devices in a system can be used in parallel.

LiberTEM is supported on Linux, Mac OS X and Windows. Other platforms that allow installation of Python 3.7+ and the required packages will likely work as well. The GUI is running in a web browser.

Installation

The short version:

$ virtualenv -p python3 ~/libertem-venv/
$ source ~/libertem-venv/bin/activate
(libertem-venv) $ python -m pip install "libertem[torch]"

# optional for GPU support
# See also https://docs.cupy.dev/en/stable/install.html
(libertem-venv) $ python -m pip install cupy

Please see our documentation for details!

Alternatively, to run the LiberTEM Docker image:

$ docker run -p localhost:9000:9000 --mount type=bind,source=/path/to/your/data/,dst=/data/,ro ghcr.io/libertem/libertem

or

$ singularity exec docker://ghcr.io/libertem/libertem /venv/bin/libertem-server

Deployment for offline data processing on a single-node system for a local user is thoroughly tested and can be considered stable. Deployment on a cluster is experimental and still requires some additional work, see Issue #105. Back-end support for live data processing is still experimental as well, see https://github.com/LiberTEM/LiberTEM-live.

Applications

Since LiberTEM is programmable through user-defined functions (UDFs), it can be used for a wide range of processing tasks on array-like data and data streams. The following applications have been implemented already:

Some of these applications are available through an interactive web GUI. Please see the applications section of our documentation for details!

The Python API and user-defined functions (UDFs) can be used for complex operations such as arbitrary linear operations and other features like data export. Example Jupyter notebooks are available in the examples directory. If you are having trouble running the examples, please let us know by filing an issue or by joining our Gitter chat.

LiberTEM is suitable as a high-performance processing backend for other applications, including live data streams. Contact us if you are interested!

LiberTEM is evolving rapidly and prioritizes features following user demand and contributions. Currently we are working on live data processing, improving application support for sparse data and event-based detectors, performance improvements for GPU processing, and implementing analysis methods for various applications of pixelated STEM and other large-scale detector data. If you like to influence the direction this project is taking, or if you'd like to contribute, please join our gitter chat and our general mailing list.

File formats

LiberTEM currently opens most file formats used for pixelated STEM. See our general information on loading data and format-specific documentation for more information!

  • Raw binary files
  • NumPy .npy binary files
  • Thermo Fisher EMPAD detector files
  • Quantum Detectors MIB format
  • Nanomegas .blo block files
  • Direct Electron DE5 files (HDF5-based) and Norpix SEQ files for DE-Series detectors
  • Gatan K2 IS raw format
  • Stacks of Gatan DM3 and DM4 files (via openNCEM)
  • Single-file Gatan DM4 scans when saved using C-ordering
  • FRMS6 from PNDetector pnCCD cameras (currently alpha, gain correction still needs UI changes)
  • FEI SER files (via openNCEM)
  • MRC (via openNCEM)
  • HDF5-based formats such as HyperSpy files, NeXus and EMD
  • TVIPS binary files
  • Sparse data in Raw CSR (compressed sparse row) format, as is possible to generate from event-based detectors
  • Please contact us if you are interested in support for an additional format!

Live processing and detectors (experimental)

See LiberTEM-live!

License

LiberTEM is licensed under GPLv3. The I/O parts are also available under the MIT license, please see LICENSE files in the subdirectories for details.

Acknowledgements

We are very grateful for your continuing support for LiberTEM!

See the acknowledgement page for a list of authors and contributors to LiberTEM and its subprojects. See also our info on funding and industry partners.

Other

We gratefully acknowledge funding from the Initiative and Network Fund of the Helmholtz Association within the Helmholtz Young Investigator Group moreSTEM under Contract No. VH-NG-1317 at Forschungszentrum Jülich in Germany.

We gratefully acknowledge funding from the Information & Data Science Pilot Project 'Ptychography 4.0' of the Helmholtz Association.

We kindly acknowledge funding from Google Summer of Code 2019 under the umbrella of the Python software foundation.

STEMx equipment and software for 4D STEM data acquisition with K2 IS camera courtesy of Gatan Inc.

LiberTEM development is supported by AIDAS

LiberTEM development is part of the Joint Lab MDMC.

Forschungszentrum Jülich is supporting LiberTEM with funding for personnel, access to its infrastructure and administrative support.

Files

Files (29.8 MB)

Name Size Download all
md5:8c3b4a2e7237667e17d9b7325afd29ba
1.7 MB Download
md5:3a3542acc674c5ffecd8d17c4b9195fc
2.0 MB Download
md5:013dd04329537c27af95f0b3fe4189b4
26.2 MB Download

Additional details

Funding

VIDEO – Versatile and Innovative Detector for Electron Optics 780487
European Commission
CritCat – Towards Replacement of Critical Catalyst Materials by Improved Nanoparticle Control and Rational Design 686053
European Commission
ESTEEM3 – Enabling Science and Technology through European Electron Microscopy 823717
European Commission
3D MAGiC – Three-dimensional magnetization textures: Discovery and control on the nanoscale 856538
European Commission

References

  • [Weber2018] Weber, Dieter (2018): Development of {IT} system and {TEM} camera performance. Zenodo. 10.5281/zenodo.2450624
  • [Bon2012] Pierre Bon and Serge Monneret and Benoit Wattellier (2012): Noniterative boundary-artifact-free wavefront reconstruction from its derivatives. The Optical Society. 10.1364/ao.51.005698
  • [K2IS2018] {{Gatan Inc.}} (2018): {K2} {IS} camera. https://web.archive.org/web/20180809021832/http://www.gatan.com/products/tem-imaging-spectroscopy/k2-camera
  • [Clausen2018] Clausen, Alexander and Weber, Dieter and {Probonopd} and Caron, Jan and Nord, Magnus and Müller-Caspary, Knut and Ophus, Colin and Dunin-Borkowski, Rafal (2018): Libertem/Libertem: 0.1.0. Zenodo. 10.5281/zenodo.1477847
  • [Sauter2013] Nicholas K. Sauter and Johan Hattne and Ralf W. Grosse-Kunstleve and Nathaniel Echols (2013): New Python-based methods for data processing. International Union of Crystallography ({IUCr}). 10.1107/s0907444913000863
  • [Koennecke2015] Mark Könnecke and Frederick A. Akeroyd and Herbert J. Bernstein and Aaron S. Brewster and Stuart I. Campbell and Björn Clausen and Stephen Cottrell and Jens Uwe Hoffmann and Pete R. Jemian and David Männicke and Raymond Osborn and Peter F. Peterson and Tobias Richter and Jiro Suzuki and Benjamin Watts and Eugen Wintersberger and Joachim Wuttke (2015): The {NeXus} data format. International Union of Crystallography ({IUCr}). 10.1107/s1600576714027575
  • [Fangohr2018] Fangohr, Hans and Aplin, Steve and Barty, Anton and Beg, Marijan and Bondar, Valerii and Boukhelef, Djelloul and Brockhauser, Sandor and Danilevski, Cyril and Ehsan, Wajid and Esenov, Sergey and Flucke, Gero and Giovanetti, Gabriele and Goeries, Dennis and Hauf, Steffen and Heisen, Burkhard and Hickin, David and Khakhulin, Dmitry and Klimovskaia, Anna and Kluyver, Thomas and Kuhn, Manuela and Kuster, Markus and Lang, Philipp-Michael and Maia, Luis and Mariani, Valerio and Mekinda, Leonce and Michelat, Thomas and Parenti, Andrea and Previtali, Gianpietro and Santos, Hugo and Silenzi, Alessandro and Sztuk-Dambietz, Jolanta and Szuba, Janusz and Teichmann, Martin and Weger, Kerstin and Wiggins, John and Wrona, Krzysztof and Xu, Chen (2018): Data Analysis Support in Karabo at European XFEL. JACoW, Geneva, Switzerland. 10.18429/jacow-icalepcs2017-tucpa01
  • [Jesse2016] S. Jesse and M. Chi and A. Belianinov and C. Beekman and S. V. Kalinin and A. Y. Borisevich and A. R. Lupini (2016): Big Data Analytics for Scanning Transmission Electron Microscopy Ptychography. Springer Nature. 10.1038/srep26348
  • [Belianinov2015] Alex Belianinov and Rama Vasudevan and Evgheni Strelcov and Chad Steed and Sang Mo Yang and Alexander Tselev and Stephen Jesse and Michael Biegalski and Galen Shipman and Christopher Symons and Albina Borisevich and Rick Archibald and Sergei Kalinin (2015): Big data and deep data in scanning and electron microscopies: deriving functionality from multidimensional data sets. Springer Nature. 10.1186/s40679-015-0006-6
  • [Krajnak2016] Matus Krajnak and Damien McGrouther and Dzmitry Maneuski and Val O{\textquotesingle} Shea and Stephen McVitie (2016): Pixelated detectors and improved efficiency for magnetic imaging in {STEM} differential phase contrast. Elsevier {BV}. 10.1016/j.ultramic.2016.03.006
  • [Yang2015] H Yang and L Jones and H Ryll and M Simson and H Soltau and Y Kondo and R Sagawa and H Banba and I MacLaren and P D Nellist (2015): 4D {STEM}: High efficiency phase contrast imaging using a fast pixelated detector. {IOP} Publishing. 10.1088/1742-6596/644/1/012032
  • [Sagawa2017] Ryusuke Sagawa and Hao Yang and Lewys Jones and Martin Simson and Martin Huth and Heike Soltau and Peter D. Nellist and Yukihito Kondo (2017): Development of Fast Pixelated {STEM} Detector and its Applications using 4-Dimensional Dataset. Cambridge University Press ({CUP}). 10.1017/s1431927617000940
  • [doi:10.1002/9783527808465.EMC2016.6284] MacLaren, Ian and Nord, Magnus and Ross, Andrew and Krajnak, Matus and Hart, Martin and Doye, Alastair and McGrouther, Damien and Bali, Rantej and Banerjee, Archan and Hadfield, Robert (2016): Pixelated STEM detectors: opportunities and challenges. American Cancer Society. 10.1002/9783527808465.EMC2016.6284
  • [Nguyen2016] Kayla X. Nguyen and Prafull Purohit and Robert Hovden and Emrah Turgut and Mark W. Tate and Lena F. Kourkoutis and Gregory D. Fuchs and Sol M. Gruner and David A. Muller (2016): 4D-{STEM} for Quantitative Imaging of Magnetic Materials with Enhanced Contrast and Resolution. Cambridge University Press ({CUP}). 10.1017/s1431927616009430
  • [Cowley1979] J.M. Cowley (1979): Coherent interference in convergent-beam electron diffraction and shadow imaging. Elsevier {BV}. 10.1016/s0304-3991(79)80021-2
  • [Hoppe1969] W. Hoppe (1969): Beugung im inhomogenen Primärstrahlwellenfeld. I. Prinzip einer Phasenmessung von Elektronenbeungungsinterferenzen. International Union of Crystallography ({IUCr}). 10.1107/s0567739469001045
  • [Hegerl1970] R. Hegerl and W. Hoppe (1970): Dynamische Theorie der Kristallstrukturanalyse durch Elektronenbeugung im inhomogenen Primärstrahlwellenfeld. Wiley. 10.1002/bbpc.19700741112
  • [Nellist1995] P. D. Nellist and B. C. McCallum and J. M. Rodenburg (1995): Resolution beyond the {\textquotesingle}information limit{\textquotesingle} in transmission electron microscopy. Springer Nature. 10.1038/374630a0
  • [McCallum1993] B. C. McCallum and J. M. Rodenburg (1993): Simultaneous reconstruction of object and aperture functions from multiple far-field intensity measurements. The Optical Society. 10.1364/josaa.10.000231
  • [Friedman1992] S L Friedman and J M Rodenburg (1992): Optical demonstration of a new principle of far-field microscopy. {IOP} Publishing. 10.1088/0022-3727/25/2/003
  • [Humphreys1988] C.J. Humphreys and D.J. Eaglesham and D.M. Maher and H.L. Fraser (1988): {CBED} and {CBIM} from semiconductors and superconductors. Elsevier {BV}. 10.1016/0304-3991(88)90371-3
  • [Steeds1979] J. W. Steeds (1979): Convergent Beam Electron Diffraction. Springer {US}. 10.1007/978-1-4757-5581-7_15
  • [Tate2016] Mark W. Tate and Prafull Purohit and Darol Chamberlain and Kayla X. Nguyen and Robert Hovden and Celesta S. Chang and Pratiti Deb and Emrah Turgut and John T. Heron and Darrell G. Schlom and Daniel C. Ralph and Gregory D. Fuchs and Katherine S. Shanks and Hugh T. Philipp and David A. Muller and Sol M. Gruner (2016): High Dynamic Range Pixel Array Detector for Scanning Transmission Electron Microscopy. Cambridge University Press ({CUP}). 10.1017/s1431927615015664
  • [Jiang2018] Yi Jiang and Zhen Chen and Yimo Han and Pratiti Deb and Hui Gao and Saien Xie and Prafull Purohit and Mark W. Tate and Jiwoong Park and Sol M. Gruner and Veit Elser and David A. Muller (2018): Electron ptychography of 2D materials to deep sub-{\aa}ngström resolution. Springer Nature. 10.1038/s41586-018-0298-5
  • [Ophus2014] Colin Ophus and Peter Ercius and Michael Sarahan and Cory Czarnik and Jim Ciston (2014): Recording and Using 4D-{STEM} Datasets in Materials Science. Cambridge University Press ({CUP}). 10.1017/s1431927614002037
  • [Simson2015] M. Simson and H. Ryll and H. Banba and R. Hartmann and M. Huth and S. Ihle and L. Jones and Y. Kondo and K. Muller and P.D. Nellist and R. Sagawa and J. Schmidt and H. Soltau and L. Striider and H. Yang (2015): 4D-{STEM} Imaging With the {pnCCD} (S){TEM}-Camera. Cambridge University Press ({CUP}). 10.1017/s1431927615011836
  • [Pennycook2015] Timothy J. Pennycook and Andrew R. Lupini and Hao Yang and Matthew F. Murfitt and Lewys Jones and Peter D. Nellist (2015): Efficient phase contrast imaging in {STEM} using a pixelated detector. Part 1: Experimental demonstration at atomic resolution. Elsevier {BV}. 10.1016/j.ultramic.2014.09.013
  • [Yang2015a] Hao Yang and Timothy J. Pennycook and Peter D. Nellist (2015): Efficient phase contrast imaging in {STEM} using a pixelated detector. Part {II}: Optimisation of imaging conditions. Elsevier {BV}. 10.1016/j.ultramic.2014.10.013
  • [QuantumMerlin2017] {Quantum Detectors} (2017): Merlin for {EM} technical datasheet. https://quantumdetectors.com/products/merlinem/
  • [Lawrence2017] Ethan L. Lawrence and Shery L. Y. Chang and Peter A. Crozier (2017): In situ {TEM} observations of Oxygen Surface Dynamics in {CeO}2 Cubes. Cambridge University Press ({CUP}). 10.1017/s1431927617010637
  • [Li2013] Xueming Li and Paul Mooney and Shawn Zheng and Christopher R Booth and Michael B Braunfeld and Sander Gubbens and David A Agard and Yifan Cheng (2013): Electron counting and beam-induced motion correction enable near-atomic-resolution single-particle cryo-{EM}. Springer Nature. 10.1038/nmeth.2472
  • [XSpectrum-LambdaM2] X-Spectrum: {Lambda2M} large area {Medipix3} based detector array. http://www.x-spectrum.de/index_htm_files/X-Spectrum_datasheet_2M.pdf
  • [Li:2018ngp] Li, Xin and Dyck, Ondrej and Kalinin, Sergei V. and Jesse, Stephen (2018): Compressed Sensing of Scanning Transmission Electron Microscopy {(STEM)} on Non-Rectangular Scans. 1805.04957
  • [Nielsen1994] Nielsen, Jakob (1994): Usability Engineering. Elsevier Science. https://www.ebook.de/de/product/21119819/jakob_nielsen_usability_engineering.html
  • [Lee2011] Craig A. Lee and Samuel D. Gasster and Antonio Plaza and Chein-I Chang and Bormin Huang (2011): Recent Developments in High Performance Computing for Remote Sensing: A Review. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/jstars.2011.2162643
  • [Tezduyar1996] T. Tezduyar and S. Aliabadi and M. Behr and A. Johnson and V. Kalro and M. Litke (1996): Flow simulation and high performance computing. Springer Nature America, Inc. 10.1007/bf00350249
  • [Allcock2002] Bill Allcock and Joe Bester and John Bresnahan and Ann L. Chervenak and Ian Foster and Carl Kesselman and Sam Meder and Veronika Nefedova and Darcy Quesnel and Steven Tuecke (2002): Data management and transfer in high-performance computational grid environments. Elsevier {BV}. 10.1016/s0167-8191(02)00094-7
  • [Pan2016] Ming Pan and Cory Czarnik (2016): Image Detectors for Environmental Transmission Electron Microscopy ({ETEM}). Springer International Publishing. 10.1007/978-3-319-22988-1_5
  • [Knuth1974] Donald E. Knuth (1974): Structured Programming with go to Statements. Association for Computing Machinery ({ACM}). 10.1145/356635.356640
  • [Busch2016] Busch, Kat (2016): The Rules of Optimization: Why So Many Performance Efforts Fail. https://hackernoon.com/the-rules-of-optimization-why-so-many-performance-efforts-fail-cf06aad89099
  • [Delvecchio2011] Delvecchio, Paul (2011): De-Mystifying Software Performance Optimization. https://software.intel.com/en-us/articles/de-mystifying-software-performance-optimization
  • [Mavridis2017] Ilias Mavridis and Helen Karatza (2017): Performance evaluation of cloud-based log file analysis with Apache Hadoop and Apache Spark. Elsevier {BV}. 10.1016/j.jss.2016.11.037
  • [Dean2008] Jeffrey Dean and Sanjay Ghemawat (2008): {MapReduce}. Association for Computing Machinery ({ACM}). 10.1145/1327452.1327492
  • [Stegmaier2017] Stegmaier, Johannes (2017): New Methods to Improve Large-Scale Microscopy Image Analysis with Prior Knowledge and Uncertainty. KIT Scientific Publishing. 10.5445/ksp/1000060221
  • [Goscinski2014] Wojtek J. Goscinski and Paul McIntosh and Ulrich Felzmann and Anton Maksimenko and Christopher J. Hall and Timur Gureyev and Darren Thompson and Andrew Janke and Graham Galloway and Neil E. B. Killeen and Parnesh Raniga and Owen Kaluza and Amanda Ng and Govinda Poudel and David G. Barnes and Toan Nguyen and Paul Bonnington and Gary F. Egan (2014): The multi-modal Australian {ScienceS} Imaging and Visualization Environment ({MASSIVE}) high performance computing infrastructure: applications in neuroscience and neuroinformatics research. Frontiers Media {SA}. 10.3389/fninf.2014.00030
  • [Foust2015] Gabriel Foust and Jaakko Järvi and Sean Parent (2015): Generating reactive programs for graphical user interfaces from multi-way dataflow constraint systems. {ACM} Press. 10.1145/2814204.2814207
  • [Shneiderman2009] Ben Shneiderman and Catherine Plaisant and Maxine Cohen and Steven Jacobs (2009): Designing the User Interface: Strategies for Effective Human-Computer Interaction (5th Edition). Pearson. https://www.amazon.com/Designing-User-Interface-Human-Computer-Interaction/dp/0321537351?SubscriptionId=AKIAIOBINVZYXZQZ2U3A&tag=chimbori05-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=0321537351
  • [Tsukamoto2014] Takafumi Tsukamoto and Takuo Yasunaga (2014): Development of a user-friendly system for image processing of electron microscopy by integrating a web browser and {PIONE} with Eos. Oxford University Press ({OUP}). 10.1093/jmicro/dfu070
  • [Rogic] Ivan Rogic: {React}, {Redux} and {Immutable.js}: Ingredients for Efficient Web Applications. https://www.toptal.com/react/react-redux-and-immutablejs
  • [Lachlan] Lachlan Miller (2017): Redux-driven {UI} design. https://medium.com/@lachlanmiller_52885/redux-driven-ui-design-8586a84d808f
  • [Taylor1996] R.N. Taylor and N. Medvidovic and K.M. Anderson and E.J. Whitehead and J.E. Robbins and K.A. Nies and P. Oreizy and D.L. Dubrow (1996): A component- and message-based architectural style for {GUI} software. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/32.508313
  • [pyFAI] {Data analysis unit, European Synchrotron Radiation Facility, Grenoble} (2018): General introduction to {PyFAI}. https://pyfai.readthedocs.io/en/latest/pyFAI.html
  • [Alted2010] Francesc Alted (2010): Why Modern {CPUs} Are Starving and What Can Be Done about It. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcse.2010.51
  • [Clausen2018a] Clausen, Alexander and Weber, Dieter and {Probonopd} and Caron, Jan and Nord, Magnus and Müller-Caspary, Knut and Ophus, Colin and Dunin-Borkowski, Rafal (2018): Libertem/Libertem: 0.1.0. Zenodo. 10.5281/zenodo.1478763
  • [IntelLustre] {Intel High Performance Data Division} (2014): Architecting a High Performance Storage System. https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/architecting-lustre-storage-white-paper.pdf
  • [Patel2012] Aditya B. Patel and Manashvi Birla and Ushma Nair (2012): Addressing big data problem using {Hadoop} and {Map Reduce}. {IEEE}. 10.1109/nuicone.2012.6493198
  • [Larsen2000] Samuel Larsen and Saman Amarasinghe (2000): Exploiting superword level parallelism with multimedia instruction sets. Association for Computing Machinery ({ACM}). 10.1145/358438.349320
  • [Walt2011] St{\'{e}}fan van der Walt and S Chris Colbert and Gaël Varoquaux (2011): The {NumPy} Array: A Structure for Efficient Numerical Computation. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcse.2011.37
  • [Thadani19951AE] Moti Thadani and Yousef Y. A. Khalidi (1995): An Efficient Zero-Copy {I / O} Framework for {UNIX} ®.
  • [Stancevic2003] Dragan Stancevic (2003): Zero Copy I: User-Mode Perspective. https://www.linuxjournal.com/article/6345?page=0,0
  • [Kelly2003] Paul H J Kelly (2003): Advanced Computer Architecture: Caches and Memory Systems. https://www.doc.ic.ac.uk/~phjk/AdvancedCompArchitecture/2003-04/Lectures/Ch03/ACA-CH03-CurrentVersion.pdf
  • [Shvachko2010] Konstantin Shvachko and Hairong Kuang and Sanjay Radia and Robert Chansler (2010): The Hadoop Distributed File System. {IEEE}. 10.1109/msst.2010.5496972
  • [HDFS-Short] {Apache Software Foundation} (2018): {HDFS} Short-Circuit Local Reads. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html
  • [Cooley1965] James W. Cooley and John W. Tukey (1965): An algorithm for the machine calculation of complex Fourier series. American Mathematical Society ({AMS}). 10.1090/s0025-5718-1965-0178586-1
  • [Zaharia2016] Matei Zaharia and Michael J. Franklin and Ali Ghodsi and Joseph Gonzalez and Scott Shenker and Ion Stoica and Reynold S. Xin and Patrick Wendell and Tathagata Das and Michael Armbrust and Ankur Dave and Xiangrui Meng and Josh Rosen and Shivaram Venkataraman (2016): Apache Spark. Association for Computing Machinery ({ACM}). 10.1145/2934664
  • [Rocklin2015] Matthew Rocklin (2015): Dask: Parallel Computation with Blocked algorithms and Task Scheduling. {SciPy}. 10.25080/majora-7b98e3ed-013
  • [BLAS] (2017): {BLAS} ({Basic Linear Algebra Subprograms}). http://www.netlib.org/blas/
  • [PyTorch] (2018): PyTorch. https://pytorch.org/
  • [MKL] Intel (2018): Intel® Math Kernel Library. https://software.intel.com/en-us/mkl
  • [Godbolt] Matt Godbolt: Compiler Explorer. https://godbolt.org/
  • [Numba] (2018): Numba: A high-performance {Python} compiler. http://numba.pydata.org/
  • [ReactJS] {Facebook Inc.} (2018): React: A {JavaScript} library for building user interfaces. https://reactjs.org/
  • [ReduxJS] Dan Abramov (2018): Redux: A predictable state container for {JavaScript} apps.. https://redux.js.org/
  • [MerkelDocker] Dirk Merkel (2014): Docker: Lightweight Linux Containers for Consistent Development and Deployment. https://www.linuxjournal.com/content/docker-lightweight-linux-containers-consistent-development-and-deployment
  • [Bernstein2014] David Bernstein (2014): Containers and Cloud: From {LXC} to Docker to Kubernetes. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/mcc.2014.51
  • [Shaw1990] Mary Shaw (1990): Prospects for an engineering discipline of software. Institute of Electrical and Electronics Engineers ({IEEE}). 10.1109/52.60586
  • [Schindelin2012] Johannes Schindelin and Ignacio Arganda-Carreras and Erwin Frise and Verena Kaynig and Mark Longair and Tobias Pietzsch and Stephan Preibisch and Curtis Rueden and Stephan Saalfeld and Benjamin Schmid and Jean-Yves Tinevez and Daniel James White and Volker Hartenstein and Kevin Eliceiri and Pavel Tomancak and Albert Cardona (2012): Fiji: an open-source platform for biological-image analysis. Springer Nature. 10.1038/nmeth.2019
  • [Willinsky2005] John Willinsky (2005): The unacknowledged convergence of open source, open access, and open science. University of Illinois Libraries. 10.5210/fm.v10i8.1265
  • [Zuiderwijk2012] Anneke Zuiderwijk and Keith Jeffery and Marijn Janssen (2012): The Potential of Metadata for Linked Open Data and its Value for Users and Publishers. {JEDEM} Journal of e-Democracy and Open Government. 10.29379/jedem.v4i2.138
  • [HDF5] (2016): High Level Introduction to {HDF5}. https://support.hdfgroup.org/HDF5/Tutor/HDF5Intro.pdf
  • [Okoli2015] Chitu Okoli and Johannes Nguyen (2015): Business Models for Free and Open Source Software. Elsevier {BV}. 10.2139/ssrn.2568185
  • [ApacheFoundation] {{The APACHE Software Foundation}} (2018): The {APACHE} Software Foundation. https://www.apache.org/
  • [Linuxfoundation] {The Linux Foundation} (2018): The Linux Foundation. https://www.linuxfoundation.org/
  • [Williams1996] David B. Williams and C. Barry Carter (1996): The Transmission Electron Microscope. Springer {US}. 10.1007/978-1-4757-2519-3_1
  • [Goto2008] Kazushige Goto and Robert A. van de Geijn (2008): Anatomy of high-performance matrix multiplication. Association for Computing Machinery ({ACM}). 10.1145/1356052.1356053
  • [DaskSpark] {Anaconda Inc.} (2018): Comparison to Spark. http://docs.dask.org/en/latest/spark.html
  • [Lemire2012] Daniel Lemire and Leonid Boytsov: Decoding billions of integers per second through vectorization. 10.1002/spe.2203
  • [grosser2011polly] Grosser, Tobias and Zheng, Hongbin and Aloor, Raghesh and Simb{\"u}rger, Andreas and Gr{\"o}{\ss}linger, Armin and Pouchet, Louis-No{\"e}l: Polly-Polyhedral optimization in {LLVM}. http://perso.ens-lyon.fr/christophe.alias/impact2011/impact-07.pdf
  • [Song2018] Dongsheng Song and Zi-An Li and Jan Caron and Andr{\'{a}}s Kov{\'{a}}cs and Huanfang Tian and Chiming Jin and Haifeng Du and Mingliang Tian and Jianqi Li and Jing Zhu and Rafal E. Dunin-Borkowski (2018): Quantification of Magnetic Surface and Edge States in an {FeGe} Nanostripe by Off-Axis Electron Holography. American Physical Society ({APS}). 10.1103/physrevlett.120.167204
  • [Mehta2016] Parmita Mehta and Sven Dorkenwald and Dongfang Zhao and Tomer Kaftan and Alvin Cheung and Magdalena Balazinska and Ariel Rokem and Andrew Connolly and Jacob Vanderplas and Yusra AlSayyad: Comparative Evaluation of Big-Data Systems on Scientific Image Analytics Workloads. http://arxiv.org/abs/1612.02485v1
  • [Goy2018] Alexandre Goy and Kwabena Arthur and Shuai Li and George Barbastathis (2018): Low Photon Count Phase Retrieval Using Deep Learning. American Physical Society ({APS}). 10.1103/physrevlett.121.243902
  • [DistributedJourney] {{Anaconda, Inc}} (2018): Journey of a Task. https://distributed.dask.org/en/latest/journey.html
  • [AbramovFlux] Dan Abramov (2015): The Case for Flux. https://medium.com/swlh/the-case-for-flux-379b7d1982c6
  • [ReduxCore] Dan Abramov (2018): Redux: Core Concepts. https://redux.js.org/introduction/core-concepts
  • [ReduxReact] Dan Abramov (2018): {Redux}: Usage with {React}. https://redux.js.org/basics/usage-with-react
  • [FIO] Jens Axboe: Flexible I/O Tester. https://github.com/axboe/fio
  • [Caron2018] Caron, Jan (2018): Model-based reconstruction of magnetisation distributions in nanostructures from electron optical phase images [E-Book]. Forschungszentrum, Zentralbibliothek. http://hdl.handle.net/2128/19740
  • [Guzzinati2019] Giulio Guzzinati and Wannes Ghielens and Christoph Mahr and Armand Béché and Andreas Rosenauer and Toon Calders and Jo Verbeeck (2019): Electron Bessel beam diffraction for precise and accurate nanoscale strain mapping. 10.1063/1.5096245
  • [Panova2019] Ouliana Panova and Colin Ophus and Christopher J. Takacs and Karen C. Bustillo and Luke Balhorn and Alberto Salleo and Nitash Balsara and Andrew M. Minor (2019): Diffraction imaging of nanocrystalline structures in organic semiconductor molecular thin films. Springer Science and Business Media {LLC}. 10.1038/s41563-019-0387-3
  • [Gibson1997] J. M. Gibson and M. M. J. Treacy (1997): Diminished Medium-Range Order Observed in Annealed Amorphous Germanium. American Physical Society ({APS}). 10.1103/physrevlett.78.1074
  • [Ophus_2019] Colin Ophus (2019): Four-Dimensional Scanning Transmission Electron Microscopy (4D-{STEM}): From Scanning Nanodiffraction to Ptychography and Beyond. Cambridge University Press ({CUP}). 10.1017/s1431927619000497
  • [6980942] S. C. F. {Lin} and C. Y. {Wong} and G. {Jiang} and M. A. {Rahman} and N. M. {Kwok} (2014): Radial fourier analysis ({{RFA}}) image descriptor. 10.1109/FSKD.2014.6980942
  • [Lichte2008] Lichte, H. and Lehmann, M. (2008): Electron holography–basics and applications. 10.1088/0034-4885/71/1/016102
  • [Schubert2018] Erich Schubert and Michael Gertz (2018): Numerically stable parallel computation of (co-)variance. {ACM} Press. 10.1145/3221269.3223036
  • [ophus_colin_2019_3592520] Ophus, Colin and Savitzky, Benjamin (2019): {Simulated calibration dataset for 4D scanning transmission electron microscopy}. Zenodo. 10.5281/zenodo.3592520
  • [giulio_guzzinati_2019_2566137] Giulio Guzzinati and Wannes Ghielens and Christoph Mahr and Armand Béché and Andreas Rosenauer and Toon Calders and Jo Verbeeck (2019): {Electron Bessel beam diffraction patterns, line scan of Si/SiGe multilayer}. Zenodo. 10.5281/zenodo.2566137
  • [Weber2020] Dieter Weber and Alexander Clausen and Rafal E. Dunin-Borkowski (2020): Next-Generation Information Technology Systems for Fast Detectors in Electron Microscopy. World Scientific. 10.1142/9789811204579_0005
  • [Zeltmann2019] Steven E Zeltmann and Alexander Müller and Karen C Bustillo and Benjamin Savitzky and Lauren Hughes and Andrew M Minor and Colin Ophus (2019): Patterned Probes for High Precision {4D-STEM} Bragg Measurements. 10.1016/j.ultramic.2019.112890
  • [strauch_achim_2021_5113449] Strauch, Achim and Clausen, Alexander and Weber, Dieter and Müller-Caspary, Knut (2021): {High-resolution {{4D STEM}} dataset of {{SrTiO3}} along the [1 0 0] axis at high magnification}. Zenodo. 10.5281/zenodo.5113449
  • [strauch_achim_2021_5113235] Strauch, Achim and Weber, Dieter and Clausen, Alexander and Müller-Caspary, Knut (2021): {Synthetic 4D STEM dataset based on a SrTiO3 supercell with two additional artificial spatial frequencies}. Zenodo. 10.5281/zenodo.5113235
  • [t_pennycook_2021_4476506] T. Pennycook (2021): Graphene simulated dataset. Zenodo. 10.5281/zenodo.4476506
  • [ROSE1976251] H. Rose (1976): Nonstandard imaging methods in electron microscopy. https://doi.org/10.1016/S0304-3991(76)91538-2