Orchestration and management of data generated by big-data electron microscopy instruments: A Discovery report

For this report, representatives from academic facilities in Australia and overseas that operated, or were planning to operate, electron microscopy and correlative light–electron microscopy instruments that produced large volumes of data, were interviewed. The interviews aimed to collect information on how the facilities had set up their data workflows for data capture, data transfer, data movement, data storage and overall data orchestration, which tools they used for data processing (including the supporting infrastructure), and how data and metadata were managed. Based on this survey, the report provides a review of the informatics and data-management landscape at Australian facilities, including tools, methods and procedures currently in use or commented on during interviews. Furthermore, the report proposes recommendations to the stakeholders of the Australian Characterisation Commons at Scale (ACCS) project and its Work Package 4 for tools, methods and procedures that have been considered interesting, promising or relevant to examine further over the first year of Work Package 4 (2021), and guide ACCS work packages into the future.

This work was undertaken under the Australian Characterisation Commons at Scale (ACCS) project, in particular under Work Package 4: “Big-data electron and correlative microscopy from instrument to publication”.