An Effective Approach for Parsing Large Log Files ( Datasets , experiments results , code)

Because of their contribution to the overall reliability assurance process, software logs have become important in the analysis of software systems. Logs are often the only data points that demonstrate how software behaves in operation. Unfortunately, logs are often unstructured, making analysis difficult. There has been many studies that aim to automatically parse large log files. The primary goal is to create templates from log data samples that can later be used to recognize future logs. In this paper, we propose ULP, a Unified Log Parsing tool, which is highly accurate and efficient. ULP is built on the idea of regional frequency analysis. First, log events are organized into groups using a text processing method. Frequency analysis is then applied locally to instances of the same group to identify static and dynamic content of log events. When applied to 10 log datasets of the the LogPai project, ULP achieves an average accuracy of 89.2%, which outperforms the accuracy of four leading log parsing tools, namely Drain, Logram, SPELL and
AEL. Additionally, ULP can parse up to a 4 million log events in less than 3 minutes. ULP is available online as an open source. It can be used by practitioners and researchers to parse effectively and efficiently large log files so as to support log analysis tasks.