The DAQ Debugger for iFDAQ of the COMPASS Experiment

In general, state-of-the-art Data Acquisition Systems
(DAQ) in high energy physics experiments must satisfy high
requirements in terms of reliability, efficiency and data rate capability.
This paper presents the development and deployment of a debugging
tool named DAQ Debugger for the intelligent, FPGA-based Data
Acquisition System (iFDAQ) of the COMPASS experiment at CERN.
Utilizing a hardware event builder, the iFDAQ is designed to be
able to readout data at the average maximum rate of 1.5 GB/s of
the experiment. In complex softwares, such as the iFDAQ, having
thousands of lines of code, the debugging process is absolutely
essential to reveal all software issues. Unfortunately, conventional
debugging of the iFDAQ is not possible during the real data taking.
The DAQ Debugger is a tool for identifying a problem, isolating
the source of the problem, and then either correcting the problem
or determining a way to work around it. It provides the layer
for an easy integration to any process and has no impact on the
process performance. Based on handling of system signals, the
DAQ Debugger represents an alternative to conventional debuggers
provided by most integrated development environments. Whenever
problem occurs, it generates reports containing all necessary
information important for a deeper investigation and analysis. The
DAQ Debugger was fully incorporated to all processes in the iFDAQ
during the run 2016. It helped to reveal remaining software issues
and improved significantly the stability of the system in comparison
with the previous run. In the paper, we present the DAQ Debugger
from several insights and discuss it in a detailed way.