Eventdisplay: an Analysis and Reconstruction Package for VERITAS

Eventdisplay v490 is a significant update with many changes to the code compared to previous versions.

This is the first release of v490 tested for V6 analysis. Eventdisplay v490 is ready for publication-type analysis

The default analysis methods for v490 are:

• AP cleaning (new default in scripts)
• dispBDT direction reconstruction
• gamma/hadron separation using BDTs (for reduced high-voltage operations: soft box cuts)

The introduction of these methods provide a significant improvement in reconstruction quality and sensitivity, see the comparison with v487 for several targets.

The usage of Eventdisplay requires code, scripts, and analysis files. For this release, the corresponding repositories are:

• Eventdisplay v490.0
• Eventdisplay_AnalysisScripts_VTS v490.0
• Eventdisplay_AnalysisFiles_VTS v490.0

For details on how to install the release, see the this readme.

Main changes.

Eventdisplay v490 adds significant improvements to the analysis

New analysis methods:

• AP cleaning (image/border cleaning with improved afterpulsing robustness)
• optimised next-neighbour cleaning (NN cleaning; improved low-energy threshold)
• dispBDT direction reconstruction (improved angular resolution at low elevations / large offsets)
• improvements for DL3 / gammapy interface
• improved gamma/hadron separation code (including critical bug fixes)

Code maintenance:

• code cleanup and removal of lot of obsolete code (EventDisplay_v4)
• parameter files cleanup Eventdisplay_AnalysisFiles_VTS)
• scripts improvements (Eventdisplay_AnalysisScripts_VTS)

Detailed release notes: Afterpulsing cleaning (image/border cleaning with improved robustness)

• removes islands consisting of a single image and a single border pixel
• based on Jamie's suggestion presented in this VERITAS memo
• see VERITAS memo on AP cleaning in Eventdisplay
• implementation of median and pedestal 68% containment for testing of image/border cleaning settings. These values are calculated and written to the pedestal output file, but not used in the analysis.
• AP cleaning is the new default cleaning method
• switch this on by adding one parameter 1 to * -1 IMAGECLEANINGTHRESHOLDS 5. 2.5 1 in the runparameter file
• use runparameter file EVNDISP.reconstruction.runparameter.AP.v4x

Optimised next-neighbour cleaning for improved low-energy reconstruction

• based on M.Shayduk et al, arXiv:1307.4939
• for VERITAS implementation, see Maria's presentations (e.g. (Winter 2023 collaboration meeting)[https://veritas.sao.arizona.edu/GammaWiki/images/5/54/VCMWinter2023_NN_Kherlakian.pdf]
• use runparameter file EVNDISP.reconstruction.runparameter.AP.v4x
• integration deals with short runs (~5 min long) with low number of events required for IPR Graphs by averaging over all four telescopes
• note that this requires to a separate evndisp analysis run, dedicated lookup tables and instrument response functions.
• significant (factor x2!) improvement in run time

Epoch and throughput definitions

• Epochs are now defined in terms of dates and not run numbers (see VERITAS.Epochs.runparameter)
• Remove duplication of date/MJD in season definition. Having both MJD and SQL-strings lead to errors in definitions of yearly seasons. Removed the MJD, which is now calculated in VEvndispRunParameter from the sql string.

BDT gamma/hadron separation

• vastly improved efficiency for BDT training
• significantly improved cut optimisation with consistent background rate calculation
• use MC-generated effective areas for signal and any set of background runs for background calculation (before: restriction to Crab Nebula runs); this increases significantly the available statistics
• overlapping energy bins for training; this removes steps in effective areas

Overwrite of runinfo DB values

• Command line parameters to overwrite target runinfo DB values: Add possibility to overwrite the DB settings for target information (read usually from the runinfo db table): -overwritedb_rightascension, -overwritedb_declination, -overwritedb_target (this complements the existing command line options for the wobble offset: overwritedb_wobblenorth and overwritedb_wobbleeast)

evndisp robustness against missing events ncreased robustness for reading number of samples of readout window from VBF files. This recovers those runs, where for some reason the more than 10,000 events in the beginning of the run are empty, but the rest of the run is ok. New dispBDT direction reconstruction algorithms for improved direction reconstruction at low zenith angles and large offsets

• not available in this beta.1 release
• see VERITAS memo on dispBDTs
• dispBDT reconstruction is integrated in the mscw_energy binary, meaning that to redo the reconstruction a re-run of evndisp is not necessary.
• improved gamma/hadron separation:
• significant improvement to efficiency of trainTMVAforGammaHadronSeparation with a large reduction of the memory footprint (which was beyond 20 GB) and improvements in execution speed
• add Size2ndMax and NImages as training parameters (improved signal/background efficiency, see ELOG 2357)
• write a temporary root file with events after signal/background pre-cuts is written to disk and used in a second step for the training (TMVA unfortunately reads in all events before cuts)
• a random fraction of events is selected from each input file to match the number of training events requested
• if there are less events, the training option string is adjusted and the training proceeds with a lower number of events than requested (!!)

Support for DL3 export using V2DL3 and gammapy

• see VERITAS memo by Sonal
  • added propagation of VPM info to DL3 level (allows to select runs with / without VPM monitoring data)

Improved MC / Data comparison with additional plots for theta2 and disp

• weighting of events according to the azimuth of the data runs
• Improvements in plotting of instrument response functions
• re-enable plotting of instrument response functions vs true energy: Effective area files contain since v497 two trees: fEffArea (with reconstructed energy axis) and fEffAreaH2F (with true energy axis). Change reading routines to re-enable plotting of IRFs vs true energy.
• improve angular resolution plotting and fix issue with plotting cumulative theta2 distributions.
• improved reading and generation of energy thresholds and lookup table plots

Evndisp trace integration

• performance updates in trace integration and TZero calculation

Lookup tables

• Energy axes for energy reconstruction lookup tables
• removed dynamic setting of energy axes limits (1D histogram for tables) in lookup table code and fixed it to 150 TeV (well beyond what we do in VERITAS). Dynamic settings were introduced for CTA (telescope types), but the version in Eventdisplay_v4 had a minor bug
• more flexibility to find correct NSB bins when combing lookup tables; add command line parameters noise tolerance in combineEffectiveAreas
• Remove obsolete TableEnergy filling

Addition of cluster cleaning algorithms

• use EVNDISP.reconstruction.runparameter.CC.v4x
• not part of the standard analysis (but feel free to test)

Automatic container generation

• integration of some automatic docker container generation with each release (see dockerfiles directory
• newest images are available for each release from here: https://github.com/VERITAS-Observatory/EventDisplay_v4/pkgs/container/eventdisplay_v4
• (this needs a bit further testing)

Code maintenance

• improved code documentation
• use consistently now a code linter
  • requires installation of the unix tool astyle. Run make formatSourceCode before submitting any code
• removal of lots of obsolete and not maintained code
  • CTA related code (use this Eventdisplay for CTA Analysis)
  • frogs and Model3d (code very much out of date)
• cleanup of compilation warnings and C++17
  • General cleanup of compilation warnings and changes to allow for C++ 14 and C++ 17compilation.
  • test this by CI matrix on Github (which consist of testing compilation and linking on root:6.24.06-centos7, root:6.22.08-ubuntu20.04, root:6.26.06-ubuntu22.04, root:6.26.06-fedora35, root:latest)

Image/border pixel writer

• writing of image/border pixel lists on evndisp stage (optional, very useful debugging of cleaning studies)

  • Option -writeimagepixellist to evndisp allows to write out list of pixel (with integrated charge and timing) surviving the image cleaning.
  • This option increases the output files of evndisp by approximately a factor 2.5-3. It is therefore switched off by default.
  • Written are image, border pixels, and one additional ring of pixels around them:

    PixelListN : PixelListN/i
    PixelID   : PixelID[PixelListN]/i 
    PixelType : PixelType[PixelListN]/i 
    PixelIntensity : PixelIntensity[PixelListN]/F
    PixelTimingT0 : PixelTimingT0[PixelListN]/F
    

    PixelType indicates the following:

• PixelType == 1: image pixel

• PixelType == 2: border pixel
• PixelType == 3: neighbour pixel to image/border

DB text functionality

• Use DB text files instead of direct query to VERITAS data base
  • allows to download information from VERITAS DB (e.g.,run data, VPM data, etc) and re-use it
  • avoids overloading the database connection when analysing hundreds of runs simultaneously
  • allows to use a basic root docker image for compilation and no need of integration of mysql while compiling Eventdisplay
  • requires reading of DB information using the scripts available in the db_scripts directory of Eventdisplay_AnalysisScripts_VTS
  • used automatically by ANALYSIS.evndisp.sh when a run directory is found in ${VERITAS_DATA_DIR}/DBTEXT

Sofa replacement of slalib

• Replacement of slalib routines by sofa for all coordinate transformations
  • see PR #180
  • Eventdisplay has been using a copy of the slalib code since the very beginning (VASlalib). All astronometry routines are replaced by the sofa package (same authors as slalib; but permissive licence) for the following reasons:
    • slalib (as used in Eventdisplay) is not maintained since many years and the status of the routines is unclear (e.g. means that timing routines do not include leap seconds introduced recently)
    • slalib's licence unclear and very likely the current usage Eventdisplay is not allowed
    • sofa is a well maintained IAU package
  • #PR 180 lists some tests listing differences between using the two different astronometry packages: generally consistent results with max differences in reconstructions observations of <2 arcseconds.
  • Note that this involves the installation of the sofa package before compiling eventdisplay. Use the script install_sofa.sh to download and install the package.

mergeVBF

• addition of mergeVBF binary
  • mergeVBF is used for the simulation production to merge several (possibly 1000s) of simulation vbf files into one file.

Readout window

Increased robustness for reading number of samples of readout window from VBF files. This recovers those runs, where for some reason the more than 10,000 events in the beginning of the run are empty, but the rest of the run is ok.