There are several important schema changes to improve TDAQ configuration description:

All such changes have been discussed during 2005 TDAQ Workshop in Mainz (click link for more information).

Template Applications

In the past it was necessary to create one configuration database object per one process running online. The total number of such processes including HLT processing tasks may reach in the final system 10,000. However most of such applications have very similar data values and only differ by values of few attributes (usually by name). The primary idea is to describe similar applications by single object and put little knowledge into DAL algorithms to calculate unique attribute values. In this case we will gain storage space and performance (less space, faster read, easier readable and editable).

The proposal document is available from EDMS: https://edms.cern.ch/document/684859. There are several extensions created after document publish date, which are listed below.

How to get all applications with data collection parameters

The Partition::get_all_applications() algorithm is implemented for C++ and Java DAL and it should be used instead of old Partition::get_applications() one, that knows nothing about template applications.

The C++ algorithm's prototype can be found in generated dal/Partition.h file:

void get_all_applications(std::vector<daq::core::AppConfig>& out, Configuration& db,
                          std::set<std::string> * app_types = 0,
                          std::set<const Segment *> * use_segments = 0,
                          std::set<const Computer *> * use_hosts = 0) const

where:

Above optional parameters can be combined.

To test get_all_applications() algorithm work there is dedicated binary dal_dump_app_config:

Usage: dal_dump_app_config [-d database-name] -p partition [-t [types ...]] [-c [ids ...]] [-s [ids ...]]
Options/Arguments:
  --data | -d  database-name    name of the database (ignore TDAQ_DB variable)
  --partition-name | -p partition    name of the partition object
  --application-types | -t types    filter out all applications except given classes (and their subclasses)
  --hosts | -c ids    filter out all applications except those which run on given hosts
  --segments | -s ids    filter out all applications except those which belong to given segments

Example below shows how to get all template and run control applications on host lxplus001 for a test partition:

bash$ dal_dump_app_config -d oksconfig:daq/partitions/lxplus_template_tests.data.xml \
      -p lxplus-test -c lxplus001 -t TemplateApplication RunControlApplication
===================================================================================================================================
| num | Application Object        | Host               | Segment             | MP:id | Application unique id                      |
===================================================================================================================================
|   1 | test-L2@L2PUTemplateApp   | lxplus001@Computer | 01-19@HLTSegment    | 32767 | test-L2@L2PUTemplateApp:01-19:lxplus001:0  |
|   2 | test-L2@L2PUTemplateApp   | lxplus001@Computer | 01-19@HLTSegment    | 32766 | test-L2@L2PUTemplateApp:01-19:lxplus001:1  |
|   3 | test-L2@L2PUTemplateApp   | lxplus001@Computer | 01-19@HLTSegment    | 32765 | test-L2@L2PUTemplateApp:01-19:lxplus001:2  |
|   4 | test-L2@L2PUTemplateApp   | lxplus001@Computer | 01-19@HLTSegment    | 32764 | test-L2@L2PUTemplateApp:01-19:lxplus001:3  |
|   5 | test-L2@L2PUTemplateApp   | lxplus001@Computer | 01-19@HLTSegment    | 32763 | test-L2@L2PUTemplateApp:01-19:lxplus001:4  |
|   6 | test-L2@L2PUTemplateApp   | lxplus001@Computer | 01-19@HLTSegment    | 32762 | test-L2@L2PUTemplateApp:01-19:lxplus001:5  |
|   7 | test-L2@L2PUTemplateApp   | lxplus001@Computer | 01-19@HLTSegment    | 32761 | test-L2@L2PUTemplateApp:01-19:lxplus001:6  |
|   8 | test-L2@L2PUTemplateApp   | lxplus001@Computer | 01-19@HLTSegment    | 32760 | test-L2@L2PUTemplateApp:01-19:lxplus001:7  |
|   9 | test-L2@L2PUTemplateApp   | lxplus001@Computer | 01-19@HLTSegment    | 32759 | test-L2@L2PUTemplateApp:01-19:lxplus001:8  |
|  10 | test-L2@L2PUTemplateApp   | lxplus001@Computer | 01-19@HLTSegment    | 32758 | test-L2@L2PUTemplateApp:01-19:lxplus001:9  |
|  11 | RC@RunControlApplication  | lxplus001@Computer | setup@OnlineSegment |     0 | RC@RunControlApplication                   |
|  12 | 19@RunControlApplication  | lxplus001@Computer | 01-19@HLTSegment    |     0 | 19@RunControlApplication                   |
|  13 | efd@RunControlApplication | lxplus001@Computer | efd_sf@EF_SubFarm   |     0 | efd@RunControlApplication                  |
===================================================================================================================================
Message Passing Node ID Calculation

The dal package implements algorithms for the Message Passing Node ID calculations and replaces obsolete Application package. If you are using Application package, do the following:

The DAL's algorithm supports different ranges for node 16-bits IDs depending on application type instead of fixed 1K range supported by obsolete Application package as shown below:

Application Type   Number of instances   Node ID Ranges 
PT   32K   [32K .. 64K[ 
L2PU   8K   [24K .. 32K[ 
SFI   4K   [20K .. 24K[ 
ROD   2K   [18K .. 20K[ 
ROB   2K   [16K .. 18K[ 
EFD   1K   [15K .. 16K[ 
SFO   1K   [14K .. 15K[ 
DFM   1K   [13K .. 14K[ 
L2SV   1K   [12K .. 13K[ 
ROS   1K   [11K .. 12K[ 
ROIB   1K   [10K .. 11K[ 
L2RH   1K   [09K .. 10K[ 
unknown / not used   9K   [00K .. 09K[ 

The node IDs of template applications go down from maximum range values; the IDs of the non-template applications of the same type are growing from minimal values to maximum. For correct node message passing node ID calculation a database class is considered to be a template class describing PT, L2PU or SFI, if it's name contains "Template" literal with one of the corresponding literals "PT", "L2PU" or "SFI", e.g. classes with names L2PUTemplateApplication or TemplateL2PUApp are considered as classes describing template L2PU applications. The attribute TemplateApplication::Type described by the proposal document is replaced by above convention.

External Packages

There are several dependencies of the ATLAS online software from external packages having non-CMT structure. To describe paths to shared libraries and binaries mapped on CMT platforms and platform-dependent environment variables it was proposed (see EDMS document https://edms.cern.ch/document/732735) to introduce several new configuration classes:

The Variable class has been extended. There is new 0..N relationship TagValues pointing to the TagMapping object. In the context of an application the DAL algorithms calculating value of variable checks platform for application and if explicitly defined via TagValues relationship, the algorithm takes variable value from the Value attribute of the TagMapping object.

See also graphical view of above classes in the dal/data/schema/views/SoftwareRepository.view.ps file.

The SW_ExternalPackage objects, related TagMapping and Variable objects are automatically generated by the dal_create_sw_repository utility running in the DAQRelease. For external packages it uses the following data file syntax:

macro sw.external.package.X:installation.path 'inst-path'  # X - any external package, inst_path - root directory of it's installation
macro sw.external.package.X:cmt.tag cmt-tag                # cmt tag for given platform, e.g. i686-slc3-gcc323-dbg
macro sw.external.package.X:lib.mapping shared-lib-mapping # path to libraries from package root, e.g. slc3_ia32_gcc323/lib
macro sw.external.package.X:bin.mapping binary-mapping     # path to binaries from package root, e.g. slc3_ia32_gcc323/bin
macro sw.external.package.X:uses Y                         # Y - sw package, which is used by package X
macro sw.external.package.X:uses Z                         # Z - another sw package, which is used by package X
macro sw.external.package.X:needs.environment 'VAR1'       # package X needs process environment variable VAR1
macro sw.external.package.X:needs.environment 'VAR2'       # package X needs another process environment variable VAR2
macro sw.environment.variable.VAR3:value 'FOO'             # value of environment variable VAR3, e.g. '${X}/slc3_ia32_gcc323/root'

Above data file is generated for each cmt platform during DAQRelease build and is passed to the dal_create_sw_repository utility.

Example of generated COOL Description

Below there are several generated interlinked config objects, that allow to build correct run-time environment for online application using COOL package.

The COOL package description:

Object COOL@SW_ExternalPackage
Name   CMT external package COOL
InstallationPath   ${COOL_HOME}
Uses   XercesC@SW_ExternalPackage
SEAL@SW_ExternalPackage
CORAL@SW_ExternalPackage
POOL@SW_ExternalPackage
Binaries   i686-slc3-gcc323-opt-for-slc3_ia32_gcc323/bin@TagMapping
i686-slc3-gcc323-dbg-for-slc3_ia32_gcc323/bin@TagMapping
i686-slc3-gcc344-opt-for-slc3_ia32_gcc344/bin@TagMapping
i686-slc3-gcc344-dbg-for-slc3_ia32_gcc344/bin@TagMapping
SharedLibraries   i686-slc3-gcc323-opt-for-slc3_ia32_gcc323/lib@TagMapping
i686-slc3-gcc323-dbg-for-slc3_ia32_gcc323/lib@TagMapping
i686-slc3-gcc344-opt-for-slc3_ia32_gcc344/lib@TagMapping
i686-slc3-gcc344-dbg-for-slc3_ia32_gcc344/lib@TagMapping

SEAL is one of the packages used by COOL:

Object SEAL@SW_ExternalPackage
Name   CMT external package SEAL
InstallationPath   ${SEAL_HOME}
Uses   XercesC@SW_ExternalPackage
ProcessEnvironment   SEAL_KEEP_MODULES@Variable
SEAL_PLUGINS@Variable
SEAL_CONFIGURATION_FILE@Variable
Binaries    ... as in COOL ...
SharedLibraries    ... as in COOL ...

The SEAL_PLUGINS is one of the variables used by SEAL. It is multi-value environment variable variable, which value depends on platform:

Object "SEAL_PLUGINS@Variable"
Name   SEAL_PLUGINS
Value   /afs/cern.ch/sw/lcg/app/releases/COOL/COOL_1_3_1/slc3_ia32_gcc323/lib/modules:
/afs/cern.ch/sw/lcg/app/releases/CORAL/CORAL_1_4_2/slc3_ia32_gcc323/lib/modules:
/afs/cern.ch/sw/lcg/app/releases/POOL/POOL_2_4_1/slc3_ia32_gcc323/lib/modules:
/afs/cern.ch/sw/lcg/app/releases/SEAL/SEAL_1_8_1/slc3_ia32_gcc323/lib/modules
TagValues   i686-slc3-gcc344-opt-to-SEAL_PLUGINS@TagMapping  # points on specific value for gcc344-opt
i686-slc3-gcc344-dbg-to-SEAL_PLUGINS@TagMapping  # points on specific value for gcc344-dbg

For a binary, that belongs to the TDAQ sw repository (that in turn uses COOL) the run-time environment will have all required variables and paths set, e.g.:

bash$ dal_dump_apps -d oksconfig:daq/partitions/be_test.data.xml -p be_test -a oks2cool -s
Dump 1 non-template applications:
### application 'oks2cool@ResourceApplication' ###
- possible program file names:
  * /afs/cern.ch/atlas/project/tdaq/cmt/pre-01-06-00/installed/i686-slc3-gcc323-opt/bin/oks2cool
- environment variables:
  * ROOTSYS="/afs/cern.ch/sw/lcg/external/root/5.10.00c/slc3_ia32_gcc323/root"
  * SEAL_CONFIGURATION_FILE="/afs/cern.ch/sw/lcg/app/releases/COOL/COOL_1_3_1/src/RelationalCool/tests/seal.opts"
  * SEAL_KEEP_MODULES="true"
  * SEAL_PLUGINS="/afs/cern.ch/sw/lcg/app/releases/COOL/COOL_1_3_1/slc3_ia32_gcc323/lib/modules:
                  /afs/cern.ch/sw/lcg/app/releases/CORAL/CORAL_1_4_2/slc3_ia32_gcc323/lib/modules:
                  /afs/cern.ch/sw/lcg/app/releases/POOL/POOL_2_4_1/slc3_ia32_gcc323/lib/modules:
                  /afs/cern.ch/sw/lcg/app/releases/SEAL/SEAL_1_8_1/slc3_ia32_gcc323/lib/modules"
  * TDAQ_APPLICATION_OBJECT_ID="oks2cool@ResourceApplication"
  # ... skip the rest ...
- binary search paths:  # i.e. what goes into PATH variable
  * /afs/cern.ch/atlas/project/tdaq/cmt/pre-01-06-00/installed/share/bin
  * /afs/cern.ch/atlas/project/tdaq/cmt/pre-01-06-00/installed/i686-slc3-gcc323-opt/bin
  * /afs/cern.ch/atlas/project/tdaq/cmt/pre-01-06-00/external/share/bin
  * /afs/cern.ch/atlas/project/tdaq/cmt/pre-01-06-00/external/i686-slc3-gcc323-opt/bin
  * /afs/cern.ch/atlas/project/tdaq/DataFlow/releases/tdaq-common-01-04-00/installed/share/bin
  * /afs/cern.ch/atlas/project/tdaq/DataFlow/releases/tdaq-common-01-04-00/installed/i686-slc3-gcc323-opt/bin
  * /afs/cern.ch/atlas/project/tdaq/DataFlow/releases/tdaq-common-01-04-00/external/share/bin
  * /afs/cern.ch/atlas/project/tdaq/DataFlow/releases/tdaq-common-01-04-00/external/i686-slc3-gcc323-opt/bin
  * /afs/cern.ch/sw/lcg/external/Boost/1.33.1/slc3_ia32_gcc323/bin/
  * /afs/cern.ch/sw/lcg/app/releases/COOL/COOL_1_3_1/slc3_ia32_gcc323/bin/
  * /afs/cern.ch/sw/lcg/external/XercesC/2.7.0/slc3_ia32_gcc323/bin/
  * /afs/cern.ch/sw/lcg/app/releases/SEAL/SEAL_1_8_1/slc3_ia32_gcc323/bin/
  * /afs/cern.ch/sw/lcg/app/releases/CORAL/CORAL_1_4_2/slc3_ia32_gcc323/bin/
  * /afs/cern.ch/sw/lcg/app/releases/POOL/POOL_2_4_1/slc3_ia32_gcc323/bin/
  * /afs/cern.ch/sw/lcg/external/root/5.10.00c/slc3_ia32_gcc323/bin/
  * /afs/cern.ch/sw/lcg/external/oracle/10.2.0.1/slc3_ia32_gcc323/bin/
  * /afs/cern.ch/sw/lcg/external/mysql/5.0.18/slc3_ia32_gcc323/bin/
  * /afs/cern.ch/sw/lcg/external/uuid/1.38/slc3_ia32_gcc323/bin/
  * /afs/cern.ch/sw/lcg/external/pcre/4.4/slc3_ia32_gcc323/bin/
- paths to shared libraries:  # i.e. what goes into LD_LIBRARY_PATH variable
  * /afs/cern.ch/atlas/project/tdaq/cmt/pre-01-06-00/installed/i686-slc3-gcc323-opt/lib
  * /afs/cern.ch/atlas/project/tdaq/cmt/pre-01-06-00/external/i686-slc3-gcc323-opt/lib
  * /afs/cern.ch/atlas/project/tdaq/DataFlow/releases/tdaq-common-01-04-00/installed/i686-slc3-gcc323-opt/lib
  * /afs/cern.ch/atlas/project/tdaq/DataFlow/releases/tdaq-common-01-04-00/external/i686-slc3-gcc323-opt/lib
  * /afs/cern.ch/sw/lcg/external/Boost/1.33.1/slc3_ia32_gcc323/lib/
  * /afs/cern.ch/sw/lcg/app/releases/COOL/COOL_1_3_1/slc3_ia32_gcc323/lib/
  * /afs/cern.ch/sw/lcg/external/XercesC/2.7.0/slc3_ia32_gcc323/lib/
  * /afs/cern.ch/sw/lcg/app/releases/SEAL/SEAL_1_8_1/slc3_ia32_gcc323/lib/
  * /afs/cern.ch/sw/lcg/app/releases/CORAL/CORAL_1_4_2/slc3_ia32_gcc323/lib/
  * /afs/cern.ch/sw/lcg/app/releases/POOL/POOL_2_4_1/slc3_ia32_gcc323/lib/
  * /afs/cern.ch/sw/lcg/external/root/5.10.00c/slc3_ia32_gcc323/lib/
  * /afs/cern.ch/sw/lcg/external/oracle/10.2.0.1/slc3_ia32_gcc323/lib/
  * /afs/cern.ch/sw/lcg/external/mysql/5.0.18/slc3_ia32_gcc323/lib/
  * /afs/cern.ch/sw/lcg/external/uuid/1.38/slc3_ia32_gcc323/lib/
  * /afs/cern.ch/sw/lcg/external/pcre/4.4/slc3_ia32_gcc323/lib/

Note, the PATH and LD_LIBRARY_PATH are fully constructed from used sw packages and partition's repository root. They cannot be defined by any other mean (e.g. creating in database explicit LD_LIBRARY_PATH variable).

Resource Set

Backup hosts