2. Installation

2.1. Activating pre-installed DQ2 clients
2.1.1. CERN/lxplus
2.1.2. US
2.1.3. NDGF
2.2. External Dependencies
2.3. Installation Alternatives
2.4. Install DQ2 clients with pacman
2.5. Install DQ2 clients with pacman - special US/OSG notes
2.6. Install DQ2 clients with RPM
2.6.1. RPM-based installation with APT Library
2.6.2. APT Repository
2.6.3. Installation of Packages
2.6.4. DQ2 Post-Installation

Abstract

This section presents the installation instructions.

In the end of this section you should be able to:

  • Activate pre-installed DQ2 clients.

  • Install the DQ2 clients with pacman (the preferred way).

  • Install the DQ2 clients with RPMs.

2.1. Activating pre-installed DQ2 clients

2.1.1. CERN/lxplus

  1. start with a clean shell on lxplus.cern.ch

  2. source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh

  3. voms-proxy-init -voms atlas

Please take note that the setup.sh script sets up the grid environment in a way to work with the ATLAS software release! It is highly important to notice that only ATLAS releases 14.x.x and above are supported, since 13.x.x releases clash with the grid environment.

There is also a script setup_previous.sh which will setup the previous release, and not the current stable - should it be needed.

2.1.2. US

  1. start with a clean shell

  2. source /afs/usatlas.bnl.gov/osg/client/current/setup.[c]sh

  3. source /afs/usatlas.bnl.gov/Grid/Don-Quijote/DQ2Clients/setup.[c]sh

  4. export DQ2_LOCAL_SITE_ID=[YOUR SITE ID]

  5. voms-proxy-init -voms atlas

Please select your appropriate DQ2 LOCAL SITE ID. It is recommended to explicitly put the site id of the user area that has been assigned to you, like SWT2_CPB_USERDISK. If you do not know what to put there, please ask your site administrator. If you only want to use dq2-get at BNL, you can simply put BNL.

If you have any more questions regarding the US setup, you can also check https://www.racf.bnl.gov/experiments/usatlas/griduser

2.1.3. NDGF

Look here for future updates.

2.2. External Dependencies

DQ2 requires a set of external dependencies which vary depending on the actions requested by the user. If you install by using pacman or RPMs you have the guarantee that all external dependencies will be kept up to date. You may use the tarball based installation, in which case you will need to ensure the required dependencies are already available in the machine.

The communication with the central dataset catalogues require either CURL or Python-CURL (details below) and may require Grid Security Infrastructure (GSI) libraries installed, in case the user is writing into the catalogues.

The complete set of dependencies is shown below:

Table 1. External dependencies table

Dependency nameRole 
Python CURLProvides the Python bindings for the 'curl' command for HTTP access. 
Globus/VDTProvides the Grid security infrastructure 

2.3. Installation Alternatives

There are two installation alternatives for the DQ2 client tools:

  • pacman-based installation. If you do not have root privileges on your machine, you may use a pacman-based installation. This is the recommended procedure for non-root users. pacman facilitates the installation and resolution of external dependencies as well as updating the installation whenever DQ2 is updated by a simple command. All required dependencies are injected with this installation method.

  • RPM-based installation. If you have root privileges on your machine, this is the recommended procedure. RPMs facilitates the installation and resolution of external dependencies as well as updating the installation whenever DQ2 is updated by a simple command. All required dependencies are injected with this installation method.

[Warning]Warning

If you are installing DQ2 Client in a machine shared by multiple users (similar to Grid User Interfaces), a centralised RPM-based installation is highly recommended.

2.4. Install DQ2 clients with pacman

Installing with pacman is the preferred way. This guide details installing pacman and the DQ2 clients. Please take note that installing the DQ2 clients should be done by your site administrator in a central way.

curl -O http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-latest.tar.gz
tar xfz pacman-latest.tar.gz
rm pacman-latest.tar.gz
cd pacman-3.25
source setup.sh

Now we are ready to install the DQ2 clients.

cd ..
rm -rf DQ2Clients
mkdir DQ2Clients
cd DQ2Clients
pacman -trust-all-caches -allow tar-overwrite -get http://atlas.web.cern.ch/Atlas/GROUPS/DATABASE/project/ddm/releases/pacman/cache:DQ2Clients
source setup.sh
cd ..

Now you must set your DQ2_LOCAL_SITE_ID. This corresponds to the your site in TiersOfATLAS. For example, if you are in Innsbruck, you put:

export DQ2_LOCAL_SITE_ID=HEPHY-UIBK

Now, every time you want to activate the DQ2 Clients, you simply source the setup.sh in you DQ2Clients directory.

If you want to upgrade the clients, check this webpage for the latest release: https://twiki.cern.ch/twiki/bin/view/Atlas/DQ2Clients and then execute (in the directory where you installed the tools before):

pacman -get "http://atlas.web.cern.ch/Atlas/GROUPS/DATABASE/project/ddm/releases/pacman/cache:DQ2Clients | version('version number')

For example:

pacman -get "http://atlas.web.cern.ch/Atlas/GROUPS/DATABASE/project/ddm/releases/pacman/cache:DQ2Clients | version('0.1.13')

It is recommended to export the variable DQ2_LOCAL_SITE_ID to your environment. Please take note that your site may have defined multiple DQ2 sites. In this case please make very sure, that you select the correct one (it might need to be space-token enabled), if you intend to create new datasets. Please ask your site-administrator if you are unsure and don't know what this means!

2.5. Install DQ2 clients with pacman - special US/OSG notes

There are two pacman packages in OSG that are guaranteed to work and where support is provided. These are the packages OSG:wn and wlcg-client. OSG:wn is the package for the pilot and the different binaries, while the wlcg-client are the clients for interactive use. DQ2Client requires one of these 3: wn-client (=OSG:wn) or wlcg-client or gLiteUI. Other installations (from OSG, custom from VDT picking your packages or somwhere else) may give you grid functionalities but are not guaranteed to work and provide all capabilities. For more information, please see https://www.usatlas.bnl.gov/twiki/bin/view/Admins/WlcgClient

2.6. Install DQ2 clients with RPM

2.6. Install DQ2 clients with RPM

2.6.1. RPM-based installation with APT Library

To install the software using RPMs, the first step is to setup APT. APT is a set of command line tools for handling installation of packages, which includes resolution and installation of package dependencies. The DQ2 software provides its own APT repository.

There are 3 APT repositories for DQ2 itself:

  • Stable: the stable version of DDM components. This is the repository you should use.

  • Unstable: an unstable version of DDM components. This typically contains new functionality which is under testing. It should only be used by experts and those involved in early DDM beta-testing.

  • Nightly: a set of RPMs generated on a nightly-basis. This repository is typically not usable and is only for internal comsumption by the developers.

Additionally, there is another APT repository for gLite externals, which includes gLite FTS, LFC file catalogue clients and Globus RLS clients, as well as VDT/Globus.

The first step is to check whether the APT library clients are available on your machine. If not, install the them, as root.

Example 1. APT Install Help

$ su -
> `which` apt-get
> up2date apt-get 1
> apt-get
apt 0.5.15cnc7 for linux i386 compiled on Aug 23 2005 16:04:14
Usage: apt-get [options] command
       apt-get [options] install|remove pkg1 [pkg2 ...]
       apt-get [options] source pkg1 [pkg2 ...]

apt-get is a simple command line interface for downloading and
installing packages. The most frequently used commands are update
and install.

Commands:
   update - Retrieve new lists of packages
   upgrade - Perform an upgrade
   install - Install new packages (pkg is libc6 not libc6.rpm)
   remove - Remove packages
   source - Download source archives
   build-dep - Configure build-dependencies for source packages
   dist-upgrade - Distribution upgrade, see apt-get(8)
   clean - Erase downloaded archive files
   autoclean - Erase old downloaded archive files
   check - Verify that there are no broken dependencies

Options:
  -h  This help text.
  -q  Loggable output - no progress indicator
  -qq No output except for errors
  -d  Download only - do NOT install or unpack archives
  -s  No-act. Perform ordering simulation
  -y  Assume Yes to all queries and do not prompt
  -f  Attempt to continue if the integrity check fails
  -m  Attempt to continue if archives are unlocatable
  -u  Show a list of upgraded packages as well
  -b  Build the source package after fetching it
  -D  When removing packages, remove dependencies as possible
  -V  Show verbose version numbers
  -c=? Read this configuration file
  -o=? Set an arbitary configuration option, eg -o dir::cache=/tmp
See the apt-get(8), sources.list(5) and apt.conf(5) manual
pages for more information and options.
                       This APT has Super Cow Powers.
                    
1

Up2date is a RedHat tool to install external packages. You may instead download apt-get for your distribution's RPM repository.

2.6.2. APT Repository

After having setup apt-get, you need to configure it to fetch RPMs from the repositories. You will need to setup two repositories: one for DQ2, another one for Grid externals (in case you need them).

To setup DQ2 repository:

Example 2. DQ2 APT Repository Configuration

> vi /etc/apt/sources.list.d/dq2.list
> cat /etc/apt/sources.list.d/dq2.list
rpm http://ddm-build.cern.ch ddm/apt/el4-32 stable external 1
rpm http://dashb-build.cern.ch apt/el4-32 stable
> vi /etc/apt/preferences
> cat /etc/apt/preferences
Package: *
Pin: origin ddm-build.cern.ch
Pin-Priority: 1000 2
                    
1

Note we are using the Enterprise Linux 4, 32-bit stable repository.

2

We must set the Pin-priority so that DQ2 external dependencies override system RPMs if necessary.

To setup gLite repository:

Example 3. gLite APT Repository Configuration

> vi /etc/apt/sources.list.d/lcg.list
> cat /etc/apt/sources.list.d/lcg.list
rpm http://glitesoft.cern.ch/EGEE/gLite/APT/R3.0/ rhel30 externals Release3.0 updates 1
> vi /etc/apt/sources.list.d/lcg-ca.list
> cat /etc/apt/sources.list.d/lcg-ca.list
rpm http://linuxsoft.cern.ch/ LCG-CAs/current production 2    
                    
1

Please continue to use the following repository until gLite converts to RHEL 4.0

2

This will setup the Certificate Authority public certificates.

Now you must refresh apt-get to retrieve all required RPMs. You can do it by using apt-cache command.

Example 4. APT-Cache Repository search

> apt-get update
> apt-cache search dq2
dq2-agents - to do
dq2-agents-subscription - to do
dq2-clientapi - to do
dq2-clientapi-cli - to do
dq2-common - to do
dq2-common-client - to do
dq2-common-client-curl - to do
dq2-common-client-pycurl - to do
dq2-common-dao - to do
dq2-common-dao-mysql - to do
dq2-common-dao-oracle - to do
dq2-common-server - to do
dq2-common-validator - to do
dq2-content - to do
dq2-content-client - to do
dq2-content-server - to do
dq2-content-server-oracle - to do
dq2-filecatalog - to do
dq2-filecatalog-lfc - to do
dq2-filecatalog-lrc - to do
dq2-filecatalog-rls - Interface to the Globus RLS. Used by the NDGF site services to add entries.
dq2-info - to do
dq2-location - to do
dq2-location-client - to do
dq2-location-server - to do
dq2-location-server-oracle - to do
dq2-repository - to do
dq2-repository-client - to do
dq2-repository-server - to do
dq2-repository-server-oracle - to do
dq2-service - to do
dq2-service-fetcher - to do
dq2-service-launcher - to do
dq2-service-udplogging - to do
dq2-siteservices - to do
dq2-subscription - to do
dq2-subscription-client - to do
dq2-subscription-server - to do
dq2-subscription-server-oracle - to do
                    

[Warning]Warning

The first time you run apt-get, the tool may report you have broken dependencies. If so, follow the instructions and do apt-get --fix-broken install to fix this. Then you can continue the installation process.

2.6.3. Installation of Packages

To install DQ2 client and all externals you need to execute:

Example 5. DQ2 Install

> apt-get install dq2-clientapi-cli dq2-common-client-curl
                    

There are a few more RPMs you need to install, regarding certificate authorities (CA) and VOMS role (if these are being used).

Example 6. Install other utilities

> apt-get install fetch-crl
> apt-get install lcg-CA
                    

As root, you should enable a script to rewew the certificate authorities' certificates, using a cron-job. This is done using the fetch-crl util installed before.

Example 7. Automatic renew of CA certificates

> crontab -e
> crontab -l
57 14,20,2,8 * * * /usr/sbin/fetch-crl -l /etc/grid-security/certificates/ -o /etc/grid-security/certificates/
                    

2.6.4. DQ2 Post-Installation

Now that DQ2 is fully installed along with all external dependencies, you need to configure it. At this point, you will only setup the default configuration files and in a subsequent section you will see how to tune these for your specific site, considering the load, throughput, etc.

Example 8. DQ2 Configuration

> . /opt/dq2/profile.d/dq2_common_post_install.sh

 Are you sure you want to configure DQ2 and
 potentially override ANY existing configuration files?
  (answer exactly 'yes' to proceed)
  > yes 1


 - Overwriting /opt/dq2/etc with DEFAULT values
 - Overwriting /opt/dashboard/etc with DEFAULT values

                    
1

Make sure you write exactly yes when prompted by the script.