ATLAS Distributed Data Management

DQ2 Clients

Mario Lassnig

CERN / University of Innsbruck, Austria

Miguel Branco

ATLAS Distributed Data Management Client Guide

The ATLAS Distributed Data Management (DDM) project is developed by the ATLAS Computing Group (CERN PH/ADP) in collaboration with the ARDA group (CERN LCG/ARDA). The software it provides is able to schedule movement of large datasets across computing sites. This guide covers the client suite to interact with the DDM system.


Table of Contents

1. Overview
2. Installation
2.1. Activating pre-installed DQ2 clients
2.1.1. CERN/lxplus
2.1.2. US
2.1.3. NDGF
2.2. External Dependencies
2.3. Installation Alternatives
2.4. Install DQ2 clients with pacman
2.5. Install DQ2 clients with pacman - special US/OSG notes
2.6. Install DQ2 clients with RPM
2.6.1. RPM-based installation with APT Library
2.6.2. APT Repository
2.6.3. Installation of Packages
2.6.4. DQ2 Post-Installation
3. Enduser tools
3.1. dq2-ls
3.1.1. Listing datasets
3.1.2. Listing dataset replicas
3.1.3. Listing files in a dataset
3.1.4. Listing PFNs of files in a dataset
3.1.5. Creating PoolFileCatalog.xml files
3.2. dq2-get
3.2.1. Download a dataset
3.2.2. Download parts of a dataset
3.2.3. Download a dataset into your private mass storage path
3.2.4. Download a part of a dataset from a specific site
3.3. dq2-put
3.3.1. Creating a new dataset from local disk
3.3.2. Creating a new version of a dataset
4. Commandline utilities
4.1. dq2-check-replica-consistency
4.2. dq2-close-dataset
4.3. dq2-delete-datasets
4.4. dq2-delete-files
4.5. dq2-delete-replicas
4.6. dq2-delete-subscription
4.7. dq2-destinations
4.8. dq2-erase
4.9. dq2-freeze-dataset
4.10. dq2-get-metadata
4.11. dq2-get-number-files
4.12. dq2-get-replica-metadata
4.13. dq2-list-dataset
4.14. dq2-list-dataset-by-creationdate
4.15. dq2-list-dataset-replicas
4.16. dq2-list-dataset-site
4.17. dq2-list-datasets-container
4.18. dq2-list-erased-datasets
4.19. dq2-list-file-replicas
4.20. dq2-list-files
4.21. dq2-list-subscription
4.22. dq2-list-subscription-info
4.23. dq2-list-subscription-site
4.24. dq2-metadata
4.25. dq2-ping
4.26. dq2-register-container
4.27. dq2-register-dataset
4.28. dq2-register-datasets
4.29. dq2-register-datasets-container
4.30. dq2-register-files
4.31. dq2-register-location
4.32. dq2-register-subscription
4.33. dq2-register-version
4.34. dq2-reset-subscription
4.35. dq2-reset-subscription-site
4.36. dq2-sample
4.37. dq2-sources
5. Frequently asked questions
6. Appendix
6.1. Contact
6.2. Author
6.3. Acknowledgements
6.4. Licence

List of Tables

1. External dependencies table

List of Examples

1. APT Install Help
2. DQ2 APT Repository Configuration
3. gLite APT Repository Configuration
4. APT-Cache Repository search
5. DQ2 Install
6. Install other utilities
7. Automatic renew of CA certificates
8. DQ2 Configuration

1. Overview

This document describes the installation and usage of the DQ2 client suite.

The DQ2 Clients have two main components:

  • DQ2 Enduser tools, responsible for high-level workflow-like operation.

  • Commandline utilities, responsible for single specialised tasks in a POSIX-fashion.

This document focuses on the Client suite of DQ2. The client suite consist of tools, that communicate with a tightly coupled Python API, which in turn interacts with the Central Dataset Catalogues. This allows users to create and modify datasets, as well as move data by subscription requests.