Report Open Access

HEP Application Delivery on HPC Resources

Shaffer, Tim; Blomer, Jakob; Ganis, Gerardo

JSON Export

  "files": [
      "links": {
        "self": ""
      "checksum": "md5:05c79632e65bd95b006fb87742e3049e", 
      "bucket": "fa4dda36-1363-4b30-901d-25243a6a50b8", 
      "key": "Report_Tim_Shaffer.pdf", 
      "type": "pdf", 
      "size": 453520
  "owners": [
  "doi": "10.5281/zenodo.61157", 
  "stats": {
    "version_unique_downloads": 22.0, 
    "unique_views": 32.0, 
    "views": 33.0, 
    "version_views": 33.0, 
    "unique_downloads": 22.0, 
    "version_unique_views": 32.0, 
    "volume": 10430960.0, 
    "version_downloads": 23.0, 
    "downloads": 23.0, 
    "version_volume": 10430960.0
  "links": {
    "doi": "", 
    "latest_html": "", 
    "bucket": "", 
    "badge": "", 
    "html": "", 
    "latest": ""
  "created": "2016-08-31T07:41:59+00:00", 
  "updated": "2020-01-20T13:14:51.826183+00:00", 
  "conceptrecid": "640738", 
  "revision": 10, 
  "id": 61157, 
  "metadata": {
    "access_right_category": "success", 
    "doi": "10.5281/zenodo.61157", 
    "description": "<p>Project Specification</p>\n\n<p>High-performance computing (HPC) contributes a significant and growing share of&nbsp;resource to high-energy physics (HEP). Individual supercomputers such as&nbsp;Edison or&nbsp;Titan in the U.S. or SuperMUC in Europe deliver a raw performance of the same order of&nbsp;magnitude than the Worldwide LHC Computing Grid. As we have seen with codes from&nbsp;ALICE and ATLAS, it is notoriously difficult to deploy high-energy physics applications&nbsp;on supercomputers, even though they often run a standard Linux on Intel x86_64 CPUs.</p>\n\n<p>The three main problems are:</p>\n\n<p>1. Limited or no Internet access;</p>\n\n<p>2. The lack of privileged local system rights;</p>\n\n<p>3. The concept of cluster submission or whole-node submission of jobs in contrast to&nbsp;single CPU slot submission in HEP.</p>\n\n<p>Generally, the delivery of applications to hardware resources in high-energy physics is&nbsp;done by CernVM-FS [1]. CernVM-FS is optimized for high-throughput resources.&nbsp;Nevertheless, some successful results on HPC resources where&nbsp;achieved using the Parrot&nbsp;system[2] that allows to use CernVM-FS without special privileges. Building on these&nbsp;results, the project aims to prototype a toolkit for application delivery that seamlessly&nbsp;integrates with HEP experiments job submission systems, for instance with ALICE AliEn&nbsp;or ATLAS PanDA. The&nbsp;task includes a performance study of the parrot-induced&nbsp;overhead which will be used to guide performance tuning for both CernVM-FS and&nbsp;Parrot on typical&nbsp;supercomputers. The project should further deliver a lightweight&nbsp;scheduling shim that translates HEP&rsquo;s job slot allocation to a whole&nbsp;node or cluster-based&nbsp;allocation. Finally, in order to increase the turn-around of the evaluation of new&nbsp;supercomputers, a set of &quot;canary jobs&quot; should be&nbsp;collected that validate HEP codes on&nbsp;new resources.</p>\n\n<p>[1]</p>\n\n<p>[2]</p>\n\n<p>Abstract</p>\n\n<p>On high performance computing (HPC) resources, users have less control over&nbsp;worker&nbsp;nodes than in the grid. Using HPC resources for high energy physics&nbsp;applications&nbsp;becomes more complicated because individual nodes often&nbsp;don&#39;t have Internet&nbsp;connectivity or a filesystem configured to use as a local&nbsp;cache. The current solution in&nbsp;CVMFS preloads the cache from a gateway node onto the shared cluster file system.&nbsp;This approach works but does not scale&nbsp;well into large production environments. In this&nbsp;project, we develop an in&nbsp;memory cache for CVMFS, and assess approaches to running&nbsp;jobs without&nbsp;special privilege on the worker nodes. We propose using Parrot and CVMFS&nbsp;with RAM cache as a viable approach to HEP application delivery on&nbsp;HPC resources.</p>", 
    "license": {
      "id": "CC-BY-4.0"
    "title": "HEP Application Delivery on HPC Resources", 
    "relations": {
      "version": [
          "count": 1, 
          "index": 0, 
          "parent": {
            "pid_type": "recid", 
            "pid_value": "640738"
          "is_last": true, 
          "last_child": {
            "pid_type": "recid", 
            "pid_value": "61157"
    "communities": [
        "id": "cernopenlab"
    "keywords": [
      "CERN openlab summer student"
    "publication_date": "2016-08-30", 
    "creators": [
        "affiliation": "CERN openlab Summer Student", 
        "name": "Shaffer, Tim"
        "affiliation": "Summer Student Supervisor", 
        "name": "Blomer, Jakob"
        "affiliation": "Summer Student Supervisor ", 
        "name": "Ganis, Gerardo"
    "access_right": "open", 
    "resource_type": {
      "subtype": "report", 
      "type": "publication", 
      "title": "Report"
All versions This version
Views 3333
Downloads 2323
Data volume 10.4 MB10.4 MB
Unique views 3232
Unique downloads 2222


Cite as