Report Open Access

Big Data Analysis and Machine Learning at Scale with Oracle Cloud Infrastructure

Michał Bień

DCAT Export

<?xml version='1.0' encoding='utf-8'?>
<rdf:RDF xmlns:rdf="" xmlns:adms="" xmlns:dc="" xmlns:dct="" xmlns:dctype="" xmlns:dcat="" xmlns:duv="" xmlns:foaf="" xmlns:frapo="" xmlns:geo="" xmlns:gsp="" xmlns:locn="" xmlns:org="" xmlns:owl="" xmlns:prov="" xmlns:rdfs="" xmlns:schema="" xmlns:skos="" xmlns:vcard="" xmlns:wdrs="">
  <rdf:Description rdf:about="">
    <dct:identifier rdf:datatype=""></dct:identifier>
    <foaf:page rdf:resource=""/>
        <rdf:type rdf:resource=""/>
        <foaf:name>Michał Bień</foaf:name>
    <dct:title>Big Data Analysis and Machine Learning at Scale with Oracle Cloud Infrastructure</dct:title>
    <dct:issued rdf:datatype="">2019</dct:issued>
    <dcat:keyword>CERN openlab</dcat:keyword>
    <dcat:keyword>summer student programme</dcat:keyword>
    <dct:issued rdf:datatype="">2019-11-22</dct:issued>
    <owl:sameAs rdf:resource=""/>
        <skos:notation rdf:datatype=""></skos:notation>
    <dct:isVersionOf rdf:resource=""/>
    <dct:isPartOf rdf:resource=""/>
    <dct:description>&lt;p&gt;This work has successfully deployed two different use cases of interest for High Energy Physics&amp;nbsp;&lt;br&gt; using cloud resources:&amp;nbsp;&lt;br&gt;  CMS Big data reduction: This use case consists in running a data reduction workloads for&amp;nbsp;&lt;br&gt; physics data. The code and implementation has originally been developed by CERN openlab&amp;nbsp;&lt;br&gt; in collaboration with CMS and Intel in 2017-2018. It aims at demonstrating the scalability of a&amp;nbsp;&lt;br&gt; data reduction workflow, by processing ROOT files using Apache Spark&amp;nbsp;&lt;br&gt;  Spark DL Trigger: This use case consists in the deployment of a full data preparation and&amp;nbsp;&lt;br&gt; machine learning pipeline, starting from data ingestion (4.5 TB of ROOT data), to the training&amp;nbsp;&lt;br&gt; of classifier using neural networks. This use case is implemented using Apache Spark and&amp;nbsp;&lt;br&gt; the Keras API, following previous work in collaboration with CERN openlab.&amp;nbsp;&lt;br&gt; Resources for this work have been deployed using Oracle Cloud Infrastructure (OCI). In particular&amp;nbsp;&lt;br&gt; this project has allowed to complete:&amp;nbsp;&lt;br&gt;  Setup of the project using Oracle Container Engine for Kubernetes and Oracle Cloud&amp;nbsp;&lt;br&gt; resources&amp;nbsp;&lt;br&gt;  Troubleshooting of the oci-hdfs-connector to run Apache Spark at scale on OCI Object&amp;nbsp;&lt;br&gt; Storage&amp;nbsp;&lt;br&gt;  Measurements of OCI Object Storage performance for the selected use cases&amp;nbsp;&lt;br&gt;  Investigations and performance measurements of the resource utilisation on Oracle&amp;nbsp;&lt;br&gt; Container Engine for Kubernetes (OKE), when running the TensorFlow/Keras neural network&amp;nbsp;&lt;br&gt; model training at scale, using CPU resources, and when using GPU.&amp;nbsp;&lt;br&gt; Notable results of this project:&amp;nbsp;&lt;br&gt;  Produced several key improvements to the oci-hdfs-connector. The improvements are&amp;nbsp;&lt;br&gt; necessary to run the latest Spark version (Spark 2.4.x) on Oracle Cloud. The connector is&amp;nbsp;&lt;br&gt; distributed by Oracle with open source licensing, and the improvements will be fed back to&amp;nbsp;&lt;br&gt; Oracle.&amp;nbsp;&lt;br&gt;  Improved instrumentation infrastructure for measuring Spark workloads on cloud resources,&amp;nbsp;&lt;br&gt; by streamlining the deployment of Spark performance dashboard on Kubernetes and&amp;nbsp;&lt;br&gt; developing a Helm chart&amp;nbsp;&lt;br&gt;  Produced a solution for direct measurement of I/O latency for Spark workloads reading from&amp;nbsp;&lt;br&gt; OCI or S3 storage. The results are of general interest for Spark users, notably including the&amp;nbsp;&lt;br&gt; Spark service at CERN&amp;nbsp;&lt;br&gt;  Developed methods to parallelize TensorFlow/Keras on Kubernetes using TensorFlow 2.0&amp;nbsp;&lt;br&gt; new tf.distribute features. These are of general interest for ML practitioners, notably including&amp;nbsp;&lt;br&gt; the users of CERN cloud services.&lt;/p&gt;</dct:description>
    <dct:accessRights rdf:resource=""/>
      <dct:RightsStatement rdf:about="info:eu-repo/semantics/openAccess">
        <rdfs:label>Open Access</rdfs:label>
    <dct:license rdf:resource=""/>
        <dcat:accessURL rdf:resource=""></dcat:accessURL>
        <dcat:downloadURL rdf:resource=""></dcat:downloadURL>
All versions This version
Views 385384
Downloads 469469
Data volume 911.6 MB911.6 MB
Unique views 359358
Unique downloads 446446


Cite as