Report Open Access

Big Data Analysis and Machine Learning at Scale with Oracle Cloud Infrastructure

Michał Bień

MARC21 XML Export

<?xml version='1.0' encoding='UTF-8'?>
<record xmlns="">
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">CERN openlab</subfield>
  <datafield tag="653" ind1=" " ind2=" ">
    <subfield code="a">summer student programme</subfield>
  <controlfield tag="005">20200120173234.0</controlfield>
  <controlfield tag="001">3550777</controlfield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">1943637</subfield>
    <subfield code="z">md5:f4ba7f91816350a8bcd872a4ba138c71</subfield>
    <subfield code="u"></subfield>
  <datafield tag="542" ind1=" " ind2=" ">
    <subfield code="l">open</subfield>
  <datafield tag="260" ind1=" " ind2=" ">
    <subfield code="c">2019-11-22</subfield>
  <datafield tag="909" ind1="C" ind2="O">
    <subfield code="p">openaire</subfield>
    <subfield code="p">user-cernopenlab</subfield>
    <subfield code="o"></subfield>
  <datafield tag="100" ind1=" " ind2=" ">
    <subfield code="a">Michał Bień</subfield>
  <datafield tag="245" ind1=" " ind2=" ">
    <subfield code="a">Big Data Analysis and Machine Learning  at Scale with Oracle Cloud Infrastructure</subfield>
  <datafield tag="980" ind1=" " ind2=" ">
    <subfield code="a">user-cernopenlab</subfield>
  <datafield tag="540" ind1=" " ind2=" ">
    <subfield code="u"></subfield>
    <subfield code="a">Creative Commons Attribution 4.0 International</subfield>
  <datafield tag="650" ind1="1" ind2="7">
    <subfield code="a">cc-by</subfield>
    <subfield code="2"></subfield>
  <datafield tag="520" ind1=" " ind2=" ">
    <subfield code="a">&lt;p&gt;This work has successfully deployed two different use cases of interest for High Energy Physics&amp;nbsp;&lt;br&gt;
using cloud resources:&amp;nbsp;&lt;br&gt;
 CMS Big data reduction: This use case consists in running a data reduction workloads for&amp;nbsp;&lt;br&gt;
physics data. The code and implementation has originally been developed by CERN openlab&amp;nbsp;&lt;br&gt;
in collaboration with CMS and Intel in 2017-2018. It aims at demonstrating the scalability of a&amp;nbsp;&lt;br&gt;
data reduction workflow, by processing ROOT files using Apache Spark&amp;nbsp;&lt;br&gt;
 Spark DL Trigger: This use case consists in the deployment of a full data preparation and&amp;nbsp;&lt;br&gt;
machine learning pipeline, starting from data ingestion (4.5 TB of ROOT data), to the training&amp;nbsp;&lt;br&gt;
of classifier using neural networks. This use case is implemented using Apache Spark and&amp;nbsp;&lt;br&gt;
the Keras API, following previous work in collaboration with CERN openlab.&amp;nbsp;&lt;br&gt;
Resources for this work have been deployed using Oracle Cloud Infrastructure (OCI). In particular&amp;nbsp;&lt;br&gt;
this project has allowed to complete:&amp;nbsp;&lt;br&gt;
 Setup of the project using Oracle Container Engine for Kubernetes and Oracle Cloud&amp;nbsp;&lt;br&gt;
 Troubleshooting of the oci-hdfs-connector to run Apache Spark at scale on OCI Object&amp;nbsp;&lt;br&gt;
 Measurements of OCI Object Storage performance for the selected use cases&amp;nbsp;&lt;br&gt;
 Investigations and performance measurements of the resource utilisation on Oracle&amp;nbsp;&lt;br&gt;
Container Engine for Kubernetes (OKE), when running the TensorFlow/Keras neural network&amp;nbsp;&lt;br&gt;
model training at scale, using CPU resources, and when using GPU.&amp;nbsp;&lt;br&gt;
Notable results of this project:&amp;nbsp;&lt;br&gt;
 Produced several key improvements to the oci-hdfs-connector. The improvements are&amp;nbsp;&lt;br&gt;
necessary to run the latest Spark version (Spark 2.4.x) on Oracle Cloud. The connector is&amp;nbsp;&lt;br&gt;
distributed by Oracle with open source licensing, and the improvements will be fed back to&amp;nbsp;&lt;br&gt;
 Improved instrumentation infrastructure for measuring Spark workloads on cloud resources,&amp;nbsp;&lt;br&gt;
by streamlining the deployment of Spark performance dashboard on Kubernetes and&amp;nbsp;&lt;br&gt;
developing a Helm chart&amp;nbsp;&lt;br&gt;
 Produced a solution for direct measurement of I/O latency for Spark workloads reading from&amp;nbsp;&lt;br&gt;
OCI or S3 storage. The results are of general interest for Spark users, notably including the&amp;nbsp;&lt;br&gt;
Spark service at CERN&amp;nbsp;&lt;br&gt;
 Developed methods to parallelize TensorFlow/Keras on Kubernetes using TensorFlow 2.0&amp;nbsp;&lt;br&gt;
new tf.distribute features. These are of general interest for ML practitioners, notably including&amp;nbsp;&lt;br&gt;
the users of CERN cloud services.&lt;/p&gt;</subfield>
  <datafield tag="773" ind1=" " ind2=" ">
    <subfield code="n">doi</subfield>
    <subfield code="i">isVersionOf</subfield>
    <subfield code="a">10.5281/zenodo.3550776</subfield>
  <datafield tag="024" ind1=" " ind2=" ">
    <subfield code="a">10.5281/zenodo.3550777</subfield>
    <subfield code="2">doi</subfield>
  <datafield tag="980" ind1=" " ind2=" ">
    <subfield code="a">publication</subfield>
    <subfield code="b">report</subfield>
All versions This version
Views 385384
Downloads 469469
Data volume 911.6 MB911.6 MB
Unique views 359358
Unique downloads 446446


Cite as