Report Open Access

Big Data Analysis and Machine Learning at Scale with Oracle Cloud Infrastructure

Michał Bień

DataCite XML Export

<?xml version='1.0' encoding='utf-8'?>
<resource xmlns:xsi="" xmlns="" xsi:schemaLocation="">
  <identifier identifierType="DOI">10.5281/zenodo.3550777</identifier>
      <creatorName>Michał Bień</creatorName>
    <title>Big Data Analysis and Machine Learning  at Scale with Oracle Cloud Infrastructure</title>
    <subject>CERN openlab</subject>
    <subject>summer student programme</subject>
    <date dateType="Issued">2019-11-22</date>
  <resourceType resourceTypeGeneral="Report"/>
    <alternateIdentifier alternateIdentifierType="url"></alternateIdentifier>
    <relatedIdentifier relatedIdentifierType="DOI" relationType="IsVersionOf">10.5281/zenodo.3550776</relatedIdentifier>
    <relatedIdentifier relatedIdentifierType="URL" relationType="IsPartOf"></relatedIdentifier>
    <rights rightsURI="">Creative Commons Attribution 4.0 International</rights>
    <rights rightsURI="info:eu-repo/semantics/openAccess">Open Access</rights>
    <description descriptionType="Abstract">&lt;p&gt;This work has successfully deployed two different use cases of interest for High Energy Physics&amp;nbsp;&lt;br&gt;
using cloud resources:&amp;nbsp;&lt;br&gt;
 CMS Big data reduction: This use case consists in running a data reduction workloads for&amp;nbsp;&lt;br&gt;
physics data. The code and implementation has originally been developed by CERN openlab&amp;nbsp;&lt;br&gt;
in collaboration with CMS and Intel in 2017-2018. It aims at demonstrating the scalability of a&amp;nbsp;&lt;br&gt;
data reduction workflow, by processing ROOT files using Apache Spark&amp;nbsp;&lt;br&gt;
 Spark DL Trigger: This use case consists in the deployment of a full data preparation and&amp;nbsp;&lt;br&gt;
machine learning pipeline, starting from data ingestion (4.5 TB of ROOT data), to the training&amp;nbsp;&lt;br&gt;
of classifier using neural networks. This use case is implemented using Apache Spark and&amp;nbsp;&lt;br&gt;
the Keras API, following previous work in collaboration with CERN openlab.&amp;nbsp;&lt;br&gt;
Resources for this work have been deployed using Oracle Cloud Infrastructure (OCI). In particular&amp;nbsp;&lt;br&gt;
this project has allowed to complete:&amp;nbsp;&lt;br&gt;
 Setup of the project using Oracle Container Engine for Kubernetes and Oracle Cloud&amp;nbsp;&lt;br&gt;
 Troubleshooting of the oci-hdfs-connector to run Apache Spark at scale on OCI Object&amp;nbsp;&lt;br&gt;
 Measurements of OCI Object Storage performance for the selected use cases&amp;nbsp;&lt;br&gt;
 Investigations and performance measurements of the resource utilisation on Oracle&amp;nbsp;&lt;br&gt;
Container Engine for Kubernetes (OKE), when running the TensorFlow/Keras neural network&amp;nbsp;&lt;br&gt;
model training at scale, using CPU resources, and when using GPU.&amp;nbsp;&lt;br&gt;
Notable results of this project:&amp;nbsp;&lt;br&gt;
 Produced several key improvements to the oci-hdfs-connector. The improvements are&amp;nbsp;&lt;br&gt;
necessary to run the latest Spark version (Spark 2.4.x) on Oracle Cloud. The connector is&amp;nbsp;&lt;br&gt;
distributed by Oracle with open source licensing, and the improvements will be fed back to&amp;nbsp;&lt;br&gt;
 Improved instrumentation infrastructure for measuring Spark workloads on cloud resources,&amp;nbsp;&lt;br&gt;
by streamlining the deployment of Spark performance dashboard on Kubernetes and&amp;nbsp;&lt;br&gt;
developing a Helm chart&amp;nbsp;&lt;br&gt;
 Produced a solution for direct measurement of I/O latency for Spark workloads reading from&amp;nbsp;&lt;br&gt;
OCI or S3 storage. The results are of general interest for Spark users, notably including the&amp;nbsp;&lt;br&gt;
Spark service at CERN&amp;nbsp;&lt;br&gt;
 Developed methods to parallelize TensorFlow/Keras on Kubernetes using TensorFlow 2.0&amp;nbsp;&lt;br&gt;
new tf.distribute features. These are of general interest for ML practitioners, notably including&amp;nbsp;&lt;br&gt;
the users of CERN cloud services.&lt;/p&gt;</description>
All versions This version
Views 385384
Downloads 469469
Data volume 911.6 MB911.6 MB
Unique views 359358
Unique downloads 446446


Cite as