Data Science and Openshift on the MOC

Pull request accepted by RedHat into main repo on 06/11/2020.

Metors

• Michael Clifford, RedHat Inc.
• Anand Sanmukhani, RedHat Inc.

Team

• Yousif Khaireddin ykh@bu.edu
• Zhuofa (Marco) Chen zfchen@bu.edu
• Krishna Palle pallekc@bu.edu
• Fangya Xu fangya@bu.edu
• Nihar Dwivedi ndwivedi@bu.edu
• Ojasvi Jhamb ojhamb@bu.edu

Project Report and Final Video

Click here for a comprehensive Project Report

Our GitHub repo containing our Prometheus Anomaly Detector can be found here

The Final presentation video for our project can be found here.

The slides for this presentation can be found here.

Goals

• Understanding the current ML models (SARIMA and Prophet) and workings of Prometheus Anomaly Detection (PAD), an existing open-source project developed by the AI team at RedHat to analyze time-series data generated by cloud infrastructure

• Developing and deploying a new ML model (LSTM) onto DataHub, a blueprint for building an AI application as a service platform on the OpenShift Container Platform.

• Testing the workings of DataHub on the MOC and reporting any problems for future ramifications.

Users/Personas

• The open-source community

• MOC users that want to analyze time-series data generated by cloud infrastructure and send alerts for potential anomalies

• Site Reliability Engineers that would like to monitor the health of their applications

Scope and Features

• Ensuring the deployment and functionality of DataHub on the MOC

• Extension of the current Prometheus Anomaly Detection tool with a new model(s)

• Exploring/reporting any issues and bugs we encounter along with the project for future ramifications

Solution Concept

• Installing OpenShift client and datahub on our locally for familiarization

• Developing and testing PAD ML algorithms on JupiterHub

• Testing the current models on JupyterHub

• Building a simple LSTM model on JupyterHub

• Experimenting with different architectures, optimizers, and layers in our model

• Automating the tuning of our model

• Comparing our model’s performance with that of the current deployment

• Deploying our models onto the DataHub

Acceptance Criteria

Developing at least one new ML model (LSTM) and having it monitor any metric while running on DataHub.

Release Planning

1. Project familiarization (02/05)

   1. MassOpenCloud (MOC)

   2. OpenShift

   3. Kubernetes

   4. OpenDataHub

   5. Prometheus Anomaly Detection (PAD)

2. Deploy DataHub onto MOC (03/01)

3. Set up Data Hub on Openshift (03/15)

4. Extend Prometheus with a new model(s) using Jupyter notebooks (04/15)

5. Test the models on Prometheus using data provided by the mentor (05/01)

6. Migrate built ML Algorithms and models onto DataHub (05/05)

Links to Presentations / Class Lecture

• Sprint 1
• Sprint 2
• Sprint 3
• Sprint 4
• MapReduce Talk
• Sprint 5