Skip to content

Latest commit

 

History

History
36 lines (24 loc) · 1.74 KB

README.md

File metadata and controls

36 lines (24 loc) · 1.74 KB

Edge_ML_KS-Test: Distributed KS Test for Edge ML

This is a prototype for the software system explained in the paper: Distributed Monitoring for Data Distribution Shifts in Edge-ML Fraud Detection feel free to use this code for inspiration/Implement the monitoring in your environment.

Overview

This code contains an implementation of the Kolmogorov-Smirnov (KS) Test, designed explicitly for distributed edge computing environments (Edge-ML for instance). Our design ensures efficiency for both the distributed clients and the aggregating server.

Page Twosystem diagram (6) (2)

Getting Started

Prerequisites

Before running the framework, ensure you have the following prerequisites:

  • Python 3.x
  • Access to AWS SQS or a local stack setup for it
  • Access to AWS lambda or a local stack setup for it (or just use the sqs_server as I did in the mock e2e simulation)
  • Redis Instance

Installation

  1. Install Dependencies: Refer to requirements.txt for the necessary Python packages and install them using pip install -r requirements.txt.

  2. Setup cloud/local environment: The queue, the lambda, and the Redis instance

Configuration

  1. Configure AWS SQS/Local Stack: Update the client configuration framework/edge_client/client_configs.json and server configuration framework/queue_consumer_server/server_configs.json files with the details of your SQS setup.
  2. Configure Redis: Update the Redis client configs in the SQSServer code.

Execution

  1. Run the mock Simulation: Execute application_e2e_simulation.py to start an end-to-end mock of the system.
  2. Edit the code to match your needs and environment...