Skip to content

Latest commit

 

History

History
102 lines (74 loc) · 2.93 KB

README.md

File metadata and controls

102 lines (74 loc) · 2.93 KB
Raw

Building a Data Lakehouse for Analyzing Elon Musk Tweets using MinIO, Apache Airflow, Apache Drill and Apache Superset

Table of Contents

About

A Data Project using Apache-Airflow (Orchestrator) and MinIO(Object Storage like s3), Apache-Drill (SQL Query Engine) and Apache-Superset (Visualization).

Getting Started

Below is a design of the project. (The drawio file can be found in docs/architecture.drawio: Architecture

Prerequisites

Usage

  1. We need to create and .env from sample.env:
cp sample.env .env
  1. Add the Twitter Bearer Token in the .env file as below:
TWITTER_BEARER_TOKEN="TOKEN-GOES-HERE"
  1. 1 - If you want to change the MINIO_BUCKET_NAME, you have to change in the conf/drill/storage-plugins-override.conf:
connection: "s3a://twitter-data"
  1. 2 - If you want to change the MINIO_ROOT_USER and MINIO_ROOT_PASSWORD, you have to change in the conf/drill/core-site.xml:
<property>
    <name>fs.s3a.access.key</name>
    <value>minioadmin</value>
</property>

<property>
    <name>fs.s3a.secret.key</name>
    <value>minioadmin</value>
</property>
  1. We can simply run the pipeline using docker-compose:
  • If you are using linux, you need to change the permission of apps/logs/ to allow the container to write to this directory:
chmod -R 777 app/logs superset.db
  • To START
docker compose up -d
  • To SHUTDOWN
docker compose down
  1. Then we can connect to each below respectively:

Resources

License

This project is licensed under the MIT License - see the LICENSE file for details.

Credits

See as you fit.

Contact

If you have any questions or would like to get in touch, you can email: mailto:mike.kenneth47@gmail.com OR twitter