Lightweight library to write, orchestrate and test your SQL ETL. Writing ETL with data integrity in mind.
-
Updated
Jan 11, 2024 - Python
#
data-engineering-workflows
Here are 6 public repositories matching this topic...
Data Engineering portfolio projects, resources used to study data tools...
python
sql
apache-spark
databases
data-engineering
apache-kafka
data-pipelines
apache-airflow
data-engineering-workflows
etl-automation
data-portfolio
-
Updated
Mar 25, 2024 - Jupyter Notebook
Examples that I use to learn and show Apache Beam
etl
examples
data-engineering
apache-beam
data-processing
etl-pipeline
data-engineering-workflows
data-engineering-pipeline
-
Updated
Oct 24, 2018 - Python
A first course for data engineering workflows
-
Updated
May 21, 2018 - Python
This repository is the collection point for all of the projects completed during the Udacity Data Engineering Nano Degree program.
-
Updated
May 30, 2021 - Jupyter Notebook
Leveraging AWS Cloud Services, an ETL pipeline transforms YouTube video statistics data. Data is downloaded from Kaggle, uploaded to an S3 bucket, and cataloged using AWS Glue for querying with Athena. AWS Lambda and Glue converts to Parquet format and stores it in a cleansed S3 bucket. AWS QuickSight then visualizes the materialised data.
python
aws
spark
aws-lambda
etl
aws-s3
pandas
pyspark
data-engineering
aws-iam
aws-cloudwatch
data-pipeline
etl-pipeline
aws-glue
data-engineering-workflows
data-engineering-pipeline
aws-lambda-layers
aws-data-engineering-project
data-engineering-project
-
Updated
May 30, 2024 - Python
Improve this page
Add a description, image, and links to the data-engineering-workflows topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the data-engineering-workflows topic, visit your repo's landing page and select "manage topics."