Home | Logs | Amazon Kinsesis | Kinsesis Analytics | Help/Resources.
Welcome to Data Engineering 3: Real-Time Data Processing.
- For "official" communication we are using Moodle
- For informal and in-class communication, we are using Slack
- On this page you will find additional courseware, R and Python scripts, which accompany the course
- In case you need help, you can find the instructor's and the TA's contact details at the bottom of this page
- Before you'd reshare/reuse any of these materials keep in mind that Datapao sponsors this course. Datapao owns the copyright of the Spark notebooks (distributed separately) and possibly other materials in this repository. Reach out to Zoltan C. Toth if you have any questions about this.
Data Engineering 3: Real-Time Data Processing
Prerequisites:
Data Analysis 1: Exploration – Business Analytics track;
Data Analysis 2: Finding Patterns with Regressions – Business Analytics track;
Data Engineering 2: Big Data and Cloud Computing
Course description In this course, you will learn which are the underlying concepts of real-time data processing systems. You will get an introduction of different technologies you can employ to solve real-time use-cases with a special focus on Amazon Web Services and Apache Spark
Learning outcomes By the end of the course you will:
- Understand the building blocks of a real-time data infrastructure
- Understand the caveats that arise when you take these systems to production
- You will have hands-on knowledge on how to build a simple, end-to-end real-time pipeline with Amazon Web Services (AWS) and Apache Spark
Note: Data, presentations, and code for the exercises will be provided.
- Heart Logs: Event Data, Stream Processing, and Data Integration Download
Course schedule and materials for each session This class runs for 2x300 minutes.
Week 1:
- Spark Recap and Aggregations in Spark
- Data Infrastructures overview
- Streaming Basics
- Spark Streaming basics
Week 2: Problems in Production
- Message Brokers
- AWS: Amazon Kinesis and Kinesis SQL
- Handling Late Data
- Exactly once processing
Instructor | Teaching Assistant |
---|---|
Zoltan C. Toth | Miklos Petridisz |
TothZ@ceu.edu | miklospetridisz1@gmail.com |
+36 30 291 3599 | +36 30 537 9243 |