Skip to content

daq-tools/DynamoDBExporter

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Tinybird DynamoDB Export Utilities

This repository contains example solutions for replicating data out of DynamoDB into Tinybird.

Both solutions, scanToExport and DDBStreamCDC contain their own readme files on usage.

DDBStreamCDC [Recommended]

This solution leverages AWS DynamoDB Export to S3 & AWS DynamoDB Streams functionalities via a Python Lambda to forward both snapshots and changes to a Tinybird Datasource with Keys automatically indexed, which is then deduplicated for the latest values and made available for users.

A simple implementation pathway is provided which may be expanded upon.

More detail in the dedicated readme

scanToExport [Legacy]

This solution uses a simple DynamoDB Scan to export a file to an S3 bucket, and then uses the Tinybird Datasource Replace functionality, implemented as a Python Lambda.

It is simpler in nature than the CDC solution, though it lacks the low latency updates and you may find a Scan is more expensive in practice.

About

Open-source DynamoDB integration for Tinybird

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 95.9%
  • Shell 4.1%