TELEMAX is a company that plans to build networks in rapidly growing, underserved markets around the world. The company offers innovative communications hardware that enables them to create many speedefficient networking links with inexpensive hardware.
Presently, they need, to deploy an effective architecture for NoSQL-based data warehousing built from real-time data being generated that can be analyzed in the future to optimize their topologies continuously. The team decided the AWS cloud is the perfect environment to support their solution needs and they will approach your organization for consultation on the same.
To create data in a Kinesis stream that can be copied to the DynamoDB database.
1. Create AWS Kinesis stream.
2. Create AWS DynamoDB.
3. Create AWS Lambda function to store Kinesis events in the DynamoDB.
4. Grant Permission to the Lambda function to
- read data from the Kinesis stream
- write data to DynamoDB
- write logs to CloudWatch
1. Open terminal
2. Before you can execute the terraform script, your need to configure your aws environment first.
aws configure
AWS Access Key ID: See IAM > Security credentials > Access keys > Create access key
AWS Secret Access Key: See IAM > Security credentials > Access keys > Create access key
Default region name: us-east-1
Default output format: json
3. Now you can apply the terraform changes.
terraform init
terraform apply --auto-approve
Result: The Kinesis stream and Lambda function are created and ready to consume events.
4. Open Colaboratory and create new Notebook.
5. Execute command: pip install boto3
6. Execute script: realtime_data_produce.py
Note: Before running the script, update the aws access token and region in the script.
Result: Events are written to Kinesis.
7. Scan the DynamoDB table for data records (DynamoDB > Explore items > "your table" > Scan > Run)
Result: Events are stored in the DynamoDB. See Output Screenshot.
8. Finally cleanup allocated aws resources.
terraform destroy --auto-approve
