Skip to content

AndresJejen/aws_cdk_emr

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Data Infraestructure on AWS with AWS CDK (Cloud development kit)

In this repo you can find a CDK stack that builds a serverless application that runs AWS EMR when is needed.

What is AWS CDK

Is a AWS Service that allows to create Infrastructure as code using the most popular languages like (Python, Typescript, Java and C#). You can deploy a simple stack to a complex modular infrastructure, edit it and make your cloud application scalable.
Also cam be used to deploy data infrastructure, and maintain it in asingle project, no more fitting roles and permissions in the console UI. As simple as write a line of code you can add or revoke access to your infrastructure.

What is this Repo About?

We will build a Apache Spark as a service application, the idea is user AWS EMR as a service. We only load a script in AWS S3 and we trigger a process that creates the cluster, run the job and destroy the cluster at end.

Used Services in this project

  • AWS CDK
  • AWS Cloud Formation
  • AWS S3
  • AWS EMR
  • AWS Lambda

Getting started

  1. Set environmentVariables
        export account_id=<Your Account Id>
        export bucket_name=<Name of the bucket>
        export data_bucket_arn=<arn of data bucket>
    
  2. Install node js
  3. Install aws cdk
        npm install -g aws-cdk
    
  4. Set your credentials user In this step you need a aws account and aws cli v2 installed in your computer
        aws configure --profile cdk
    
  5. start your AWS cdk project
        cdk init app --language typescript
    
  6. Write your code infrastructure
  7. transpile and synthesize the code (if you're using typescript)
        npm run build
        cdk synth
    
  8. Bootstrap
        cdk bootstrap --profile cdk
    
  9. Deploy
        cdk deploy --profile cdk
    
  10. Load your script
  11. Wait for AWS makes magic and drink a coffee
  12. Watch the result

Data in this example

Useful commands

  • npm run build compile typescript to js
  • npm run watch watch for changes and compile
  • npm run test perform the jest unit tests
  • cdk deploy deploy this stack to your default AWS account/region
  • cdk diff compare deployed stack with current state
  • cdk synth emits the synthesized CloudFormation template

Who do I talk to?

Resources

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published