Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace dynamodb scan with query #31

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Replace dynamodb scan with query #31

wants to merge 12 commits into from

Conversation

goruha
Copy link
Member

@goruha goruha commented Jun 12, 2024 •
edited
Loading

what

  • Replace DynamoDB scan with query

why

  • Because scan works by batch 1Mb of data.
If the total size of scanned items exceeds the maximum dataset size limit of 1 MB, the scan completes and results are returned to the user. The LastEvaluatedKey value is also returned and the requestor can use the LastEvaluatedKey to continue the scan in a subsequent operation.
  • Reduce cost. query is cheaper then scan
  • Fix false negatives get plan cases

CleanShot 2024-06-12 at 14 33 55@2x

references

Migration

  • Create dynamodb index ``
          - name: commitSHA-index
            hash_key: commitSHA
            range_key: createdAt
            projection_type: ALL
            non_key_attributes: []
            read_capacity: null
            write_capacity: null

@goruha goruha requested review from a team as code owners June 12, 2024 12:35
@goruha goruha requested review from hans-d and kevcube June 12, 2024 12:35
goruha and others added 3 commits June 12, 2024 12:36
@goruha @github-actions
Update contents of the dist directory
e66d0ec
@goruha
Update workflows
38ed404
@goruha
Merge branch 'replace-dynamodb-scan-with-query' of github.com:cloudpo…
2c73d5b 
…sse/github-action-terraform-plan-storage into replace-dynamodb-scan-with-query

* 'replace-dynamodb-scan-with-query' of github.com:cloudposse/github-action-terraform-plan-storage:
  Update contents of the dist directory
@goruha goruha added the major Breaking changes (or first stable release) label Jun 12, 2024
osterman
osterman previously approved these changes Jun 12, 2024
View reviewed changes
@goruha goruha added the do not merge Do not merge this PR, doing so would cause problems label Jun 13, 2024
mcalhoun
mcalhoun requested changes Sep 19, 2024
View reviewed changes
src/modules/terraformPlan/repo/DynamoDbMetadataRepo.ts Outdated
FilterExpression:
"#owner = :owner and #repo = :repo and #commitSHA = :commitSHA and #component = :component and #stack = :stack",
"#owner = :owner and #repo = :repo and #component = :component and #stack = :stack",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we removing the condition to match the commit SHA? I don't see how this could be correct?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mcalhoun because we filter sha with KeyCondition
https://github.com/cloudposse/github-action-terraform-plan-storage/pull/31/files#diff-95366d8215594fbafa600c4d01acd3040d43de241933dcf7a6ad86ead80e391aR33

KeyConditionExpression: "#commitSHA= :commitSHA",

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mcalhoun mcalhoun mcalhoun requested changes

@osterman osterman osterman left review comments

@hans-d hans-d Awaiting requested review from hans-d hans-d is a code owner automatically assigned from cloudposse/contributors

@kevcube kevcube Awaiting requested review from kevcube kevcube is a code owner automatically assigned from cloudposse/contributors

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
do not merge Do not merge this PR, doing so would cause problems major Breaking changes (or first stable release)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@goruha @osterman @mcalhoun