Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add S3 versioning to managed schemas #291

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Add S3 versioning to managed schemas #291

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
5d345ff
Adding versioning
javsanbel2 Nov 28, 2024
a950ff7
add s3
javsanbel2 Nov 28, 2024
7ec0ae7
fix status
javsanbel2 Nov 28, 2024
c9b2cd3
test 3
javsanbel2 Dec 2, 2024
ea901e4
fix tf bug
javsanbel2 Dec 3, 2024
e7bd8a5
rename variables
javsanbel2 Dec 9, 2024
399f011
add changelog
javsanbel2 Dec 12, 2024
1a37bae
Update VARIABLES.md
javsanbel2 Dec 12, 2024
96f6d9a
Remove max versions
javsanbel2 Dec 16, 2024
90148b7
fix
javsanbel2 Dec 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [7.8.0] - 2024-12-12
### Added
- Hive databases backed by S3 can now have versioning enabled.

## [7.7.0] - 2024-11-19
### Changed
- Updated the hms namespaces for metrics for both readwrite and readonly.
Expand Down
7 changes: 6 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,12 @@ module "apiary" {
admin_roles = "role1_arn,role2_arn" //kms key management will be restricted to these roles.
client_roles = "role3_arn,role4_arn" //s3 bucket read/write and kms key usage will be restricted to these roles.
customer_accounts = "account_id1,account_id2" //this will override module level apiary_customer_accounts
}
},
{
schema_name = "db_s3_versioning_enabled",
s3_versioning_enabled = "Enabled", // Enabled/Disabled/Suspended. Once enabled it can only be suspended
s3_versioning_expiration_days = 2 // If Enabled, default 7
},
]
apiary_customer_accounts = ["aws_account_no_1", "aws_account_no_2"]
# single policy with multiple conditions will use AND operator
Expand Down
4 changes: 2 additions & 2 deletions VARIABLES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@
| hms\_ecs\_metrics\_readonly\_namespace | ECS readwrite metrics namespace | `string` | `hmsreadonlylegacy` | no |
| hms\_ecs\_metrics\_readwrite\_namespace | ECS readonly metrics namespace | `string` | `hmsreadwritelegacy` | no |
| hms\_k8s\_metrics\_readonly\_namespace | K8s readwrite metrics namespace | `string` | `hms_readonly` | no |
| hms\_k8s\_metrics\_readwrite\_namespace | K8s readonly metrics namespace | `string` | `hms_readwrite` | no |
| s3\_versioning\_expiration\_days | Number of days (TTL) before objects are expired. Bucket need to have versioning enabled. | `number` | `7` | no |

### apiary_assume_roles

Expand Down Expand Up @@ -367,4 +367,4 @@ apiary_managed_schemas = [
producer_roles = "arn:aws:iam::000000000:role/role-1,arn:aws:iam::000000000:role/role-2"
}
]
```
```
26 changes: 26 additions & 0 deletions s3.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,32 @@ resource "aws_s3_bucket" "apiary_data_bucket" {
}
}

resource "aws_s3_bucket_versioning" "apiary_data_bucket_versioning" {
for_each = {
for schema in local.schemas_info : "${schema["schema_name"]}" => schema
}
bucket = each.value["data_bucket"]
versioning_configuration {
status = lookup(each.value, "s3_versioning_enabled", "Disabled")
}
}

resource "aws_s3_bucket_lifecycle_configuration" "apiary_data_bucket_versioning_lifecycle" {
for_each = {
for schema in local.schemas_info : "${schema["schema_name"]}" => schema
}
bucket = each.value["data_bucket"]
# Rule enabled when expiration max days is set
rule {
id = "expire-noncurrent-versions-days"
status = lookup(each.value, "s3_versioning_enabled", "") != "" ? "Enabled" : "Disabled"

noncurrent_version_expiration {
noncurrent_days = tonumber(lookup(each.value, "s3_versioning_expiration_days", var.s3_versioning_expiration_days))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question, should we only configure the version expiration? Or should we also configure the max version retain , like 1,2,3 ? If only expiration configured, all older version will be deleted after the days meet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replied privately, conclusion was to start only with days.

}
}
}

resource "aws_s3_bucket_inventory" "apiary_bucket" {
for_each = var.s3_enable_inventory == true ? {
for schema in local.schemas_info : "${schema["schema_name"]}" => schema
Expand Down
6 changes: 6 additions & 0 deletions variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1095,6 +1095,12 @@ variable "ecs_requires_compatibilities" {
default = ["EC2", "FARGATE"]
}

variable "s3_versioning_expiration_days" {
description = "Number of days (TTL) before objects are expired. Bucket need to have versioning enabled."
type = number
default = 7
}

variable "hms_ro_tolerations" {
description = <<EOF
Adds a list of tolerations for the HMS readonly pods. For example if you
Expand Down
Loading