Skip to content

Releases: ExpediaGroup/apiary-data-lake

v5.2.0: Added expiration days (if declared) to Apiary S3 bucket (#145)

23 Mar 15:25
4ea5711
Compare
Choose a tag to compare
* added bucket expiry days

* change variable declaration to number

* change variable declaration to number

* change variable declaration to number

* change variable declaration to number

* change variable declaration to number

* null variable check

* null variable check

* removed and set 365 days expiry

* test expiry days with checkpoint variable

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* test expiry days

* added expiry variable

* PR comment changes

* removed s3_object_expiration_days variable

* Update CHANGELOG.md

Co-Authored-By: Scott Barnhart <github@barnharts4.com>

* Update VARIABLES.md

Co-Authored-By: Adrian Woodhead <massdosage@gmail.com>

* Update VARIABLES.md

Co-Authored-By: Adrian Woodhead <massdosage@gmail.com>

* updated varibale.md & changelog.md

* updated varibale.md & changelog.md

* updated varibale.md & changelog.md

* Update VARIABLES.md

Co-Authored-By: Scott Barnhart <github@barnharts4.com>

* Update VARIABLES.md

Co-Authored-By: Scott Barnhart <github@barnharts4.com>

* Update release date in CHANGELOG.md

Co-authored-by: Baskar Jayakumar <bjayakumar@BELC02Q47S2G8WN.sea.corp.expecn.com>
Co-authored-by: Scott Barnhart <github@barnharts4.com>
Co-authored-by: Adrian Woodhead <massdosage@gmail.com>

Variable to configure S3 inventory output format

06 Mar 17:25
5c2a660
Compare
Choose a tag to compare
variable to configure s3 inventory output format (#141)

* variable to configure s3 inventory output format

* setting ORC as default

* update VARIABLES.md

Co-authored-by: Raj Poluri <rpoluri@expediagroup.com>
Co-authored-by: Scott Barnhart <sbarnhart@expedia.com>

v4.4.1

27 Feb 21:32
31e1fae
Compare
Choose a tag to compare
include optional fields in S3 inventory (#140)

v4.4.0: Manage S3 logs bucket. (#137)

12 Feb 19:38
66b6a18
Compare
Choose a tag to compare
* manage s3 logs bucket

* make apiary_log_bucket variable optional

* fix bucket logging

* fix bucket logging

* enable encryption and configure log expiry

* update changelog

* PR comment updates

Added Prometheus scrape annotations to Kubernetes deployments

10 Feb 18:02
f752406
Compare
Choose a tag to compare
Enable prometheus metrics when running on Kubernetes (#135)

* add prometheus scrape annotations

* only create cloudwatch dashboard when running on ecs

* update changelog

* Update CHANGELOG.md

Co-Authored-By: Adrian Woodhead <awoodhead@expediagroup.com>

Co-authored-by: Adrian Woodhead <massdosage@gmail.com>

v4.2.0: apiary kafka metastore listener (#134)

06 Feb 19:31
339b0b0
Compare
Choose a tag to compare
* configure kafka bootstrap url and topic name

* update kafka bootstrap variable name

* update changelog

* update changelog

Reduce k8s Hive Metastore process heapsize from 90 to 85 percent of container memory limit.

21 Nov 23:12
b535153
Compare
Choose a tag to compare
Reduce k8s Hive Metastore process heapsize from 90 to 85 percent (#128)

* Reduce k8s Hive Metastore process heapsize from 90 to 85 percent of container memory limit

* update changelog

Add variable to enable Atlas hive-bridge metastore listener.

18 Nov 21:51
Compare
Choose a tag to compare
Adding support of a new variable for Atlas hive sync (#127)

* Adding support of a new variable

Fix line

Fix syntax

* Pr comments

* Adding to changelog and variables.md

* PR comments

* Making zookeeper and kafka configurable

* PR comments - Removing extra full stop

* update changelog and variable names

* update changelog

* update changelog

deploy metastore services on kubernetes

13 Nov 19:22
ba7e7cd
Compare
Choose a tag to compare
deploy metastore services on kubernetes (#126)

* configure metastore route53 using load balancer alias

* k8s.tf

* add required environment variables

* add image_pull_secrets

* annotate k8s deployment with iam role

* add kiam assume policy

* default to admin arn when kiam_arn is blank

* add AWS_DEFAULT_REGION

* configure iam.amazonaws.com/role annotation as role name instead of arn

* move annotations to pod template

* add k8s readonly service

* fix ro iam role

* add hms-readwrite k8s service

* add hms_readonly k8s service

* configure k8s services as NodePort

* configure 90% memory as heapsize for k8s containers

* create LoadBalancers for metastore services

* configure metastore dns as cname of k8s load balancers

* Revert "configure metastore dns as cname of k8s load balancers"

This reverts commit 7182dbce39d7c7966a4ee396fedbf29f783cb434.

* configure metastore dns as cname of k8s load balancers

* set ttl to fix dns records

* cleanup ec2 deployment

* create kubernetes_deployment only when instance_type is k8s

* fix

* make ecs LBs optional

* cloudwatch fix

* route53 fix

* vpce fix

* make nlb optional in cloudwatch dashboards

* add k8s lb to vpc endpoint service

* update k8s lb reference

* update changelog and add load_balancer_source_ranges

* fix load_balancer_source_ranges

* add load_balancer_source_ranges to k8s-readwrite

* configure load_balancer_source_ranges and ingress_cidr

* update VARIABLES.md and license headers

* add env variables

* update changelog

* cleanup root_vol_type,root_vol_size variables

Add support for deploying on EC2

01 Jul 20:35
f952a92
Compare
Choose a tag to compare
Add limited support for running Hive Metastore in EC2. (#114)

* add instance_type variable and logic to disable ecs specific resources

* configure hive metastore on EC2 instances

* lambda to update hive db locations

* configure using SSM

* configure SNS event listener on EC2 nodes

* terraform fmt

* update changelog

* update variable.md

* PR comment updates

* PR comment updates

* add iam_name_root variable

* update iam role policy resource names

* add Name tag to S3 bucket

* update changelog