feat: added the ability to use the default IBM Cloud® Databases rando…

…mly generated keys for disk and backups encryption in the fscloud submodule, and the DA using new input `use_ibm_owned_encryption_key`<br>* Exposed the ability to set the IBM terraform provider visability in the DA using new input `provider_visibility` ([Learn more](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/guides/custom-service-endpoints))<br>* updated the scope of the KMS auth policy that is created so the policy is now scoped to the exact KMS key. If upgrading from a previous version, this will destroy the old policy, however the new one will be created before its destroyed to ensure no impact to every day services.<br>* Fixed an issue in the original ICD hosting model logic for conditionally including the group block during a database restore operation (#330)

README.md

@@ -91,7 +91,7 @@ You need the following permissions to run this module.
 | <a name="input_admin_pass"></a> [admin\_pass](#input\_admin\_pass) | The password for the database administrator. If the admin password is null, the admin user ID cannot be accessed. You can specify more users in a user block. | `string` | `null` | no |
 | <a name="input_auto_scaling"></a> [auto\_scaling](#input\_auto\_scaling) | The rules to allow the database to increase resources in response to usage. Only a single autoscaling block is allowed. Make sure you understand the effects of autoscaling, especially for production environments. [Learn more](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-autoscaling&interface=cli#autoscaling-considerations). | <pre>object({<br/>    disk = object({<br/>      capacity_enabled             = optional(bool, false)<br/>      free_space_less_than_percent = optional(number, 10)<br/>      io_above_percent             = optional(number, 90)<br/>      io_enabled                   = optional(bool, false)<br/>      io_over_period               = optional(string, "15m")<br/>      rate_increase_percent        = optional(number, 10)<br/>      rate_limit_mb_per_member     = optional(number, 3670016)<br/>      rate_period_seconds          = optional(number, 900)<br/>      rate_units                   = optional(string, "mb")<br/>    })<br/>    memory = object({<br/>      io_above_percent         = optional(number, 90)<br/>      io_enabled               = optional(bool, false)<br/>      io_over_period           = optional(string, "15m")<br/>      rate_increase_percent    = optional(number, 10)<br/>      rate_limit_mb_per_member = optional(number, 114688)<br/>      rate_period_seconds      = optional(number, 900)<br/>      rate_units               = optional(string, "mb")<br/>    })<br/>  })</pre> | `null` | no |
 | <a name="input_backup_crn"></a> [backup\_crn](#input\_backup\_crn) | The CRN of a backup resource to restore from. The backup is created by a database deployment with the same service ID. The backup is loaded after both provisioning is complete and the new deployment that uses that data starts. Specify a backup CRN is in the format `crn:v1:<...>:backup:`. If not specified, the database is provisioned empty. | `string` | `null` | no |
-| <a name="input_backup_encryption_key_crn"></a> [backup\_encryption\_key\_crn](#input\_backup\_encryption\_key\_crn) | The CRN of a KMS (Key Protect or Hyper Protect Crypto Services) key to use for encrypting the disk that holds deployment backups. Applies only if `kms_encryption_enabled` is true. Limitations exist for regions. For more information, see [Key Protect integration](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-key-protect&interface=ui#key-byok) or [Hyper Protect Crypto Services integration](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-hpcs#use-hpcs-backups). | `string` | `null` | no |
+| <a name="input_backup_encryption_key_crn"></a> [backup\_encryption\_key\_crn](#input\_backup\_encryption\_key\_crn) | The CRN of a Hyper Protect Crypto Services use for encrypting the disk that holds deployment backups. There are limitation per region on the Hyper Protect Crypto Services and region for those services. See https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-hpcs#use-hpcs-backups | `string` | `null` | no |
 | <a name="input_cbr_rules"></a> [cbr\_rules](#input\_cbr\_rules) | The list of context-based restriction rules to create. | <pre>list(object({<br/>    description = string<br/>    account_id  = string<br/>    rule_contexts = list(object({<br/>      attributes = optional(list(object({<br/>        name  = string<br/>        value = string<br/>    }))) }))<br/>    enforcement_mode = string<br/>  }))</pre> | `[]` | no |
 | <a name="input_elasticsearch_version"></a> [elasticsearch\_version](#input\_elasticsearch\_version) | The version of Databases for Elasticsearch to deploy. Possible values: `8.7`, `8.10`, `8.12`, `8.15` which requires an Enterprise Platinum pricing plan. If no value is specified, the current preferred version for IBM Cloud Databases is used. | `string` | `null` | no |
 | <a name="input_elser_model_type"></a> [elser\_model\_type](#input\_elser\_model\_type) | Trained ELSER model to be used for Elastic's Natural Language Processing. Possible values: `.elser_model_1`, `.elser_model_2` and `.elser_model_2_linux-x86_64`. [Learn more](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html) | `string` | `".elser_model_2_linux-x86_64"` | no |
@@ -112,7 +112,7 @@ You need the following permissions to run this module.
 | <a name="input_service_endpoints"></a> [service\_endpoints](#input\_service\_endpoints) | The type of endpoint of the database instance. Possible values: `public`, `private`, `public-and-private`. | `string` | `"public"` | no |
 | <a name="input_skip_iam_authorization_policy"></a> [skip\_iam\_authorization\_policy](#input\_skip\_iam\_authorization\_policy) | Whether to create an IAM authorization policy that permits all Databases for Elasticsearch instances in the resource group to read the encryption key from the Hyper Protect Crypto Services instance specified in the `existing_kms_instance_guid` variable. If set to `false`, specify a value for the KMS instance in the `existing_kms_instance_guid` variable. No policy is created if `kms_encryption_enabled` is false. | `bool` | `false` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | The list of tags to be added to the Databases for Elasticsearch instance. | `list(string)` | `[]` | no |
-| <a name="input_use_default_backup_encryption_key"></a> [use\_default\_backup\_encryption\_key](#input\_use\_default\_backup\_encryption\_key) | Whether to use the IBM Cloud Databases generated keys. | `bool` | `false` | no |
+| <a name="input_use_default_backup_encryption_key"></a> [use\_default\_backup\_encryption\_key](#input\_use\_default\_backup\_encryption\_key) | Whether to use the IBM Cloud Databases generated keys for backup encryption. | `bool` | `false` | no |
 | <a name="input_users"></a> [users](#input\_users) | The list of users that have access to the database. Multiple blocks are allowed. The user password must be 10-32 characters. In most cases, you can use IAM service credentials (by specifying `service_credential_names`) to control access to the database instance. This block creates native database users. [Learn more](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-user-management&interface=ui). | <pre>list(object({<br/>    name     = string<br/>    password = string # pragma: allowlist secret<br/>    type     = optional(string)<br/>    role     = optional(string)<br/>  }))</pre> | `[]` | no |
 
 ### Outputs

cra-config.yaml

@@ -6,5 +6,7 @@ CRA_TARGETS:
     PROFILE_ID: "bfacb71d-4b84-41ac-9825-e8a3a3eb7405" # SCC profile ID (currently set to IBM Cloud Framework for Financial Services 1.6.0 profile).
     CRA_ENVIRONMENT_VARIABLES:
       TF_VAR_existing_kms_instance_crn: "crn:v1:bluemix:public:hs-crypto:us-south:a/abac0df06b644a9cabc6e44f55b3880e:e6dce284-e80f-46e1-a3c1-830f7adff7a9::"
+      TF_VAR_kms_key_crn: "crn:v1:bluemix:public:hs-crypto:us-south:a/abac0df06b644a9cabc6e44f55b3880e:e6dce284-e80f-46e1-a3c1-830f7adff7a9:key:76170fae-4e0c-48c3-8ebe-326059ebb533"
+      TF_VAR_provider_visibility: "public"
       TF_VAR_resource_group_name: "test-es-cra"
-      TF_VAR_use_existing_resource_group: "false"
+      TF_VAR_use_existing_resource_group: false

examples/fscloud/variables.tf

@@ -104,7 +104,7 @@ variable "backup_crn" {
 
 variable "backup_encryption_key_crn" {
   type        = string
-  description = "The CRN of a Hyper Protect Crypto Services use for encrypting the disk that holds deployment backups. Only used if var.kms_encryption_enabled is set to true. There are limitation per region on the Hyper Protect Crypto Services and region for those services. See https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-hpcs#use-hpcs-backups"
+  description = "The CRN of a Hyper Protect Crypto Services use for encrypting the disk that holds deployment backups. There are limitation per region on the Hyper Protect Crypto Services and region for those services. See https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-hpcs#use-hpcs-backups"
   default     = null
   # Validation happens in the root module
 }

ibm_catalog.json

@@ -89,6 +89,23 @@
                 {
                   "key": "ibmcloud_api_key"
                 },
+                {
+                  "key": "provider_visibility",
+                  "options": [
+                    {
+                      "displayname": "private",
+                      "value": "private"
+                    },
+                    {
+                      "displayname": "public",
+                      "value": "public"
+                    },
+                    {
+                      "displayname": "public-and-private",
+                      "value": "public-and-private"
+                    }
+                  ]
+                },
                 {
                   "key": "use_existing_resource_group"
                 },
@@ -190,6 +207,9 @@
                 {
                   "key": "access_tags"
                 },
+                {
+                  "key": "use_ibm_owned_encryption_key"
+                },
                 {
                   "key": "tags"
                 },

main.tf

@@ -25,24 +25,52 @@ locals {
   # Determine if host_flavor is used
   host_flavor_set = var.member_host_flavor != null ? true : false
 
-  # Determine what KMS service is being used for database encryption
-  kms_service = var.kms_key_crn != null ? (
-    can(regex(".*kms.*", var.kms_key_crn)) ? "kms" : (
-      can(regex(".*hs-crypto.*", var.kms_key_crn)) ? "hs-crypto" : "unrecognized key type"
-    )
-  ) : "no key crn"
-
   create_kp_auth_policy = var.kms_encryption_enabled == false || var.skip_iam_authorization_policy ? 0 : 1
+
+  parsed_kms_key_crn = var.kms_key_crn != null ? split(":", var.kms_key_crn) : []
+  kms_service        = length(local.parsed_kms_key_crn) > 0 ? local.parsed_kms_key_crn[4] : null
+  kms_scope          = length(local.parsed_kms_key_crn) > 0 ? local.parsed_kms_key_crn[6] : null
+  kms_account_id     = length(local.parsed_kms_key_crn) > 0 ? split("/", local.kms_scope)[1] : null
+  kms_key_id         = length(local.parsed_kms_key_crn) > 0 ? local.parsed_kms_key_crn[9] : null
 }
 
 # Create IAM Access Policy to allow Key protect to access Elasticsearch instance
 resource "ibm_iam_authorization_policy" "policy" {
-  count                       = local.create_kp_auth_policy
-  source_service_name         = "databases-for-elasticsearch"
-  source_resource_group_id    = var.resource_group_id
-  target_service_name         = local.kms_service
-  target_resource_instance_id = var.existing_kms_instance_guid
-  roles                       = ["Reader"]
+  count                    = local.create_kp_auth_policy
+  source_service_name      = "databases-for-elasticsearch"
+  source_resource_group_id = var.resource_group_id
+  roles                    = ["Reader"]
+  description              = "Allow all Elastic Search instances in the resource group ${var.resource_group_id} to read the ${local.kms_service} key ${local.kms_key_id} from the instance GUID ${var.existing_kms_instance_guid}"
+  resource_attributes {
+    name     = "serviceName"
+    operator = "stringEquals"
+    value    = local.kms_service
+  }
+  resource_attributes {
+    name     = "accountId"
+    operator = "stringEquals"
+    value    = local.kms_account_id
+  }
+  resource_attributes {
+    name     = "serviceInstance"
+    operator = "stringEquals"
+    value    = var.existing_kms_instance_guid
+  }
+  resource_attributes {
+    name     = "resourceType"
+    operator = "stringEquals"
+    value    = "key"
+  }
+  resource_attributes {
+    name     = "resource"
+    operator = "stringEquals"
+    value    = local.kms_key_id
+  }
+  # Scope of policy now includes the key, so ensure to create new policy before
+  # destroying old one to prevent any disruption to every day services.
+  lifecycle {
+    create_before_destroy = true
+  }
 }
 
 # workaround for https://github.com/IBM-Cloud/terraform-provider-ibm/issues/4478
@@ -83,9 +111,9 @@ resource "ibm_database" "elasticsearch" {
   ## This is used to conditionally add one, OR, the other group block depending on var.local.host_flavor_set
   ## This block is for if host_flavor IS set to specific pre-defined host sizes and not set to "multitenant"
   dynamic "group" {
-    for_each = local.host_flavor_set && var.member_host_flavor != "multitenant" ? [1] : []
+    for_each = local.host_flavor_set && var.member_host_flavor != "multitenant" && var.backup_crn == null ? [1] : []
     content {
-      group_id = "member" # Only member type is allowed for postgresql
+      group_id = "member" # Only member type is allowed for elasticsearch
       host_flavor {
         id = var.member_host_flavor
       }
@@ -100,9 +128,9 @@ resource "ibm_database" "elasticsearch" {
 
   ## This block is for if host_flavor IS set to "multitenant"
   dynamic "group" {
-    for_each = local.host_flavor_set && var.member_host_flavor == "multitenant" ? [1] : []
+    for_each = local.host_flavor_set && var.member_host_flavor == "multitenant" && var.backup_crn == null ? [1] : []
     content {
-      group_id = "member" # Only member type is allowed for postgresql
+      group_id = "member" # Only member type is allowed for elasticsearch
       host_flavor {
         id = var.member_host_flavor
       }
@@ -123,9 +151,9 @@ resource "ibm_database" "elasticsearch" {
 
   ## This block is for if host_flavor IS NOT set
   dynamic "group" {
-    for_each = local.host_flavor_set ? [] : [1]
+    for_each = local.host_flavor_set == false && var.backup_crn == null ? [1] : []
     content {
-      group_id = "member" # Only member type is allowed for postgresql
+      group_id = "member" # Only member type is allowed for elasticsearch
       memory {
         allocation_mb = var.member_memory_mb
       }
@@ -180,6 +208,8 @@ resource "ibm_database" "elasticsearch" {
 
   timeouts {
     create = "120m" #Extending provisioning time to 120 minutes
+    update = "120m"
+    delete = "15m"
   }
 }