[airbyte_connection] Error when setting "selected = false" at the sync_catalog

[PierreKerschgens]

Hi @eabrouwer3,

I'm trying to create a connection for a source where we don't want to sync every column from.

Since the sync_catalog.destination_config is mutable I tried to deselect specific columns with selected = false but it leads to this error:

Terraform will perform the following actions:

  # airbyte_connection.salesforce_to_destination will be updated in-place
  ~ resource "airbyte_connection" "salesforce_to_destination" {
      + breaking_change      = (known after apply)
      + geography            = (known after apply)
        id                   = "d8a4f9e1-anon-anon-anon-6bd166849e4b"
        name                 = "Salesforce <> Destination"
      ~ operation_ids        = [] -> (known after apply)
      ~ sync_catalog         = [
          ~ {
              ~ destination_config = {
                  ~ selected              = true -> false
                    # (5 unchanged attributes hidden)
                }
                # (1 unchanged attribute hidden)
            },
          ~ {
              ~ destination_config = {
                  ~ selected              = true -> false
                    # (5 unchanged attributes hidden)
                }
                # (1 unchanged attribute hidden)
            },
          ~ {
              ~ destination_config = {
                  ~ selected              = true -> false
                    # (5 unchanged attributes hidden)
                }
                # (1 unchanged attribute hidden)
            },
          ~ {
              ~ destination_config = {
                  ~ selected              = true -> false
                    # (5 unchanged attributes hidden)
                }
                # (1 unchanged attribute hidden)
            },
            # (18 unchanged elements hidden)
        ]
        # (7 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

airbyte_connection.salesforce_to_destination: Modifying... [id=d8a4f9e1-anon-anon-anon-6bd166849e4b]
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to airbyte_connection.salesforce_to_destination, provider "provider[\"registry.terraform.io/eabrouwer3/airbyte\"]" produced an unexpected new value: .sync_catalog: element 18 has vanished.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to airbyte_connection.salesforce_to_destination, provider "provider[\"registry.terraform.io/eabrouwer3/airbyte\"]" produced an unexpected new value: .sync_catalog: element 19 has vanished.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to airbyte_connection.salesforce_to_destination, provider "provider[\"registry.terraform.io/eabrouwer3/airbyte\"]" produced an unexpected new value: .sync_catalog: element 20 has vanished.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to airbyte_connection.salesforce_to_destination, provider "provider[\"registry.terraform.io/eabrouwer3/airbyte\"]" produced an unexpected new value: .sync_catalog: element 21 has vanished.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

[PierreKerschgens]

If someone is struggling with selecting specific columns as well I have a workaround.

In this example I just want to sync the columns 'Account' and 'AccountHistory' from Salesforce and 'contacts' from Hubspot.

variables.tf:

variable "airbyte_sync_catalog" {
  description = "Selection of tables to sync"
  type = map(
    object({
      tables = map(object({
        destination_sync_mode = string,
        sync_mode             = string,
        cursor_field          = list(string),
        primary_key           = list(list(string)),
      }))
  }))
}

dev.tfvars:

airbyte_sync_catalog = {
  salesforce = {
    tables = {
      Account        = { destination_sync_mode = "append", sync_mode = "incremental", cursor_field = ["SystemModstamp"], primary_key = [[""]] }
      AccountHistory = { destination_sync_mode = "append", sync_mode = "incremental", cursor_field = ["CreatedDate"], primary_key = [[""]] }
    }
  },
  hubspot = {
    tables = {
      contacts = { destination_sync_mode = "append", sync_mode = "incremental", cursor_field = ["updatedAt"], primary_key = [[""]] }
    }
  },
}

locals.tf:

locals {
  sync_catalog = {
    for source_db_key, source_db in var.airbyte_sync_catalog : source_db_key => {
      for table_name, table_cfg in source_db.tables : table_name => {
        alias_name            = table_name
        cursor_field          = table_cfg.cursor_field
        destination_sync_mode = table_cfg.destination_sync_mode
        primary_key           = table_cfg.primary_key
        selected              = true
        sync_mode             = table_cfg.sync_mode
      }
    }
  }
}

conf_connection.tf:

##################################
# Salesforce <> Destination
##################################

data "airbyte_source_schema_catalog" "salesforce_schema_catalog" {
  source_id = airbyte_source.salesforce.id # missing in example
}

resource "airbyte_connection" "salesforce_to_destination" {
  destination_id       = airbyte_destination.destination.id # missing in example
  name                 = "Salesforce <> Destination"
  namespace_definition = "customformat"
  namespace_format     = "salesforce"
  schedule_type        = "cron"
  source_id            = airbyte_source.salesforce.id # missing in example
  status               = "active"

  # To sync specific fields for this connection the source schema must be reduced to the desired values of var.airbyte_sync_catalog
  # setting 'selected = false' breaks the provider and generating this list in the locals creates circular dependency
  #
  # when copying this for different connection change every occurence of:
  # 2x ["salesforce"] and 2x data.airbyte_source_schema_catalog.salesforce_schema_catalog.sync_catalog

  sync_catalog = flatten([
    for database, tables in local.sync_catalog["salesforce"] :
    merge(
      data.airbyte_source_schema_catalog.salesforce_schema_catalog.sync_catalog[index(data.airbyte_source_schema_catalog.salesforce_schema_catalog.sync_catalog.*.source_schema.name, tables.alias_name)],
      { "destination_config" : local.sync_catalog["salesforce"][tables.alias_name] }
    )
  ])

  cron_schedule = {
    cron_expression = "0 15 3 ? * *"
    cron_time_zone  = "UTC"
  }
}

##################################
# Hubspot <> Destination
##################################

data "airbyte_source_schema_catalog" "hubspot_schema_catalog" {
  source_id = airbyte_source.hubspot.id # missing in example
}

resource "airbyte_connection" "hubspot_to_destination" {
  destination_id       = airbyte_destination.destination.id # missing in example
  name                 = "Hubspot <> Destination"
  namespace_definition = "customformat"
  namespace_format     = "hubspot"
  schedule_type        = "cron"
  source_id            = airbyte_source.hubspot.id # missing in example
  status               = "active"

  # To sync specific fields for this connection the source schema must be reduced to the desired values of var.airbyte_sync_catalog
  # setting 'selected = false' breaks the provider and generating this list in the locals creates circular dependency
  #
  # when copying this for different connection change every occurence of:
  # 2x ["hubspot"] and 2x data.airbyte_source_schema_catalog.hubspot_schema_catalog.sync_catalog

  sync_catalog = flatten([
    for database, tables in local.sync_catalog["hubspot"] :
    merge(
      data.airbyte_source_schema_catalog.hubspot_schema_catalog.sync_catalog[index(data.airbyte_source_schema_catalog.hubspot_schema_catalog.sync_catalog.*.source_schema.name, tables.alias_name)],
      { "destination_config" : local.sync_catalog["hubspot"][tables.alias_name] }
    )
  ])

  cron_schedule = {
    cron_expression = "0 10 5 ? * *"
    cron_time_zone  = "UTC"
  }
}

[eabrouwer3]

@PierreKerschgens this should be fixed in the latest version of the library (0.1.19). If it's not, lmk. The Airbyte API is kinda weird with this where instead of returning everything in the response, it literally removes the ones that aren't selected.

[PierreKerschgens]

Hey @eabrouwer3. I updated to version 0.1.19 and when I use "selected = false" I get tons of these kind of errors:

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to airbyte_connection.salesforce_to_s3_yellow_raw_partitions, provider
│ "provider[\"registry.terraform.io/eabrouwer3/airbyte\"]" produced an unexpected new value: .sync_catalog[20].source_schema.name: was
│ cty.StringVal("table1__c"), but now cty.StringVal("table2__c").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to airbyte_connection.salesforce_to_s3_yellow_raw_partitions, provider
│ "provider[\"registry.terraform.io/eabrouwer3/airbyte\"]" produced an unexpected new value: .sync_catalog: element 21 has vanished.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

I'm very thankful for you effort!