[Source-Mysql] Set default cursor value for cdc mode #28756

nguyenaiden · 2023-07-26T21:35:48Z

What

27340
To prepare for Destination v2, Cdc-enabled DB sources need to have a pre-defined cursor for normalization to operate properly.

How

During the discover phase, source_defined_cursor is set to true, and a new composite cursor column called _ab_cdc_cursor is selected as the predefined cursor. This column is constructed from:

Converting the emittedAt timestamp , generated at the read step, to epoch seconds.
Convert that to nanoseconds.
Initialize a threadsafe recordCounter = 1
Add the recordCounter to the converted timestamp
Increment the counter

The numeric cursor value will be computed for every record and allows for fast comparison in tie-breaking scenarios for records with the same Primary Key.

Updated Streamstate
[
    {
      "streamDescriptor": {
        "name": "users",
        "namespace": "1gb"
      },
      "streamState": {
        "stream_name": "users",
        "cursor_field": [
          "_ab_cdc_cursor"
        ],
        "stream_namespace": "1gb"
      }
    }
  ]

🚨 User Impact 🚨

New CDC syncs that have refreshed their source schema will see this cursor field be chosen.
This update requires that users do a full reset OR drop their SCD tables to ensure that syncs continue operating normally
Customers who reset their data will continue to see old behavior until their source goes through the discover() phase again.

github-actions · 2023-07-26T21:36:12Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

octavia-squidington-iii · 2023-07-26T21:58:29Z

source-mysql test report (commit `9256d31864`) - ❌

⏲️ Total pipeline duration: 21mn10s

Step	Result
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest	❌
Acceptance tests	✅


		public class MySqlCdcConnectorMetadataInjector implements CdcMetadataInjector {

		final Instant emittedAt;

                   properties.set(CDC_LOG_POS, numberType);
                   properties.set(CDC_UPDATED_AT, stringType);
                   properties.set(CDC_DELETED_AT, stringType);
+                  properties.set(CDC_DEFAULT_CURSOR, stringType);

                 @Override
                 public void addMetaData(final ObjectNode event, final JsonNode source) {
+                  final String cdcDefaultCursor =
+                      String.format("%s_%s_%s", emittedAt.toString(), source.get("file").asText(), source.get("pos").asText());

[Source-Mysql] Set default cursor value for cdc mode #28756

[Source-Mysql] Set default cursor value for cdc mode #28756

Conversation

nguyenaiden commented Jul 26, 2023 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

github-actions bot commented Jul 26, 2023 • edited by nguyenaiden Loading

Before Merging a Connector Pull Request

octavia-squidington-iii commented Jul 26, 2023

source-mysql test report (commit 9256d31864) - ❌

octavia-squidington-iii commented Jul 26, 2023

source-mysql-strict-encrypt test report (commit 9256d31864) - ❌

octavia-squidington-iii commented Jul 26, 2023

source-mysql-strict-encrypt test report (commit 9256d31864) - ❌

octavia-squidington-iii commented Jul 26, 2023

source-mysql test report (commit 9256d31864) - ❌

subodh1810 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

octavia-squidington-iii commented Jul 31, 2023

source-mysql-strict-encrypt test report (commit 319a2e9ec3) - ✅

octavia-squidington-iii commented Jul 31, 2023

source-mysql test report (commit 319a2e9ec3) - ❌

octavia-squidington-iii commented Jul 31, 2023

source-mysql test report (commit f9a9dd0dc6) - ✅

octavia-squidington-iii commented Jul 31, 2023

source-mysql-strict-encrypt test report (commit f9a9dd0dc6) - ❌

edgao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nguyenaiden Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akashkulk left a comment

Choose a reason for hiding this comment

octavia-squidington-iii commented Aug 2, 2023

source-mysql test report (commit 6d6a24ddfa) - ❌

octavia-squidington-iii commented Aug 2, 2023

source-mysql-strict-encrypt test report (commit 6d6a24ddfa) - ❌

subodh1810 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

octavia-squidington-iii commented Aug 3, 2023

source-mysql test report (commit 7d9ac53601) - ❌

nguyenaiden commented Aug 17, 2023 • edited by github-actions bot Loading

Build Passed

octavia-squidington-iii commented Aug 17, 2023

source-mysql test report (commit e2417f7d62) - ❌

octavia-squidington-iii commented Aug 17, 2023

source-mysql-strict-encrypt test report (commit e2417f7d62) - ❌

nguyenaiden commented Aug 17, 2023 • edited by github-actions bot Loading

Build Passed

octavia-squidington-iii commented Aug 17, 2023

source-mysql-strict-encrypt test report (commit 12a6b9976e) - ❌

nguyenaiden commented Aug 17, 2023

octavia-approvington commented Aug 17, 2023

octavia-squidington-iii commented Aug 17, 2023

source-mysql test report (commit 12a6b9976e) - ❌

nguyenaiden commented Jul 26, 2023 •

edited

Loading

github-actions bot commented Jul 26, 2023 •

edited by nguyenaiden

Loading

source-mysql test report (commit `9256d31864`) - ❌

source-mysql-strict-encrypt test report (commit `9256d31864`) - ❌

source-mysql-strict-encrypt test report (commit `9256d31864`) - ❌

source-mysql test report (commit `9256d31864`) - ❌

source-mysql-strict-encrypt test report (commit `319a2e9ec3`) - ✅

source-mysql test report (commit `319a2e9ec3`) - ❌

source-mysql test report (commit `f9a9dd0dc6`) - ✅

source-mysql-strict-encrypt test report (commit `f9a9dd0dc6`) - ❌

nguyenaiden Aug 1, 2023 •

edited

Loading

source-mysql test report (commit `6d6a24ddfa`) - ❌

source-mysql-strict-encrypt test report (commit `6d6a24ddfa`) - ❌

source-mysql test report (commit `7d9ac53601`) - ❌

nguyenaiden commented Aug 17, 2023 •

edited by github-actions bot

Loading

source-mysql test report (commit `e2417f7d62`) - ❌

source-mysql-strict-encrypt test report (commit `e2417f7d62`) - ❌

nguyenaiden commented Aug 17, 2023 •

edited by github-actions bot

Loading

source-mysql-strict-encrypt test report (commit `12a6b9976e`) - ❌

source-mysql test report (commit `12a6b9976e`) - ❌