✨ ✨ Source Hubspot: unnest data #30322

davydov-d · 2023-09-11T20:16:41Z

What

#29345

How

Unnest static schemas
Unnest dynamic schemas
Unnest data itself to correspond the schemas
Update CAT and unit tests, bump version, update changelog

🚨 User Impact 🚨

Most of nested original data will be duplicated with unnested columns -- not a breaking change

github-actions · 2023-09-11T20:17:03Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2023-09-11T20:35:28Z

source-hubspot test report (commit `f5e68805ad`) - ❌

⏲️ Total pipeline duration: 13mn37s

Step	Result
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

github-actions · 2023-09-12T04:37:49Z

source-hubspot test report (commit `a3184a0e00`) - ✅

⏲️ Total pipeline duration: 08mn48s

Step	Result
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

github-actions · 2023-09-19T15:46:48Z

source-hubspot test report (commit `dde64d7fb4`) - ❌

⏲️ Total pipeline duration: 12mn03s

Step	Result
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

davydov-d · 2023-09-19T15:51:25Z

The breaking change doc is here https://docs.google.com/document/d/1kgwrtIMZSVLFPcfaGLfwX62-1ywiYZ8n779-7zLKGLo/edit

github-actions · 2023-09-19T16:11:51Z

source-hubspot test report (commit `c8c1a7103c`) - ❌

⏲️ Total pipeline duration: 10mn01s

Step	Result
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

davydov-d · 2023-09-19T16:17:46Z

acceptance tests are failing due to actual and expected items mismatch - will fix it soon

…om:airbytehq/airbyte into ddavydov/#29345-source-hubspot-unnest-data

github-actions · 2023-09-19T18:53:28Z

source-hubspot test report (commit `055c9a57a2`) - ❌

⏲️ Total pipeline duration: 05mn53s

Step	Result
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

github-actions · 2023-09-19T19:14:33Z

source-hubspot test report (commit `7dfc1bf937`) - ✅

⏲️ Total pipeline duration: 05mn54s

Step	Result
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

brianjlai

As per the issue I noticed that these schemas weren't modified. And that was because they are already de-nested?

line_items
products
property_history
tickets

The code looks good and I went over most of the schemas. We should add some unit tests for the new unnester class since it has some pretty specific logic as to how it performs the unnesting. Will approve once we add those tests in.

What's the release plan? Since it's a breaking change are we giving customers a warning first, and then a window for them to refresh their schema and data?

airbyte-integrations/connectors/source-hubspot/metadata.yaml

brianjlai · 2023-09-20T06:02:12Z

airbyte-integrations/connectors/source-hubspot/source_hubspot/schemas/engagements_emails.json

-          "type": ["null", "string"]
-        }
-      }
+    "properties_hs_all_assigned_business_unit_ids": {


not sure if natalie corrected this, but i did noticed you asked. I assume it was a typo and she meant properties as you have it

if I understand this right, this column is named correctly. Please check out this answer and see if it answers your question

airbyte-integrations/connectors/source-hubspot/source_hubspot/streams.py

vercel · 2023-09-25T10:45:24Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 25, 2023 6:54pm

airbyte-oss-build-runner · 2023-09-25T11:13:30Z

source-hubspot test report (commit `9f1181256d`) - ✅

⏲️ Total pipeline duration: 11mn57s

Step	Result
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

davydov-d · 2023-09-25T11:52:19Z

@brianjlai I have updated this PR to address your comments. Please check it out one more time.
Regarding your question about the four stream schemas that were not changed: they hardly have nested data except for the custom properties which are unnested dynamically. Also, the release plan is not needed as this is no more a breaking change. We decided to leave the original data as is, and duplicate it with the unnested columns.

airbyte-oss-build-runner · 2023-09-25T12:01:22Z

source-hubspot test report (commit `6e72ef3aa5`) - ❌

⏲️ Total pipeline duration: 11mn57s

Step	Result
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

airbyte-oss-build-runner · 2023-09-25T13:13:30Z

source-hubspot test report (commit `f299c9bb64`) - ✅

⏲️ Total pipeline duration: 13mn39s

Step	Result
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

airbyte-oss-build-runner · 2023-09-25T19:04:48Z

source-hubspot test report (commit `f2452adbbd`) - ✅

⏲️ Total pipeline duration: 09mn40s

Step	Result
Connector package install	✅
Build source-hubspot docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-hubspot/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-hubspot test

brianjlai

no changes on my end, this does feel less risky now that it is non-breaking and additive to the schema. Not a blocker, but do we plan to eventually deprecate these legacy nested fields. this could lead to almost double the size of some records which could impact performance or at least cost?

davydov-d · 2023-09-27T06:10:11Z

no changes on my end, this does feel less risky now that it is non-breaking and additive to the schema. Not a blocker, but do we plan to eventually deprecate these legacy nested fields. this could lead to almost double the size of some records which could impact performance or at least cost?

I am not aware of a plan like this. I think we should ask Kat or Natalie

unnest data

f72be6e

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/hubspot labels Sep 11, 2023

update changelog

f5e6880

davydov-d marked this pull request as ready for review September 11, 2023 20:18

davydov-d requested review from lazebnyi and darynaishchenko September 11, 2023 20:19

fix migration guide

a3184a0

davydov-d added the breaking-change Don't merge me unless you are ready. label Sep 12, 2023

lazebnyi approved these changes Sep 15, 2023

View reviewed changes

lazebnyi requested a review from brianjlai September 18, 2023 09:31

fix breaking change message

dde64d7

davydov-d added the checklist-action-run label Sep 19, 2023

davydov-d requested a review from a team September 19, 2023 15:40

Merge branch 'master' into ddavydov/#29345-source-hubspot-unnest-data

c8c1a71

davydov-d added 2 commits September 19, 2023 21:44

update expected records

ffacd98

Merge branch 'ddavydov/#29345-source-hubspot-unnest-data' of github.c…

055c9a5

…om:airbytehq/airbyte into ddavydov/#29345-source-hubspot-unnest-data

update expected records

7dfc1bf

brianjlai reviewed Sep 20, 2023

View reviewed changes

Merge branch 'master' into ddavydov/#29345-source-hubspot-unnest-data

79d3f4c

duplicate unnested data

9f11812

vercel bot deployed to Preview September 25, 2023 10:47 View deployment

davydov-d removed the breaking-change Don't merge me unless you are ready. label Sep 25, 2023

davydov-d changed the title ~~🚨 🚨 Source Hubspot: unnest data~~ ✨ ✨ Source Hubspot: unnest data Sep 25, 2023

add unit test and docstring

6e72ef3

davydov-d requested a review from brianjlai September 25, 2023 11:52

update expected records

f299c9b

update docs

f2452ad

vercel bot deployed to Preview September 25, 2023 18:54 View deployment

brianjlai approved these changes Sep 26, 2023

View reviewed changes

davydov-d merged commit 8186e80 into master Sep 27, 2023
21 checks passed

davydov-d deleted the ddavydov/#29345-source-hubspot-unnest-data branch September 27, 2023 06:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ ✨ Source Hubspot: unnest data #30322

✨ ✨ Source Hubspot: unnest data #30322

davydov-d commented Sep 11, 2023 •

edited

Loading

github-actions bot commented Sep 11, 2023 •

edited by davydov-d

Loading

github-actions bot commented Sep 11, 2023

github-actions bot commented Sep 12, 2023

github-actions bot commented Sep 19, 2023

davydov-d commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

davydov-d commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

brianjlai left a comment

brianjlai Sep 20, 2023

davydov-d Sep 25, 2023

vercel bot commented Sep 25, 2023 •

edited

Loading

airbyte-oss-build-runner commented Sep 25, 2023

davydov-d commented Sep 25, 2023

airbyte-oss-build-runner commented Sep 25, 2023

airbyte-oss-build-runner commented Sep 25, 2023

airbyte-oss-build-runner commented Sep 25, 2023

brianjlai left a comment

davydov-d commented Sep 27, 2023

✨ ✨ Source Hubspot: unnest data #30322

✨ ✨ Source Hubspot: unnest data #30322

Conversation

davydov-d commented Sep 11, 2023 • edited Loading

What

How

🚨 User Impact 🚨

github-actions bot commented Sep 11, 2023 • edited by davydov-d Loading

Before Merging a Connector Pull Request

github-actions bot commented Sep 11, 2023

source-hubspot test report (commit f5e68805ad) - ❌

github-actions bot commented Sep 12, 2023

source-hubspot test report (commit a3184a0e00) - ✅

github-actions bot commented Sep 19, 2023

source-hubspot test report (commit dde64d7fb4) - ❌

davydov-d commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

source-hubspot test report (commit c8c1a7103c) - ❌

davydov-d commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

source-hubspot test report (commit 055c9a57a2) - ❌

github-actions bot commented Sep 19, 2023

source-hubspot test report (commit 7dfc1bf937) - ✅

brianjlai left a comment

Choose a reason for hiding this comment

brianjlai Sep 20, 2023

Choose a reason for hiding this comment

davydov-d Sep 25, 2023

Choose a reason for hiding this comment

vercel bot commented Sep 25, 2023 • edited Loading

airbyte-oss-build-runner commented Sep 25, 2023

source-hubspot test report (commit 9f1181256d) - ✅

davydov-d commented Sep 25, 2023

airbyte-oss-build-runner commented Sep 25, 2023

source-hubspot test report (commit 6e72ef3aa5) - ❌

airbyte-oss-build-runner commented Sep 25, 2023

source-hubspot test report (commit f299c9bb64) - ✅

airbyte-oss-build-runner commented Sep 25, 2023

source-hubspot test report (commit f2452adbbd) - ✅

brianjlai left a comment

Choose a reason for hiding this comment

davydov-d commented Sep 27, 2023

davydov-d commented Sep 11, 2023 •

edited

Loading

github-actions bot commented Sep 11, 2023 •

edited by davydov-d

Loading

source-hubspot test report (commit `f5e68805ad`) - ❌

source-hubspot test report (commit `a3184a0e00`) - ✅

source-hubspot test report (commit `dde64d7fb4`) - ❌

source-hubspot test report (commit `c8c1a7103c`) - ❌

source-hubspot test report (commit `055c9a57a2`) - ❌

source-hubspot test report (commit `7dfc1bf937`) - ✅

vercel bot commented Sep 25, 2023 •

edited

Loading

source-hubspot test report (commit `9f1181256d`) - ✅

source-hubspot test report (commit `6e72ef3aa5`) - ❌

source-hubspot test report (commit `f299c9bb64`) - ✅

source-hubspot test report (commit `f2452adbbd`) - ✅