feat(source-stripe): ref stream `Payment Methods` #44891

artem1205 · 2024-08-29T11:59:24Z

What

Resolve https://github.com/airbytehq/product-request-backlog/issues/234

ref stream Payment Methods to use customer Payment Methods endpoint

How

ref stream Payment Methods
bump CDK to v5

Review guide

airbyte-integrations/connectors/source-stripe/source_stripe/source.py
airbyte-integrations/connectors/source-stripe/source_stripe/schemas/payment_methods.json

User Impact

Breaking change due to schema changes in customer field: "object" -> "string"

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

[no ci] Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

vercel · 2024-08-29T11:59:28Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Sep 11, 2024 11:45am

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

maxi297 · 2024-08-29T19:42:41Z

airbyte-integrations/connectors/source-stripe/integration_tests/abnormal_state.json

+  {
+    "type": "STREAM",
+    "stream": {
+      "stream_state": { "customer_updated": 10000000000 },


Should the state be per partition here? It seems like we are at risk of losing records if it's not

Should the state be per partition here?

I don't think so because ParentIncrementalStipeSubStream class is used here. Thus, Customers are being synced incrementally and we'll get payment_methods only for the updated ones.

Oh! So we have validated that when a payment method is updated, the client updated field is also updated. Can we document this somewhere?

Also for the incremental part, shouldn't we rely on payment_method.* events instead? https://docs.stripe.com/api/events/types#event_types-payment_method.attached

if we did validate the assumption, can it be documented in the code so we have a reference? else we'll assume we're at risk of losing records

So we have validated that when a payment method is updated, the client updated field is also updated.

Tested this, customer is NOT being updated on payment_method update.

Also for the incremental part, shouldn't we rely on payment_method.* events instead?

Agreed, we should use this.

Moving PR to draft, need to change the implementation:
1st sync/Full refresh -- read from customers/{customer_id}/payment_methods
2nd sync/Incremental -- read from events stream with type = payment_method.*

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

…ripe-customer-payment-methods

girarda · 2024-08-30T15:43:31Z

airbyte-integrations/connectors/source-stripe/integration_tests/abnormal_state.json

+  {
+    "type": "STREAM",
+    "stream": {
+      "stream_state": { "customer_updated": 10000000000 },


if we did validate the assumption, can it be documented in the code so we have a reference? else we'll assume we're at risk of losing records

girarda · 2024-08-30T15:44:20Z

airbyte-integrations/connectors/source-stripe/source_stripe/source.py

@@ -483,6 +483,18 @@ def streams(self, config: MutableMapping[str, Any]) -> List[Stream]:
                sub_items_attr="refunds",
                **args,
            ),
+            ParentIncrementalStipeSubStream(
+                name="customer_payment_methods",


maybe a hot take, but I think we should just replace the existing payment_methods stream. It's not useful as far as I could tell

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

…ripe-customer-payment-methods # Conflicts: # airbyte-integrations/connectors/source-stripe/metadata.yaml # airbyte-integrations/connectors/source-stripe/pyproject.toml # docs/integrations/sources/stripe.md

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

…ripe-customer-payment-methods

girarda · 2024-09-04T16:59:44Z

airbyte-integrations/connectors/source-stripe/metadata.yaml

@@ -43,6 +43,12 @@ data:
        scopedImpact:
          - scopeType: stream
            impactedScopes: ["refunds"]
+      6.0.0:


does this need to be a major version bump? this feels like something we should notify users in the docs and migration guide, but I don't think it's worth stopping their syncs

customer was set as object, and now it is changed to string.
This is the main reason to mark changes as breaking.
endpoint change can also affect customer incremental sync.

shouldn't this be part of the migration guide then?

🌶️ I'd still advocate for not flagging as breaking change since only one Cloud user actually has data for this stream

ok, reverted.

shouldn't this be part of the migration guide then?

Since i removed this version from breakingChanges in metadata, CI fails on QA checks in migration guide.
Is there any other place I can place the release notes?

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

maxi297

I'm fine with the PR in the current state. If possible, I think we should add the payment method for another customer in our CATs but I'm not blocking the PR for that

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

girarda

, assuming regression tests pass for the impacted streams (Persons, Events

girarda · 2024-09-09T18:58:12Z

@artem1205 ran regression tests.

…ripe-customer-payment-methods

…ripe-customer-payment-methods # Conflicts: # airbyte-integrations/connectors/source-stripe/metadata.yaml # airbyte-integrations/connectors/source-stripe/poetry.lock # airbyte-integrations/connectors/source-stripe/pyproject.toml # airbyte-integrations/connectors/source-stripe/unit_tests/test_source.py # docs/integrations/sources/stripe.md

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

…205/source-stripe-customer-payment-methods

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

artem1205 · 2024-09-10T19:57:12Z

~~Regression tests~~ ~~after removing python package stripe~~ #45348 -- now relevant

UPD:
Regression tests after removing python package stripe #45348

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

girarda · 2024-09-11T15:41:24Z

/approve-and-merge reason="There is a migration guide, but no breaking change"

octavia-approvington · 2024-09-11T15:41:58Z

Approve this

aaronsteers · 2024-09-11T17:21:40Z

airbyte-integrations/connectors/source-stripe/source_stripe/source.py

@@ -109,7 +108,7 @@ def check_connection(self, logger: logging.Logger, config: MutableMapping[str, A
        args = self._get_stream_base_args(config)
        account_stream = StripeStream(name="accounts", path="accounts", use_cache=USE_CACHE, **args)
        try:
-            next(account_stream.read_records(sync_mode=SyncMode.full_refresh))
+            next(account_stream.read_records(sync_mode=SyncMode.full_refresh), None)


This fixes:

https://github.com/airbytehq/oncall/issues/6449

Source Stripe: add new stream Customer Payment Methods

db6a787

[no ci] Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

artem1205 self-assigned this Aug 29, 2024

octavia-squidington-iii added area/connectors Connector related issues connectors/source/stripe labels Aug 29, 2024

artem1205 added 4 commits August 29, 2024 16:49

Source Stripe: migrate to incremental substream

9e576a9

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Source Stripe: add to catalog

b9b9468

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Source Stripe: bump version + update docs

1c8d80d

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Source Stripe: update openapi spec

790be8a

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Aug 29, 2024

Source Stripe: fix unit tests + bump pytest

87188b8

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

artem1205 marked this pull request as ready for review August 29, 2024 18:48

artem1205 requested a review from a team August 29, 2024 19:32

maxi297 reviewed Aug 29, 2024

View reviewed changes

artem1205 added 2 commits August 30, 2024 11:09

Source Stripe: fmt

3147444

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Merge remote-tracking branch 'origin/master' into artem1205/source-st…

a5434c9

…ripe-customer-payment-methods

vercel bot deployed to Preview August 30, 2024 09:26 View deployment

girarda reviewed Aug 30, 2024

View reviewed changes

artem1205 marked this pull request as draft August 30, 2024 16:21

artem1205 added 4 commits September 2, 2024 15:21

Source Stripe: ref payment_methods

2ab5753

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Source Stripe: ref payment_methods

0f44f10

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Source Stripe: add UpdatedCursorIncrementalStripeSubStream

f5505e5

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Merge remote-tracking branch 'origin/master' into artem1205/source-st…

55429d6

…ripe-customer-payment-methods # Conflicts: # airbyte-integrations/connectors/source-stripe/metadata.yaml # airbyte-integrations/connectors/source-stripe/pyproject.toml # docs/integrations/sources/stripe.md

vercel bot deployed to Preview September 2, 2024 13:56 View deployment

artem1205 added 3 commits September 2, 2024 17:04

Source Stripe: add timeout to pytest

01bb687

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Source Stripe: add fix payment tests + fmt

fbc6057

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Merge remote-tracking branch 'origin/master' into artem1205/source-st…

99aa1dd

…ripe-customer-payment-methods

artem1205 changed the title ~~feat(source-stripe): add stream Customer Payment Methods~~ feat(source-stripe): ref stream Payment Methods Sep 2, 2024

artem1205 added the breaking-change Don't merge me unless you are ready. label Sep 2, 2024

Merge remote-tracking branch 'origin/master' into artem1205/source-st…

329e586

…ripe-customer-payment-methods

girarda reviewed Sep 4, 2024

View reviewed changes

artem1205 added 4 commits September 4, 2024 19:57

Source Stripe: update schema

b39d6d2

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Source Stripe: fix persons + unit tests

5ce0773

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Source Stripe: fmt

7c7db54

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Airbyte CDK: update migration guide + version

1733cdf

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

vercel bot deployed to Preview September 5, 2024 09:52 View deployment

artem1205 added 2 commits September 5, 2024 12:36

Airbyte CDK: update test name

54a3006

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Airbyte CDK: update docs

ceec5c2

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

vercel bot deployed to Preview September 5, 2024 10:42 View deployment

artem1205 requested a review from maxi297 September 5, 2024 14:06

maxi297 approved these changes Sep 5, 2024

View reviewed changes

Source Stripe: add expected records

90fa9c8

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

girarda requested a review from lazebnyi September 6, 2024 16:09

girarda approved these changes Sep 6, 2024

View reviewed changes

lazebnyi approved these changes Sep 6, 2024

View reviewed changes

artem1205 removed the breaking-change Don't merge me unless you are ready. label Sep 9, 2024

artem1205 added 5 commits September 10, 2024 16:25

Merge remote-tracking branch 'origin/master' into artem1205/source-st…

b80cee7

…ripe-customer-payment-methods

Source Stripe: update poetry lock

f6dcc50

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Merge remote-tracking branch 'refs/remotes/origin/master' into artem1…

7fe05cf

…205/source-stripe-customer-payment-methods

Source Stripe: update poetry lock

2768355

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

Source Stripe: fix check connection method

d8d8432

Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>

octavia-approvington approved these changes Sep 11, 2024

View reviewed changes

octavia-approvington merged commit b00dac8 into master Sep 11, 2024
32 of 36 checks passed

octavia-approvington deleted the artem1205/source-stripe-customer-payment-methods branch September 11, 2024 15:42

aaronsteers reviewed Sep 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(source-stripe): ref stream `Payment Methods` #44891

feat(source-stripe): ref stream `Payment Methods` #44891

artem1205 commented Aug 29, 2024 •

edited

Loading

vercel bot commented Aug 29, 2024 •

edited

Loading

maxi297 Aug 29, 2024

artem1205 Aug 30, 2024

maxi297 Aug 30, 2024

maxi297 Aug 30, 2024

girarda Aug 30, 2024

artem1205 Aug 30, 2024

girarda Aug 30, 2024

girarda Aug 30, 2024

girarda Sep 4, 2024

artem1205 Sep 4, 2024

girarda Sep 4, 2024

girarda Sep 4, 2024

artem1205 Sep 5, 2024

artem1205 Sep 5, 2024

maxi297 left a comment

girarda left a comment

girarda commented Sep 9, 2024

artem1205 commented Sep 10, 2024 •

edited

Loading

girarda commented Sep 11, 2024

octavia-approvington commented Sep 11, 2024

aaronsteers Sep 11, 2024 •

edited

Loading

feat(source-stripe): ref stream Payment Methods #44891

feat(source-stripe): ref stream Payment Methods #44891

Conversation

artem1205 commented Aug 29, 2024 • edited Loading

What

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

vercel bot commented Aug 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxi297 left a comment

Choose a reason for hiding this comment

girarda left a comment

Choose a reason for hiding this comment

girarda commented Sep 9, 2024

artem1205 commented Sep 10, 2024 • edited Loading

girarda commented Sep 11, 2024

octavia-approvington commented Sep 11, 2024

aaronsteers Sep 11, 2024 • edited Loading

Choose a reason for hiding this comment

feat(source-stripe): ref stream `Payment Methods` #44891

feat(source-stripe): ref stream `Payment Methods` #44891

artem1205 commented Aug 29, 2024 •

edited

Loading

vercel bot commented Aug 29, 2024 •

edited

Loading

artem1205 commented Sep 10, 2024 •

edited

Loading

aaronsteers Sep 11, 2024 •

edited

Loading