Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(source-stripe): ref stream Payment Methods #44891

Merged
merged 41 commits into from
Sep 11, 2024
Merged

feat(source-stripe): ref stream Payment Methods #44891

Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
db6a787
Source Stripe: add new stream Customer Payment Methods
artem1205 Aug 29, 2024
9e576a9
Source Stripe: migrate to incremental substream
artem1205 Aug 29, 2024
b9b9468
Source Stripe: add to catalog
artem1205 Aug 29, 2024
1c8d80d
Source Stripe: bump version + update docs
artem1205 Aug 29, 2024
790be8a
Source Stripe: update openapi spec
artem1205 Aug 29, 2024
87188b8
Source Stripe: fix unit tests + bump pytest
artem1205 Aug 29, 2024
3147444
Source Stripe: fmt
artem1205 Aug 30, 2024
a5434c9
Merge remote-tracking branch 'origin/master' into artem1205/source-st…
artem1205 Aug 30, 2024
2ab5753
Source Stripe: ref payment_methods
artem1205 Sep 2, 2024
0f44f10
Source Stripe: ref payment_methods
artem1205 Sep 2, 2024
f5505e5
Source Stripe: add UpdatedCursorIncrementalStripeSubStream
artem1205 Sep 2, 2024
55429d6
Merge remote-tracking branch 'origin/master' into artem1205/source-st…
artem1205 Sep 2, 2024
01bb687
Source Stripe: add timeout to pytest
artem1205 Sep 2, 2024
fbc6057
Source Stripe: add fix payment tests + fmt
artem1205 Sep 2, 2024
99aa1dd
Merge remote-tracking branch 'origin/master' into artem1205/source-st…
artem1205 Sep 2, 2024
329e586
Merge remote-tracking branch 'origin/master' into artem1205/source-st…
artem1205 Sep 3, 2024
12de7cd
Merge remote-tracking branch 'origin/master' into artem1205/source-st…
artem1205 Sep 3, 2024
d0b2a05
Merge remote-tracking branch 'origin/master' into artem1205/source-st…
artem1205 Sep 3, 2024
6e95888
Source Stripe: fix docs
artem1205 Sep 3, 2024
97fbe3f
Source Stripe: bump CDK version
artem1205 Sep 3, 2024
feed8b2
Source Stripe: fix fmt
artem1205 Sep 3, 2024
a7d7910
Source Stripe: fix fmt
artem1205 Sep 3, 2024
72ff6ad
Source Stripe: fix CAT
artem1205 Sep 3, 2024
88f4b38
Source Stripe: fix schema
artem1205 Sep 4, 2024
7895a1f
Source Stripe: bump version + docs update
artem1205 Sep 4, 2024
6ea2498
Source Stripe: bump version + docs update
artem1205 Sep 4, 2024
2aa5074
Source Stripe: fmt
artem1205 Sep 4, 2024
e2f14b6
Source Stripe: ref persons
artem1205 Sep 4, 2024
b39d6d2
Source Stripe: update schema
artem1205 Sep 4, 2024
5ce0773
Source Stripe: fix persons + unit tests
artem1205 Sep 4, 2024
7c7db54
Source Stripe: fmt
artem1205 Sep 4, 2024
1733cdf
Airbyte CDK: update migration guide + version
artem1205 Sep 5, 2024
54a3006
Airbyte CDK: update test name
artem1205 Sep 5, 2024
ceec5c2
Airbyte CDK: update docs
artem1205 Sep 5, 2024
90fa9c8
Source Stripe: add expected records
artem1205 Sep 5, 2024
b80cee7
Merge remote-tracking branch 'origin/master' into artem1205/source-st…
artem1205 Sep 10, 2024
0c78855
Merge remote-tracking branch 'origin/master' into artem1205/source-st…
artem1205 Sep 10, 2024
f6dcc50
Source Stripe: update poetry lock
artem1205 Sep 10, 2024
7fe05cf
Merge remote-tracking branch 'refs/remotes/origin/master' into artem1…
artem1205 Sep 10, 2024
2768355
Source Stripe: update poetry lock
artem1205 Sep 10, 2024
d8d8432
Source Stripe: fix check connection method
artem1205 Sep 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions airbyte-integrations/connectors/source-stripe/acceptance-test-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,6 @@ acceptance_tests:
bypass_reason: "This stream can't be seeded in our sandbox account"
- name: "external_account_cards"
bypass_reason: "This stream can't be seeded in our sandbox account"
- name: "payment_methods"
bypass_reason: "This stream can't be seeded in our sandbox account"
- name: "persons"
bypass_reason: "This stream can't be seeded in our sandbox account"
- name: "reviews"
Expand Down
1 change: 1 addition & 0 deletions airbyte-integrations/connectors/source-stripe/integration_tests/expected_records.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
{"stream": "customers", "data": {"id": "cus_LIiHR6omh14Xdg", "object": "customer", "address": {"city": "san francisco", "country": "US", "line1": "san francisco", "line2": "", "postal_code": "", "state": "CA"}, "balance": 0, "created": 1646998902, "currency": "usd", "default_source": "card_1MSHU1EcXtiJtvvhytSN6V54", "delinquent": false, "description": "test", "discount": null, "email": "test@airbyte_integration_test.com", "invoice_prefix": "09A6A98F", "invoice_settings": {"custom_fields": null, "default_payment_method": null, "footer": null, "rendering_options": null}, "livemode": false, "metadata": {}, "name": "Test", "next_invoice_sequence": 1, "phone": null, "preferred_locales": [], "shipping": {"address": {"city": "", "country": "US", "line1": "", "line2": "", "postal_code": "", "state": ""}, "name": "", "phone": ""}, "tax_exempt": "none", "test_clock": null, "updated": 1646998902}, "emitted_at": 1697627278433}
{"stream": "customers", "data": {"id": "cus_Kou8knsO3qQOwU", "object": "customer", "address": null, "balance": 0, "created": 1640123795, "currency": "usd", "default_source": "src_1MSID8EcXtiJtvvhxIT9lXRy", "delinquent": false, "description": null, "discount": null, "email": "edward.gao+stripe-test-customer-1@airbyte.io", "invoice_prefix": "CA35DF83", "invoice_settings": {"custom_fields": null, "default_payment_method": null, "footer": null, "rendering_options": null}, "livemode": false, "metadata": {}, "name": "edgao-test-customer-1", "next_invoice_sequence": 2, "phone": null, "preferred_locales": [], "shipping": null, "tax_exempt": "none", "test_clock": null, "updated": 1640123795}, "emitted_at": 1697627278435}
{"stream": "customers", "data": {"id": "cus_NGoTFiJFVbSsvZ", "object": "customer", "address": {"city": "", "country": "US", "line1": "Street 2, 34567", "line2": "", "postal_code": "94114", "state": "CA"}, "balance": 0, "created": 1675160053, "currency": "usd", "default_source": "src_1MWGs8EcXtiJtvvh4nYdQvEr", "delinquent": false, "description": "Test Customer 2 description", "discount": null, "email": "user1.sample@zohomail.eu", "invoice_prefix": "C09C1837", "invoice_settings": {"custom_fields": null, "default_payment_method": null, "footer": null, "rendering_options": null}, "livemode": false, "metadata": {}, "name": "Test Customer 2", "next_invoice_sequence": 15, "phone": null, "preferred_locales": ["en-US"], "shipping": {"address": {"city": "", "country": "US", "line1": "Street 2, 34567", "line2": "", "postal_code": "94114", "state": "CA"}, "name": "Test Customer 2", "phone": ""}, "tax_exempt": "none", "test_clock": null, "updated": 1675160053}, "emitted_at": 1697627278439}
{"stream":"payment_methods","data":{"id":"pm_1PYSZ3EcXtiJtvvhIos7UZSV","object":"payment_method","allow_redisplay":"limited","billing_details":{"address":{"city":null,"country":"CA","line1":null,"line2":null,"postal_code":"K1P 5L9","state":null},"email":"customer99900@example.com","name":"Customer 999","phone":null},"card":{"brand":"visa","checks":{"address_line1_check":null,"address_postal_code_check":"pass","cvc_check":"pass"},"country":"US","display_brand":"visa","exp_month":1,"exp_year":2026,"fingerprint":"X7e9fFB0r8MMcdo6","funding":"credit","generated_from":null,"last4":"4242","networks":{"available":["visa"],"preferred":null},"three_d_secure_usage":{"supported":true},"wallet":null},"created":1720010738,"customer":"cus_QPH7FKQ1vXX4N0","livemode":false,"metadata":{},"type":"card"},"emitted_at":1724941242506}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we set one payment method for at least two customer to ensure that stream partitions are working in end-to-end?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test steps:

  1. tried to update existing payment_method via Update method => got an error: custoemr cannot be updated with this endpoint, try to detach/attach
  2. made a post detach requests => successful
  3. tried to attach payment_method from step 2 to another customer => got an error:
{
    "error": {
        "message": "This PaymentMethod was previously used without being attached to a Customer or was detached from a Customer, and may not be used again.",
        "request_log_url": "https://dashboard.stripe.com/test/logs/req_u3sulw0pAzAzX8?t=1725528614",
        "type": "invalid_request_error"
    }
}

Conclusion: Payment cannot be set for 2 customers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we can have two customers with two different payment method right? I was basically wondering if we could have:

{
    "stream": "payment_methods",
    "data": {
        "id":<an id>,
        "customer": "cus_X",
        <...>
    }
    "emitted_at": 1724941242506
}
{
    "stream": "payment_methods",
    "data": {
        "id":<another id>,
        "customer": "cus_Y",
        <...>
    }
    "emitted_at": 1724941242506
}

... so that CATs would validate that partitioning is working.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, ok, will add another one

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added in 90fa9c8

{"stream": "cardholders", "data": {"id": "ich_1KUKBeEcXtiJtvvhCEFgko6h", "object": "issuing.cardholder", "billing": {"address": {"city": "San Francisco", "country": "US", "line1": "1234 Main Street", "line2": null, "postal_code": "94111", "state": "CA"}}, "company": null, "created": 1645143542, "email": "jenny.rosen@example.com", "individual": null, "livemode": false, "metadata": {}, "name": "Jenny Rosen", "phone_number": "+18888675309", "preferred_locales": [], "requirements": {"disabled_reason": null, "past_due": []}, "spending_controls": {"allowed_categories": [], "blocked_categories": [], "spending_limits": [], "spending_limits_currency": null}, "status": "active", "type": "individual", "updated": 1645143542}, "emitted_at": 1697627292209}
{"stream": "charges", "data": {"id": "ch_3K9FSOEcXtiJtvvh0zxb7clc", "object": "charge", "amount": 5300, "amount_captured": 5300, "amount_refunded": 0, "amount_updates": [], "application": null, "application_fee": null, "application_fee_amount": null, "balance_transaction": "txn_3K9FSOEcXtiJtvvh0KoS5mx7", "billing_details": {"address": {"city": null, "country": null, "line1": null, "line2": null, "postal_code": null, "state": null}, "email": null, "name": null, "phone": null}, "calculated_statement_descriptor": "AIRBYTE.IO", "captured": true, "created": 1640120473, "currency": "usd", "customer": null, "description": null, "destination": null, "dispute": null, "disputed": false, "failure_balance_transaction": null, "failure_code": null, "failure_message": null, "fraud_details": {}, "invoice": null, "livemode": false, "metadata": {}, "on_behalf_of": null, "order": null, "outcome": {"network_status": "approved_by_network", "reason": null, "risk_level": "normal", "risk_score": 48, "seller_message": "Payment complete.", "type": "authorized"}, "paid": true, "payment_intent": "pi_3K9FSOEcXtiJtvvh0AEIFllC", "payment_method": "src_1K9FSOEcXtiJtvvhHGu1qtOx", "payment_method_details": {"card": {"amount_authorized": 5300, "brand": "visa", "checks": {"address_line1_check": null, "address_postal_code_check": null, "cvc_check": "pass"}, "country": "US", "exp_month": 12, "exp_year": 2034, "extended_authorization": {"status": "disabled"}, "fingerprint": "X7e9fFB0r8MMcdo6", "funding": "credit", "incremental_authorization": {"status": "unavailable"}, "installments": null, "last4": "4242", "mandate": null, "multicapture": {"status": "unavailable"}, "network": "visa", "network_token": {"used": false}, "overcapture": {"maximum_amount_capturable": 5300, "status": "unavailable"}, "three_d_secure": null, "wallet": null}, "type": "card"}, "receipt_email": null, "receipt_number": "1509-9197", "receipt_url": "https://pay.stripe.com/receipts/payment/CAcaFwoVYWNjdF8xSndub2lFY1h0aUp0dnZoKJ35vqkGMgYYlboX7Hs6LBbBoR6yFToo5WeMCCwbkvCz7nl3E1KToovFFZKMJYnrpAHBlWJrVMJK6BWm", "refunded": false, "refunds": {"object": "list", "data": [], "has_more": false, "total_count": 0.0, "url": "/v1/charges/ch_3K9FSOEcXtiJtvvh0zxb7clc/refunds"}, "review": null, "shipping": null, "source": {"id": "src_1K9FSOEcXtiJtvvhHGu1qtOx", "object": "source", "amount": null, "card": {"address_line1_check": null, "address_zip_check": null, "brand": "Visa", "country": "US", "cvc_check": "pass", "dynamic_last4": null, "exp_month": 12, "exp_year": 2034, "fingerprint": "X7e9fFB0r8MMcdo6", "funding": "credit", "last4": "4242", "name": null, "three_d_secure": "optional", "tokenization_method": null}, "client_secret": "src_client_secret_3WszbFGtWT8vmMjqnNztOwhU", "created": 1640120473, "currency": null, "flow": "none", "livemode": false, "metadata": {}, "owner": {"address": null, "email": null, "name": null, "phone": null, "verified_address": null, "verified_email": null, "verified_name": null, "verified_phone": null}, "statement_descriptor": null, "status": "consumed", "type": "card", "usage": "reusable"}, "source_transfer": null, "statement_descriptor": "airbyte.io", "statement_descriptor_suffix": null, "status": "succeeded", "transfer_data": null, "transfer_group": null, "updated": 1640120473}, "emitted_at": 1697627293840}
{"stream": "charges", "data": {"id": "ch_3K9F5DEcXtiJtvvh1w2MaTpj", "object": "charge", "amount": 4200, "amount_captured": 4200, "amount_refunded": 0, "amount_updates": [], "application": null, "application_fee": null, "application_fee_amount": null, "balance_transaction": "txn_3K9F5DEcXtiJtvvh1qsqmHcH", "billing_details": {"address": {"city": null, "country": null, "line1": null, "line2": null, "postal_code": null, "state": null}, "email": null, "name": null, "phone": null}, "calculated_statement_descriptor": "AIRBYTE.IO", "captured": true, "created": 1640119035, "currency": "usd", "customer": null, "description": "edgao test", "destination": null, "dispute": null, "disputed": false, "failure_balance_transaction": null, "failure_code": null, "failure_message": null, "fraud_details": {}, "invoice": null, "livemode": false, "metadata": {}, "on_behalf_of": null, "order": null, "outcome": {"network_status": "approved_by_network", "reason": null, "risk_level": "normal", "risk_score": 63, "seller_message": "Payment complete.", "type": "authorized"}, "paid": true, "payment_intent": "pi_3K9F5DEcXtiJtvvh16scJMp6", "payment_method": "src_1K9F5CEcXtiJtvvhrsZdur8Y", "payment_method_details": {"card": {"amount_authorized": 4200, "brand": "visa", "checks": {"address_line1_check": null, "address_postal_code_check": null, "cvc_check": "pass"}, "country": "US", "exp_month": 9, "exp_year": 2028, "extended_authorization": {"status": "disabled"}, "fingerprint": "X7e9fFB0r8MMcdo6", "funding": "credit", "incremental_authorization": {"status": "unavailable"}, "installments": null, "last4": "4242", "mandate": null, "multicapture": {"status": "unavailable"}, "network": "visa", "network_token": {"used": false}, "overcapture": {"maximum_amount_capturable": 4200, "status": "unavailable"}, "three_d_secure": null, "wallet": null}, "type": "card"}, "receipt_email": null, "receipt_number": "1549-5630", "receipt_url": "https://pay.stripe.com/receipts/payment/CAcaFwoVYWNjdF8xSndub2lFY1h0aUp0dnZoKJ35vqkGMgbg2Y1Ao1M6LBYViHyCHYtYZtCIzc8I1Pm_oXAcXtgPDTNCfzyB3XOfFO4N-RK2w9sLuPjq", "refunded": false, "refunds": {"object": "list", "data": [], "has_more": false, "total_count": 0.0, "url": "/v1/charges/ch_3K9F5DEcXtiJtvvh1w2MaTpj/refunds"}, "review": null, "shipping": null, "source": {"id": "src_1K9F5CEcXtiJtvvhrsZdur8Y", "object": "source", "amount": null, "card": {"address_line1_check": null, "address_zip_check": null, "brand": "Visa", "country": "US", "cvc_check": "pass", "dynamic_last4": null, "exp_month": 9, "exp_year": 2028, "fingerprint": "X7e9fFB0r8MMcdo6", "funding": "credit", "last4": "4242", "name": null, "three_d_secure": "optional", "tokenization_method": null}, "client_secret": "src_client_secret_QyH8xuqSyiZh8oxzzIszqQ92", "created": 1640119035, "currency": null, "flow": "none", "livemode": false, "metadata": {}, "owner": {"address": null, "email": null, "name": null, "phone": null, "verified_address": null, "verified_email": null, "verified_name": null, "verified_phone": null}, "statement_descriptor": null, "status": "consumed", "type": "card", "usage": "reusable"}, "source_transfer": null, "statement_descriptor": "airbyte.io", "statement_descriptor_suffix": null, "status": "succeeded", "transfer_data": null, "transfer_group": null, "updated": 1640119035}, "emitted_at": 1697627293843}
Expand Down
8 changes: 7 additions & 1 deletion airbyte-integrations/connectors/source-stripe/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: e094cb9a-26de-4645-8761-65c0c425d1de
dockerImageTag: 5.5.3
dockerImageTag: 6.0.0
dockerRepository: airbyte/source-stripe
documentationUrl: https://docs.airbyte.com/integrations/sources/stripe
erdUrl: https://dbdocs.io/airbyteio/source-stripe?view=relationships
Expand Down Expand Up @@ -43,6 +43,12 @@ data:
scopedImpact:
- scopeType: stream
impactedScopes: ["refunds"]
6.0.0:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this need to be a major version bump? this feels like something we should notify users in the docs and migration guide, but I don't think it's worth stopping their syncs

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

customer was set as object, and now it is changed to string.
This is the main reason to mark changes as breaking.
endpoint change can also affect customer incremental sync.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be part of the migration guide then?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🌶️ I'd still advocate for not flagging as breaking change since only one Cloud user actually has data for this stream

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, reverted.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be part of the migration guide then?

Since i removed this version from breakingChanges in metadata, CI fails on QA checks in migration guide.
Is there any other place I can place the release notes?

message: Version 6.0.0 introduces fixes for the `Payment Methods` streams. The `Payment Methods` will now provide data about customers' payment methods.
upgradeDeadline: "2024-09-25"
scopedImpact:
- scopeType: stream
impactedScopes: ["payment_methods"]
suggestedStreams:
streams:
- customers
Expand Down
Loading
Loading