-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
destination bigquery 1s1t: print subquery stats #28373
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is cool!
...io/airbyte/integrations/destination/bigquery/typing_deduping/BigQueryDestinationHandler.java
Show resolved
Hide resolved
destination-bigquery test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml | ✅ |
Connector version semver check | ✅ |
Connector version increment check | ❌ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-bigquery docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest | ❌ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
destination-bigquery-denormalized test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-bigquery-denormalized docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery-denormalized test
destination-bigquery-denormalized test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-bigquery-denormalized docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery-denormalized test
destination-bigquery test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-bigquery docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
/approve-and-merge reason="current failure is a transient dockerhub thing; tests were prevoiusly passing" |
* print subquery stats * indicate truncation * use destinationhandler in test for convenience
turns out it is possible to to find the cost of each query within a transaction, it's just a little involved.
the bq.listJobs is a real api call, so it takes around 1 second ish. Arguably that shouldn't happen on the main thread, but... (1) meh, it's 1 additional second every few minutes, and (2) async should make this less bad since other threads are still doing things.
logs look something like this
also use this in the sqlgenerator integration test for convenience.