Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: snowflake destination #314

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Conversation

mpolomdeepsense
Copy link
Contributor

@mpolomdeepsense mpolomdeepsense commented Dec 19, 2024

Fixes Snowflake Uploader and SQL Uploader Stager

SQL Uploader Stager fix for error [upload_stage] Object of type Timestamp is not JSON serializable:

2024-12-19 17:22:24,127 MainProcess ERROR    1 failed documents:
2024-12-19 17:22:24,127 MainProcess ERROR    /home/marek/.cache/unstructured/ingest/pipeline/index/eada064bd94a.json: [upload_stage] Object of type Timestamp is not JSON serializable
2024-12-19 17:22:24,127 MainProcess ERROR    failed to run destination command snowflake: Pipeline did not run successfully
Traceback (most recent call last):
  File "/home/marek/unstructured/unstructured-ingest/unstructured_ingest/v2/cli/base/dest.py", line 49, in cmd
    pipeline.run()
  File "/home/marek/unstructured/unstructured-ingest/unstructured_ingest/v2/pipeline/pipeline.py", line 144, in run
    raise PipelineError("Pipeline did not run successfully")
unstructured_ingest.v2.pipeline.pipeline.PipelineError: Pipeline did not run successfully
Error: Pipeline did not run successfully

Snowflake Uploader fix for error [upload] 255001: Binding data in type (list) is not supported.:

2024-12-19 17:27:37,363 MainProcess ERROR    1 failed documents:
2024-12-19 17:27:37,364 MainProcess ERROR    /home/marek/.cache/unstructured/ingest/pipeline/index/eada064bd94a.json: [upload] 255001: Binding data in type (list) is not supported.
2024-12-19 17:27:37,364 MainProcess ERROR    failed to run destination command snowflake: Pipeline did not run successfully
Traceback (most recent call last):
  File "/home/marek/unstructured/unstructured-ingest/unstructured_ingest/v2/cli/base/dest.py", line 49, in cmd
    pipeline.run()
  File "/home/marek/unstructured/unstructured-ingest/unstructured_ingest/v2/pipeline/pipeline.py", line 144, in run
    raise PipelineError("Pipeline did not run successfully")
unstructured_ingest.v2.pipeline.pipeline.PipelineError: Pipeline did not run successfully
Error: Pipeline did not run successfully

@rbiseck3
Copy link
Collaborator

Can we update the integration test to cause the original issue and make sure the fix is tested?

@mpolomdeepsense
Copy link
Contributor Author

@rbiseck3

Added sql upload stager conform_dataframe unit tests.

There are some snowflake destination tests that I think should handle this case, but they are skipped in CI.

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants