Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Debug #907

Closed
wants to merge 87 commits into from
Closed

Debug #907

wants to merge 87 commits into from

Conversation

merelcht
Copy link
Member

Description

Development notes

Checklist

  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the relevant RELEASE.md file
  • Added tests to cover my changes
  • Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

tdhooghe and others added 30 commits October 21, 2024 17:22
@tdhooghe
check and convert pd df to snowflake df
2305c3d 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
add tests
07ef67b 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
revert import to previous state
04928ba 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
relax snowpark version
77551d6 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
revert pyproject.toml
fbb3374 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
bump snowpark version
fb621ae 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
relax python version for snowpark
cb320bd 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
downgrade snowpark to 1.22
628def6 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
upgrade cloudpickle to 2.2.1
136fb09 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@deepyaman @tdhooghe
feat(datasets): create separate ibis.FileDataset (#842)
fe3bfbe 
* feat(datasets): create separate `ibis.FileDataset`

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): deprecate `TableDataset` file I/O

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* feat(datasets): implement `FileDataset` versioning

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): try `os.path.exists`, for Windows

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* revert(datasets): use pathlib, ignore Windows test

Refs: b7ff0c7

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* docs(datasets): add `ibis.FileDataset` to contents

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* feat(datasets)!: expose `load` and `save` publicly

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): remove second filepath assignment

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@ankatiyar @tdhooghe
ci(docker): Update error code in e2e test (#888)
fce2974 
Update error code in e2e test

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
update docstring
60b4a86 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
test ci/cd without python version 3.12
3e578c6 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
fix linting error
a161191 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
add max python version for compatibility
e4509bc 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
fix error
c98aef8 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@merelcht @deepyaman @tdhooghe
chore: Update PR template with checkbox for core dataset contribution (…
e4f1c3c 
…#891)

* Update PR template with checkbox for core dataset contribution

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

* Update .github/PULL_REQUEST_TEMPLATE.md

Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

* Fix lint

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>

---------

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@ankatiyar @tdhooghe
ci(datasets): Fix mypy errors (#893)
bcec735 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@deepyaman @tdhooghe
fix(datasets): default to DuckDB in in-memory mode (#897)
c243c6e 
* fix(datasets): default to DuckDB in in-memory mode

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* test(datasets): use `object()` sentinel as default

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* docs(datasets): add default database to RELEASE.md

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@deepyaman @tdhooghe
build(datasets): make Kedro-Datasets 5.1.0 release (#887)
388f455 
Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@merelcht @tdhooghe
ci: Add GH action to check for TSC votes on core dataset changes (#896)
3c85b1f 
* Add GH action to check for TSC votes on core dataset changes
* Ignore TSC vote action in gatekeeper
* Trigger TSC vote action only on changes in core dataset

---------

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
update testing framework and docstrings
101696c 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
Update kedro-datasets/kedro_datasets/snowflake/snowpark_dataset.py
c1eeb77 
Signed-off-by: Thomas <thomas.dhooghe95@gmail.com>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
validate and check table name for nones
b9bf73e 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
remove breakpoint
2c804c9 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
change doc type hinting for tests to pass
7b7811d 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@deepyaman @tdhooghe
feat(datasets): create separate ibis.FileDataset (#842)
23b056a 
* feat(datasets): create separate `ibis.FileDataset`

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): deprecate `TableDataset` file I/O

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* feat(datasets): implement `FileDataset` versioning

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): try `os.path.exists`, for Windows

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* revert(datasets): use pathlib, ignore Windows test

Refs: b7ff0c7

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* docs(datasets): add `ibis.FileDataset` to contents

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* feat(datasets)!: expose `load` and `save` publicly

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

* chore(datasets): remove second filepath assignment

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

---------

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@ankatiyar @tdhooghe
ci(datasets): Fix mypy errors (#893)
27e58cd 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
more strict type hinting and casting for linting
178f0f4 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
add back python 3.12 to see if it is compatible
0133025 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
tdhooghe and others added 28 commits October 22, 2024 16:14
@tdhooghe
improve docstring and remove ignore from dataset-doctest
0886454 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
Merge branch 'main' into feature/save-pd-to-snowflaketable
25c28c4 
Signed-off-by: Thomas <thomas.dhooghe95@gmail.com>
@tdhooghe
add back ignore snowflake for doctesting and remove unexpected additi…
624baa7 
…ons to pyproject.toml

Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
only skip for python 3.12 or higher
80f7765 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
remove separate pytest skips as we have a module wide one
ba0a030 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
remove test from omit in pytproject.toml
36264c2 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
100 test coverage
fe43b5c 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
bump matplotlib to prevent installing test requirements from failing
10829ed 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
revert matplotlib version bump
435159b 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
exclude lines from coverage
ba6975c 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
only test python 3.12
52ce490 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
omit snowflake testing from python 3.12
71a3801 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
add python version check
b89a552 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
fix bug in ignoreopts and add back python versions 3.10 and 3.11 in m…
d967ff5 
…atrix

Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
fix bug in ignoreopts and add back python versions 3.10 and 3.11 in m…
233b4f0 
…atrix

Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
revert requires-python to 3.10
7349fac 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
ignore entire folder
e64551e 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
add back omit for snowflake as python 3.12 is not supported, hence ci…
ceacc8c 
…cd fails on coverage

Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
revert makefile to original state
e0735ad 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
remove commented out setters
d17b8a6 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
remove commented out fixtures
c5c9e4d 
Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>
@tdhooghe
Merge branch 'main' into feature/save-pd-to-snowflaketable
fbc2545
@merelcht
Skip instead of xfail
f4383c7 
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
@merelcht
clean up
6f85afd 
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
@merelcht
Try skip import
97cb8af 
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
@merelcht
Try only import skip
8e8fa8a 
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
@merelcht
Fix lint
f9263d1 
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
@merelcht
lint
c8e74ef 
Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com>
@merelcht merelcht closed this Oct 24, 2024
@merelcht merelcht deleted the debug branch October 24, 2024 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@merelcht @tdhooghe @deepyaman @ankatiyar