feat(datasets): create separate `ibis.FileDataset` #842

deepyaman · 2024-09-20T15:28:30Z

Description

Resolves #828

Development notes

So far, I copied ibis.TableDataset, removing code paths for reading database tables, and adding support for file export.

Just wanted to put this out there for early feedback, but what else should be done?

Deprecate file I/O in TableDataset
Support versioning (I don't think there's any reason why this wouldn't work out of the box for FileDataset)
Docs stuff (add FileDataset to toctree, etc.)

Update: Versioning is actually not a trivial subject, because backends don't implement a consistent interface for checking whether a file exists. I plan to do this with PyArrow filesystem, but I will do that in a follow-up PR (to limit complexity added here); this PR handles local versioning.

Checklist

Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the relevant RELEASE.md file
Added tests to cover my changes

datajoely

This is great!

kedro-datasets/kedro_datasets/ibis/file_dataset.py

deepyaman · 2024-10-04T15:55:57Z

One of the downsides of having a separate ibis.FileDataset is that the connection cache isn't shared, so you won't reuse an existing connection across ibis.FileDataset and ibis.TableDataset. Maybe this could be resolved by having the _connections class variable tied to a mixin these both inherit from, or, perhaps more simply, taking a Backend object and using a custom resolver to pass that.

I can explore this in a future PR; this is probably good enough for now.

kedro-datasets/kedro_datasets/ibis/file_dataset.py

datajoely

This is 99% there, I'd prefer some extra bits of clarity so that @deepyaman 's bus factor isn't as high when it comes to maintainability

Very excited to get this in 💪

kedro-datasets/kedro_datasets/ibis/file_dataset.py

Changes made

datajoely

🔥

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

Refs: b7ff0c7 Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

noklam

Thank you, it's a nice addition to the ibis datasets

* feat(datasets): create separate `ibis.FileDataset` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): deprecate `TableDataset` file I/O Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * feat(datasets): implement `FileDataset` versioning Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): try `os.path.exists`, for Windows Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * revert(datasets): use pathlib, ignore Windows test Refs: b7ff0c7 Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * docs(datasets): add `ibis.FileDataset` to contents Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): add docstring for `hashable` func Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): add docstring for `hashable` func Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * feat(datasets)!: expose `load` and `save` publicly Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): remove second filepath assignment Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: tdhooghe <thomas_dhooghe@mckinsey.com>

deepyaman force-pushed the feat/datasets/ibis-filedataset branch 7 times, most recently from 88f74cb to c8f076e Compare September 24, 2024 04:13

deepyaman marked this pull request as ready for review September 24, 2024 05:04

datajoely reviewed Sep 24, 2024

View reviewed changes

kedro-datasets/kedro_datasets/ibis/file_dataset.py Show resolved Hide resolved

deepyaman mentioned this pull request Sep 30, 2024

Release kedro-datasets #855

Closed

deepyaman force-pushed the feat/datasets/ibis-filedataset branch from f5436cb to af42be8 Compare October 4, 2024 15:50

deepyaman force-pushed the feat/datasets/ibis-filedataset branch 2 times, most recently from ad21e5a to fe56964 Compare October 7, 2024 00:12

deepyaman enabled auto-merge (squash) October 7, 2024 00:31

deepyaman mentioned this pull request Oct 7, 2024

feat(datasets): Release kedro-datasets 5.0.0 #869

Merged

4 tasks

deepyaman requested review from noklam, astrojuanlu, merelcht and DimedS October 7, 2024 17:28

noklam reviewed Oct 8, 2024

View reviewed changes

kedro-datasets/kedro_datasets/ibis/file_dataset.py Show resolved Hide resolved

kedro-datasets/kedro_datasets/ibis/file_dataset.py Outdated Show resolved Hide resolved

datajoely reviewed Oct 8, 2024

View reviewed changes

kedro-datasets/kedro_datasets/ibis/file_dataset.py Show resolved Hide resolved

datajoely previously requested changes Oct 8, 2024

View reviewed changes

deepyaman force-pushed the feat/datasets/ibis-filedataset branch 2 times, most recently from ff0b85a to bce8c57 Compare October 10, 2024 12:28

deepyaman commented Oct 10, 2024

View reviewed changes

kedro-datasets/kedro_datasets/ibis/file_dataset.py Outdated Show resolved Hide resolved

deepyaman force-pushed the feat/datasets/ibis-filedataset branch from 2aa5bcc to 27a4f4b Compare October 10, 2024 12:33

deepyaman requested a review from datajoely October 10, 2024 12:33

datajoely approved these changes Oct 10, 2024

View reviewed changes

deepyaman added 15 commits October 10, 2024 18:42

feat(datasets): create separate ibis.FileDataset

f0f124f

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

chore(datasets): deprecate TableDataset file I/O

4c60375

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

feat(datasets): implement FileDataset versioning

cdd235b

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

chore(datasets): try os.path.exists, for Windows

35cb5b4

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

revert(datasets): use pathlib, ignore Windows test

98e07d7

Refs: b7ff0c7 Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

docs(datasets): add ibis.FileDataset to contents

056901e

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

chore(datasets): add docstring for hashable func

2bbc2dd

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

chore(datasets): add docstring for hashable func

f31620a

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

feat(datasets)!: expose load and save publicly

8f56930

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

chore(datasets): remove second filepath assignment

180f0b7

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

docs(datasets): add FileDataset to release notes

0dcb0af

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

style(datasets): run prettier on the release notes

71d85a5

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

chore(datasets): default to writing a Parquet file

28d5721

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

docs(datasets): add docstring for the connection

23584b4

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

docs(datasets): add example of Polars read_delta

b43163e

Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu>

deepyaman force-pushed the feat/datasets/ibis-filedataset branch from 27a4f4b to b43163e Compare October 11, 2024 00:54

noklam approved these changes Oct 11, 2024

View reviewed changes

deepyaman merged commit 3d36e57 into main Oct 11, 2024
12 checks passed

deepyaman deleted the feat/datasets/ibis-filedataset branch October 11, 2024 09:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(datasets): create separate `ibis.FileDataset` #842

feat(datasets): create separate `ibis.FileDataset` #842

deepyaman commented Sep 20, 2024 •

edited

Loading

datajoely left a comment

deepyaman commented Oct 4, 2024

datajoely left a comment •

edited

Loading

datajoely left a comment

noklam left a comment

feat(datasets): create separate ibis.FileDataset #842

feat(datasets): create separate ibis.FileDataset #842

Conversation

deepyaman commented Sep 20, 2024 • edited Loading

Description

Development notes

Checklist

datajoely left a comment

Choose a reason for hiding this comment

deepyaman commented Oct 4, 2024

datajoely left a comment • edited Loading

Choose a reason for hiding this comment

datajoely left a comment

Choose a reason for hiding this comment

noklam left a comment

Choose a reason for hiding this comment

feat(datasets): create separate `ibis.FileDataset` #842

feat(datasets): create separate `ibis.FileDataset` #842

deepyaman commented Sep 20, 2024 •

edited

Loading

datajoely left a comment •

edited

Loading