fix(datasets): Use put() and get() instead of copy in TensorFlowModelDataset's _save and _load methods.
#844
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…owModelDataset`'s `_save` and `_load` methods. Signed-off-by: gitgud5000 <17186026+gitgud5000@users.noreply.github.com>
merelcht
approved these changes
Sep 25, 2024
There was a problem hiding this comment.
Thanks for the investigation and contribution of the fix @gitgud5000 ! This looks good to me 👍
Can you add a comment in the release notes and add your name to the contributors list for the upcoming release as well? 🙂
ankatiyar
approved these changes
Sep 25, 2024
Signed-off-by: gitgud5000 <17186026+gitgud5000@users.noreply.github.com>
Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com>
harm-matthias-harms
pushed a commit
to harm-matthias-harms/kedro-plugins
that referenced
this pull request
Oct 1, 2024
…owModelDataset`'s `_save` and `_load` methods. (kedro-org#844) * fix(datasets): Use `put()` and `get()` instead of `copy` in `TensorFlowModelDataset`'s `_save` and `_load` methods. Signed-off-by: gitgud5000 <17186026+gitgud5000@users.noreply.github.com> * chore: Added comments of changes in `RELEASE.md` Signed-off-by: gitgud5000 <17186026+gitgud5000@users.noreply.github.com> --------- Signed-off-by: gitgud5000 <17186026+gitgud5000@users.noreply.github.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Harm Matthias Harms <matthias.harms@quis.de>
MinuraPunchihewa
pushed a commit
to MinuraPunchihewa/kedro-plugins
that referenced
this pull request
Oct 1, 2024
…owModelDataset`'s `_save` and `_load` methods. (kedro-org#844) * fix(datasets): Use `put()` and `get()` instead of `copy` in `TensorFlowModelDataset`'s `_save` and `_load` methods. Signed-off-by: gitgud5000 <17186026+gitgud5000@users.noreply.github.com> * chore: Added comments of changes in `RELEASE.md` Signed-off-by: gitgud5000 <17186026+gitgud5000@users.noreply.github.com> --------- Signed-off-by: gitgud5000 <17186026+gitgud5000@users.noreply.github.com> Signed-off-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR resolves the issue in #839, where saving
TensorFlowModelDatasetto Azure Blob Storage fails due to incorrect use of.copy()from thefsspec.filesysteminterface.Development notes
The issue occurs in the
_saveand_loadmethods on lines 147 and 172, where.copy()is used. According to the fsspec documentation,.copy()is intended for remote-to-remote transfers. Since this involves copying from a local filesystem to remote storage,.put()should be used for saving and.get()for loading.The code has been updated to use
.put()and.get()accordingly, replacing the use of.copy().Both methods work for local-to-local and local-to-remote(& vice versa) transfers based on testing.
Checklist
RELEASE.mdfile