Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(datasets): Added the Experimental SafetensorsDataset #898

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open

feat(datasets): Added the Experimental SafetensorsDataset #898

wants to merge 22 commits into from

Conversation

MinuraPunchihewa
Copy link
Contributor

@MinuraPunchihewa MinuraPunchihewa commented Oct 18, 2024 •
edited
Loading

Description

This PR adds the SafetensorsDataset to support interactions with tensors stored in files in the Safetensors format.

Fixes #221

Development notes

I have used the PickleDataset as a base for the implementation of this dataset (as it goes about the same manner to access files).

These changes have been tested,

  1. Manually, by running the code locally to load and save tensors from and to Safetensors files.
  2. Via the existing and newly added unit tests.

Checklist

  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the relevant RELEASE.md file
  • Added tests to cover my changes
  • Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

@MinuraPunchihewa
added the skeleton for the Safetensors experimental dataset
26f714d 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
implemented the save() and load() funcs
c2a980e 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
updated the default backend
58a258d 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
implemented the describe() and exists() funcs
3723b02 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
imported the dataset to main pkg
843e815 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
fixed how data is passed to load()
9a54930 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
fixed save() to access the file path
8a5d522 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
added a release() func
1bd4117 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
added the docstrings for the dataset
eba18df 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
fixed lint issues
98bb719 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
MinuraPunchihewa and others added 4 commits October 20, 2024 12:08
@MinuraPunchihewa
added unit tests
5d0e347 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
added a few more unit tests
58c1ac7 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
fixed broken unit test
3350161 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
Merge branch 'main' into feature/safetensors_dataset
da6c32a
@MinuraPunchihewa MinuraPunchihewa marked this pull request as ready for review October 20, 2024 06:39
@MinuraPunchihewa
Copy link
Contributor Author

Hey @astrojuanlu,
This is my implementation of the SafetensorsDataset. I would appreciate a review.

@MinuraPunchihewa
fixed lint issues
aa6dd9c 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
fixed use of insecure temp files
697f34f 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
added the dataset to the documentation
4c27e73 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
listed the dependencies for the dataset
7922ed8 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
fixed typo in dataset reference
3c59813 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
Merge branch 'main' into feature/safetensors_dataset
8602809 
Signed-off-by: Minura Punchihewa <49385643+MinuraPunchihewa@users.noreply.github.com>
@DimedS DimedS added the Community Issue/PR opened by the open-source community label Nov 8, 2024
DimedS
DimedS approved these changes Nov 12, 2024
View reviewed changes
Copy link
Contributor

@DimedS DimedS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR, @MinuraPunchihewa! It looks great to me. I tested it manually, and everything works perfectly!

DimedS and others added 2 commits November 12, 2024 12:07
@DimedS
Merge branch 'main' into feature/safetensors_dataset
f634a99
@MinuraPunchihewa
updated the release notes
471b5a6 
Signed-off-by: Minura Punchihewa <minurapunchihewa17@gmail.com>
@MinuraPunchihewa
Copy link
Contributor Author

Thank you for the PR, @MinuraPunchihewa! It looks great to me. I tested it manually, and everything works perfectly!

Thank you, @DimedS. I appreciate it. I just updated the release notes as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DimedS DimedS DimedS approved these changes

@astrojuanlu astrojuanlu Awaiting requested review from astrojuanlu

@ElenaKhaustova ElenaKhaustova Awaiting requested review from ElenaKhaustova

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
Community Issue/PR opened by the open-source community
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add safetensors dataset
2 participants
@MinuraPunchihewa @DimedS