-
Notifications
You must be signed in to change notification settings - Fork 858
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(datasets) Add DivideResplitter #2963
Conversation
start_index = 0 | ||
end_index = 0 | ||
split_data = dataset[split_from] | ||
assert isinstance(new_splits_dict, dict) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This assert will never be triggered, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, at this point it's adjusted correctly (based on the python logic) but the type checkers complained
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, let's keep if they complain. w/ Py3.10 it seemed to be fine
start_index = 0 | ||
end_index = 0 | ||
split_data = dataset[split_from] | ||
assert isinstance(new_splits_dict, dict) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, at this point it's adjusted correctly (based on the python logic) but the type checkers complained
Co-authored-by: Javier <jafermarq@users.noreply.github.com>
start_index = 0 | ||
end_index = 0 | ||
split_data = dataset[split_from] | ||
assert isinstance(new_splits_dict, dict) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, let's keep if they complain. w/ Py3.10 it seemed to be fine
Co-authored-by: Javier <jafermarq@users.noreply.github.com>
Issue
Certain splits of the dataset need to be divided into smaller splits. No functionality exists to accomplish that that can be used in the FederatedDataset abstraction.
Proposal
Create a
DivideResplitter
abstraction that solves the problem. See docstrings for more explanation.