Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(datasets) Add semantic partitioner #3663

Open
wants to merge 35 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
68732b0
Initial completion
KarhouTam Jun 18, 2024
2e2137f
Add item in `partitioner/__init__.py`
KarhouTam Jun 18, 2024
cd7bcbc
Optimize test cases
KarhouTam Jun 18, 2024
f865947
Add comments, examples, more functions
KarhouTam Jun 18, 2024
96029a6
Add item on README
KarhouTam Jun 18, 2024
5f0b62e
Add paper title comment
KarhouTam Jun 18, 2024
da2f047
Correct import of example in class docs
KarhouTam Jun 18, 2024
54ac47f
Merge branch 'main' into semantic-partitioner
KarhouTam Jun 20, 2024
f178eff
Sort imports
KarhouTam Jun 20, 2024
e8cb7ab
Reformat
KarhouTam Jun 20, 2024
2edf546
Merge branch 'main' into semantic-partitioner
KarhouTam Jun 21, 2024
b4cfd2f
Refactor codes, add test cases and comments
KarhouTam Jun 21, 2024
0f88f93
Fix bugs introduced by refactoring
KarhouTam Jul 1, 2024
a1f2866
Remove unused hint
KarhouTam Jul 1, 2024
dff38c4
Fix for passing checks
KarhouTam Jul 1, 2024
6a819cb
Merge branch 'main' into semantic-partitioner
KarhouTam Jul 1, 2024
cc26c4a
Merge branch 'main' into semantic-partitioner
KarhouTam Jul 12, 2024
98f50a2
Merge branch 'main' into semantic-partitioner
KarhouTam Aug 24, 2024
87ae5fa
change partitioner name
KarhouTam Aug 24, 2024
c40a039
refactor argument
KarhouTam Aug 24, 2024
437bcdb
add item
KarhouTam Aug 24, 2024
c80a6db
add log prints and comments
KarhouTam Aug 24, 2024
28b1c6d
improve checks of dataset type
KarhouTam Aug 24, 2024
ebdb9ad
remove __main__ part
KarhouTam Aug 24, 2024
00bddf6
add arg introduction
KarhouTam Aug 24, 2024
8154948
refactor `_preprocess_dataset_images()`
KarhouTam Aug 27, 2024
97cb023
rename `sample_seed` to `shuffle_seed`
KarhouTam Aug 27, 2024
1679879
refactor some codes
KarhouTam Aug 27, 2024
5a25e5e
fix pylint check
KarhouTam Aug 27, 2024
ebc2f3a
Rename arg 'shuffle_seed` back to `rng_seed`
KarhouTam Sep 10, 2024
1192a3d
fix and refactor
KarhouTam Sep 21, 2024
b4840c5
fix check codes
KarhouTam Sep 21, 2024
d4b5e24
remove __main__
KarhouTam Sep 21, 2024
1135913
black format
KarhouTam Sep 21, 2024
7cc8acd
fix format checks
KarhouTam Sep 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions datasets/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ Create **custom partitioning schemes** or choose from the **implemented [partiti
* IID partitioning `IidPartitioner(num_partitions)`
* Dirichlet partitioning `DirichletPartitioner(num_partitions, partition_by, alpha)`
* Distribution partitioning `DistributionPartitioner(distribution_array, num_partitions, num_unique_labels_per_partition, partition_by, preassigned_num_samples_per_label, rescale)`
* Image Semantic partitioning `ImageSemanticPartitioner(num_partitions)`
* InnerDirichlet partitioning `InnerDirichletPartitioner(partition_sizes, partition_by, alpha)`
* Pathological partitioning `PathologicalPartitioner(num_partitions, partition_by, num_classes_per_partition, class_assignment_mode)`
* Natural ID partitioning `NaturalIdPartitioner(partition_by)`
Expand Down
1 change: 1 addition & 0 deletions datasets/doc/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ Here are a few of the ``Partitioner`` s that are available: (for a full list see
* IID partitioning ``IidPartitioner(num_partitions)``
* Dirichlet partitioning ``DirichletPartitioner(num_partitions, partition_by, alpha)``
* Distribution partitioning ``DistributionPartitioner(distribution_array, num_partitions, num_unique_labels_per_partition, partition_by, preassigned_num_samples_per_label, rescale)``
* Image Semantic partitioning ``ImageSemanticPartitioner(num_partitions)``
* InnerDirichlet partitioning ``InnerDirichletPartitioner(partition_sizes, partition_by, alpha)``
* PathologicalPartitioner ``PathologicalPartitioner(num_partitions, partition_by, num_classes_per_partition, class_assignment_mode)``
* Natural ID partitioner ``NaturalIdPartitioner(partition_by)``
Expand Down
2 changes: 2 additions & 0 deletions datasets/flwr_datasets/partitioner/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from .distribution_partitioner import DistributionPartitioner
from .exponential_partitioner import ExponentialPartitioner
from .iid_partitioner import IidPartitioner
from .image_semantic_partitioner import ImageSemanticPartitioner
from .inner_dirichlet_partitioner import InnerDirichletPartitioner
from .linear_partitioner import LinearPartitioner
from .natural_id_partitioner import NaturalIdPartitioner
Expand All @@ -33,6 +34,7 @@
"DistributionPartitioner",
"ExponentialPartitioner",
"IidPartitioner",
"ImageSemanticPartitioner",
"InnerDirichletPartitioner",
"LinearPartitioner",
"NaturalIdPartitioner",
Expand Down
Loading
Loading