Add Row-Wise Mode Functionality (axis=1) and Improve Metadata Handling in _collection.py #1137
Conversation
- Added functionality for row-wise mode calculation (axis=1) to support the Dask DataFrame API. - The new implementation dynamically handles row-wise mode and ensures consistent metadata handling across partitions. - Added validation for the axis parameter, with appropriate error handling for unsupported values. - Ensured compatibility with existing column-wise (axis=0) mode functionality, preserving the original behavior for that case. Resolves dask-expr issue dask#1136.
|
Unit tests for the added functionality have been submitted in the Dask repository’s PR due to the initial issue location and the entry point for the functionality being within dask/dataframe/core.py. Please note that this PR in Dask-expr must be approved before the Dask PR tests can successfully run, as the tests depend on the changes made here. |
Formatted _collection.py with Black
| # Implement axis=1 (row-wise mode) | ||
| num_columns = len(self.columns) # Maximum possible number of modes per row | ||
|
|
||
| def row_wise_mode(df): |
There was a problem hiding this comment.
you don't have to do this, we generally expect that the columns in every partition are matching
There was a problem hiding this comment.
Could you clarify which part isn't necessary? Are you referring to determining the number of columns for row-wise mode?
There was a problem hiding this comment.
Everything that is in def row_wise_mode basically
There was a problem hiding this comment.
Understood. I'll make the necessary revisions this afternoon and push them to this PR once tested.
|
|
||
| # Create metadata with the correct number of columns and float64 dtype | ||
| meta = pd.DataFrame( | ||
| {i: pd.Series(dtype="float64") for i in range(num_columns)} |
There was a problem hiding this comment.
why are you forcing float in all cases?
There was a problem hiding this comment.
Thank you for highlighting that forcing columns to float64 can lead to unintended type conversions. I initially used float64 to handle NaN values but agree it's better to preserve original data types. I'll update the implementation to dynamically determine data types based on the input, using pandas' nullable types like Int64 for integers and string for text data. This will handle missing values without unnecessary type changes, ensuring consistent data types across partitions while maintaining data integrity. Please let me know if you have further suggestions.
There was a problem hiding this comment.
Please don't do any explicit type casting, just use the nonempty meta of the input and call mode on it
Resolves Dask Expressions issue #1136 and Dask issue #11389