Add more filters #154
Add more filters #154
Conversation
Same rationale as the other changes :D - just for cleanliness and clarity.
Pull Request Test Coverage Report for Build 11975993358Details
💛 - Coveralls |
|
Everything here more or less works. I also want to add this to the Python tests to test the I/O. What's the best way to do that? |
- Kerchunk often encodes the compressor as the last filter, so we check that the compressor isn't hiding in the filters array if the compressor is null. - Similarly, the dtype is often unknown in this case, or the transform is not encoded correctly, so we ensure that the datatypes of `data` and `a2` remain the same by reinterpreting.
|
This PR is pretty much done from my end - I will continue development and refactoring the compressor/filter stack under a single |
This means we won't load the filters unless Zarr doesn't have them.
…to unsigned" This reverts commit 24a68e6.
|
Thanks again @asinghvi17 . Last time we talked you mentioned you wanted to add some last changes to this PR. Is this ready for review now? |
|
I still need to add some tests going from Python -> Julia and back again, should be able to get to that today though. Thanks for the ping :D |
|
Hey @asinghvi17 another ping. I would really like to start working on the zarrv3 branch again but would ideally merge this one and rebase to minimize conflicts. Also, if you could rebase this PR on current master there would not be unrelated test failures anymore. |
not just vectors, as it was previously constrained to
- Never use reinterpret - use array comprehensions to support 0-dimensional arrays correctly, the performance impact is negligible based on testing - only round if the target type is an integer, otherwise let it be if it's a float.
|
Sorry, this slipped off my radar for a bit. Should be done now, and all CI seems green as well. Feel free to merge whenever convenient for you. |
This PR aims to add the following filters, for compatibility with NetCDF4 and HDF5 files:
It also splits the filters into a new folder and documents the API for people who might want to add new filters.