You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Would the ParquetDataset be more appropriate here? It can accept filters as Expression, or tuple/DNF form, which would allow us to skip that filters_to_expression step.
Currently in dask-deltatable, we're using
pyarrow.dataset.dataset
, which we filter with apyarrow.Expression
:dask-deltatable/dask_deltatable/core.py
Line 78 in dbeb8cc
Would the
ParquetDataset
be more appropriate here? It can accept filters asExpression
, ortuple
/DNF form, which would allow us to skip thatfilters_to_expression
step.https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetDataset.html
The text was updated successfully, but these errors were encountered: