Reuse metadata from deltalake when reading parquet

[j-bennet]

In dask-deltatable, when calling dd.read_parquet, perhaps we can reuse the metadata already preserved in delta json, instead of collecting it from parquet files all over again.

Here:

dask-deltatable/dask_deltatable/core.py

Line 196 in cd731a9

df = dd.read_parquet(dt.file_uris(), **kwargs)

It looks like dd.read_parquet will have to go through the parquet files to read the metadata, but the DeltaTable should have all that info already.

[jrbourbeau]

Good catch @j-bennet. I think adding dataset={"schema": dt.schema().to_pyarrow()} as a keyword to this read_parquet call

dask-deltatable/dask_deltatable/core.py

Line 196 in cd731a9

df = dd.read_parquet(dt.file_uris(), **kwargs)

should do the trick. Though it'd be nice if someone could confirm this is the case.

[j-bennet]

I think adding dataset={"schema": dt.schema().to_pyarrow()} as a keyword to this read_parquet call
should do the trick. Though it'd be nice if someone could confirm this is the case.

I think Delta Log also contains columns stats, so maybe we can avoid gathering those.