You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Now that we use complex column types such as dictionaries and lists more and more, we need some features to support this usages.
Let's say we have a table like this:
I want to make a filter that only targets dictionaries with key1 (row with id 2) and nothing else (i.e. not rows 1 and 3)).
I have developed a piece of code that can do that in pyarrow for IAR purposes, but it probably should be generalized to Astra etc.)
frompyarrow.computeimportfieldaspyarrow_field, scalar, Expression, map_lookup, list_value_lengthimportpyarrowaspa
...
f.reduce(
lambdafilter_expression, columns: filter_expression| (
# Make sure that the forecast contains all the required columns in the granularity_columnsf.reduce(
lambdagranularity_condition, col: granularity_condition&~map_lookup(
pyarrow_field(FORECAST.forecast_granularity), pa.scalar(col), "first"
).is_null(),
columns,
scalar(True),
)
# Make sure that the forecast contains only the required columns in the granularity_columns# Map type is casted to list type to get the length of the list& (
list_value_length(
pyarrow_field(FORECAST.forecast_granularity).cast(
pa.list_(pa.struct([("key", pa.string()), ("value", pa.string())]))
)
)
==scalar(len(columns))
)
),
granularity_columns,
scalar(False),
)
The text was updated successfully, but these errors were encountered:
Now that we use complex column types such as dictionaries and lists more and more, we need some features to support this usages.
Let's say we have a table like this:
I want to make a filter that only targets dictionaries with
key1
(row with id 2) and nothing else (i.e. not rows 1 and 3)).I have developed a piece of code that can do that in pyarrow for IAR purposes, but it probably should be generalized to Astra etc.)
The text was updated successfully, but these errors were encountered: