Streaming Firestore data to BigQuery in a JSON column format

[jonathanrboniface]

I have noticed that the Firestore BigQuery Export extension streams data to BigQuery and stores the JSON data in a table with a column format of STRING. I have not found any guidance in the extension's documentation on how to change the column format to JSON. Is this functionality supported?

The reason I am asking is that I want to improve the performance of my queries. Currently, I need to use JSON functions to extract the data from the string, but I would prefer to be able to query the data directly. This is in accordance with the guidance on Working with JSON data: https://cloud.google.com/bigquery/docs/json-data.

[dackers86]

Originally, this feature was only available in Alpha.

We have a PR in progress #919 which may still need some additional work.

If there is enough interest we could potentially escalate this as a feature.

[jonathanrboniface]

I see this as having great value and simplifying the effort to actually work with your data after steaming it BigQuery.

Thanks for the quick response.

[jonathanrboniface]

For your information, I am working with a customer who is experiencing a significant problem with this issue. They are working on alternative solutions because the current solution is not performant enough to work with JSON data.

[leah-potato]

Agree - I'm just going through the process now and it's not easy 😩