Query parameter properties: align with OGC API Coverages

[cportele]

The draft of OGC API Coverages has a "field selection" requirements class. We need to ensure both are compatible with each other.

[m-mohr]

By the way, STAC API also has something similar which OGC APIs should probably align to: https://github.com/stac-api-extensions/fields

[cportele]

Thanks @m-mohr. We should take a closer look.

[jerstlouis]

@cportele We are considering an x-ogc-returnedByDefault JSON Schema annotation which would default to true.

In the case where a property is not returned by default, should the use of exclude-properties automatically make all available properties returned unless explicitly listed as excluded?

Since we discussed making exclude-properties and properties mutually exclusive, we could not say something like properties=*&exclude-properties=B08. See discussion in opengeospatial/ogcapi-coverages#188 (comment) .

Hoping to stabilize this ASAP because OGC API - DGGS specifies the use of properties and exclude-properties and I hope everything can be well aligned:

https://docs.ogc.org/DRAFTS/21-038.html#_parameter_properties

Thanks!

[cportele]

@jerstlouis

On a property with writeOnly: true the keyword x-ogc-returnedByDefault would not be applicable. Should it be ignored, if provided?

In any case, we first need to discuss, if we should follow the existing STAC extension - and if yes, how x-ogc-returnedByDefault would work with a fields parameter.

If we would stick to properties and exclude-properties, I would say that if any of the parameters is provided, the defaults no longer apply and what is specified in either properties or excluded-properties would identify the properties provided or excluded in the response. So, there would be no need for properties=*&exclude-properties=B08, just state exclude-properties=B08 to exclude that property, everything other property would be included. Any other approach would seem counter-intuitive to me.

[jerstlouis]

On a property with writeOnly: true the keyword x-ogc-returnedByDefault would not be applicable. Should it be ignored, if provided?

Yes, ignored, but it should just not be included in the first place.

So, there would be no need for properties=*&exclude-properties=B08, just state exclude-properties=B08 to exclude that property, everything other property would be included. Any other approach would seem counter-intuitive to me.

That is also how I feel, but @fmigneault and @m-mohr seemed to have a different intuitive understanding in opengeospatial/ogcapi-coverages#188 (comment):

I would expect exclude-properties=B02 using the same previous defaults to return only B04,B03. If everything else gets added as a result, that just seems wrong.

In any case, we first need to discuss, if we should follow the existing STAC extension

This would be very different from what we've been doing with our Features and Coverages implementations for several years, and what the DGGS API which is completing RFC and already voted on by TC to go for approval vote, so it would be quite late to change it to something completely different there.

@m-mohr I'm curious why this fields extension was not brought up earlier this year if there was a desire to align that e.g., when we were discussing the Field Selection in the T20 GDC discussions?

The - semantic in the STAC fields extension also conflicts with our current use of CQL2 arithmetic expression (though we discussed potentially using this alias mechanism instead for those).

I would not be in favor of changing everything to the STAC fields extension.

[m-mohr]

@m-mohr I'm curious why this fields extension was not brought up earlier this year if there was a desire to align that e.g., when we were discussing the Field Selection in the T20 GDC discussions?

• We did bring it to the SWGs attention even back in 2020: Future work items (Part 6++) #451 (comment)
• With regards to the Coverages "field selection", that for me is something different and solves a different usecase so connecting the different bits and pieces here was not very obvious. It's JSON property selection in an API response vs. e.g. selecting bands in a GeoTiff reponse...
• If you refer to this issue, I just can't follow all discussions in all SWGs, sorry. I posted it as soon as I found it.

But then ask yourself: Why doesn't "OGC" survey the existing landscape before inventing something new? We didn't hide it and it's there since years...

[cportele]

We did bring it to the SWGs attention even back in 2020: #451 (comment)

Just like you can not follow all discussions, we cannot be aware of all comments that were made in issues several years ago. I think we need a mechanism to discuss such topics in a better way, if we want to align OGC API building blocks and STAC extensions.

[m-mohr]

Just like you can not follow all discussions, we cannot be aware of all comments that were made in issues several years ago.

Fair, although my point was meant to be: Check other specs before writing something new (e.g. https://stac-api-extensions.github.io/), not read or remember all issues out there.

I think we need a mechanism to discuss such topics in a better way, if we want to align OGC API building blocks and STAC extensions.

In STAC and openEO I always try to survey the landscape (including related OGC APIs) for what's out there and ideally pick the "best" or "closest" option. That doesn't mean I find all possible options, of course. I don't have an idea for a better way yet, but open for suggestions. Going through each other issue trackers every month is not really an option.

How does OGC handle cross API SWG communication? Although it seems it doesn't fully work either though based of the discrepancies that sometimes occur between the specs. Is that forum OGC API - Common? Not sure if it helps if we open an issue for each existing STAC API extension in Common...

[cportele]

Check other specs before writing something new (e.g. https://stac-api-extensions.github.io/)

Good point. I will to check https://stac-api-extensions.github.io first in the future.

The "fields" extension is at "Candidate" level. How widely has it been implemented? Is there information where each STAC extension has been implemented?

[m-mohr]

That's the funny (or sad?) thing. We kept it Candidate to wait for alignment with OGC APIs (assuming you'd look at our extension and potentially propose a couple of minor improvements or so). It would've already been Stable from our side. It's implemented in various servers and a couple of clients.

Sort is stable because we already aligned. Filter is Candidate because we waited for last changes in CQL2 etc (and then had no time yet to do final alignment checks with the released versions).

While "asking" you to monitor STAC API extensions, I'm wondering is there an easy overview over OGC API functionalities (across all SWGs)? Like a one-page list of standards, parts and conformance classes with a 1-3 sentence summary? That would help with such landscape surveys quite a bit...

[cportele]

I guess that is something for @ghobona to answer (I can talk about Features/Styles/Routes).

Maybe the proper approach would be regular calls (every two or months) to inform about developments in STAC and OGC API to avoid such issues?

[m-mohr]

Maybe, although I'm also not 100% sure whether that works well for STAC as we don't really have a central "SWG" which all developments go through. I recently found a pretty mature STAC extension from the community that the PSC was not aware of afaik ;-)

And then it's not just STAC, we also have openEO and other non-OGC initiatives where alignment could happen...

Edit: Just realized we are completely off topic here, maybe we should open a new issue for this but not sure where...

Edit 2: No, there's no list of implementations unfortunately.

Edit 3: I feel like it's the task of a standardization body to follow community developments and adapt where needed. Or at least facilitate that better. By staff members of course, not the volunteers like you and me. But maybe that's just my opinion.

[cportele]

Edit: Just realized we are completely off topic here, maybe we should open a new issue for this but not sure where...

Since this is a general coordination issue, any ideas how and where to organize this, @ghobona?

Edit: In OGC we basically use the OGC API track during the TC Meeting to inform and discuss cross-resource issues. Maybe we also need a regular web-meeting to discuss such issues with STAC, OpenEO and others?

[jerstlouis]

@m-mohr

We did bring it to the SWGs attention even back in 2020

Ideally, if the stars meant for alignment to happen on the proposed way forward for "Field / Property Selection", it should have happened in that issue 4 years ago :)

With regards to the Coverages "field selection", that for me is something different and solves a different usecase so connecting the different bits and pieces here was not very obvious.

This I believe is because of The Great Divide between Vector vs. Raster / Features vs. Coverages which I hope we can mend :) The properties of features are exactly the same concept as the fields in a coverage, and we're trying to align this as much as possible (e.g., with the schemas, property selection, filter, derived fields, etc.), which facilitates the integration of vector and raster data.

Why doesn't "OGC" survey the existing landscape before inventing something new?

Since the great majority of the SWG work is done on a volunteer basis, and as you point out it's already hard enough to keep track of the standards being developed inside OGC and keeping them aligned, and that some people have a strong preference for building things based on their own ideas (I'm a 100% Not Invented Here person), it's really asking a lot to survey everything "out there".

Ideally paid employees/contractors could perhaps be tasked with doing this constant surveying for strategic communities, and even within OGC to pro-actively ensure alignment. I have tried hard to do this as much as possible within the OGC SWGs but the time involved with all the different SWGs is just too much.

But back to the properties discussion... My primary concern is for OGC API - DGGS which is really in the final stages.
Since it's still in RFC, I guess we could still technically make this change.

Regarding the STAC fields extension, I don't expect we would require support for the POST operation in the basic requirement class of most spec. It's a little bit difficult to understand how the missing/null/empty include/exclude semantic works in the GET syntax.

If we use those + / - operators, we would definitely need to move the CQL2 arithmetic etc. expressions to another parameter.

[jerstlouis]

@m-mohr

Like a one-page list of standards, parts and conformance classes with a 1-3 sentence summary?

I previously suggested exactly these things (Standards, Parts, and Conformance Classes) as a higher level kind of "building blocks" (compared to e.g., the Web API building blocks like paths/parameters/responses/schemas), with the idea that some kind of register should track these to be able to present them in a 1 pager (and you should ideally also be able to see the Web API building blocks that they use -- hence why I thought it'd make sense to have it all in the building block register).

EDIT: including the "maturity" status as well as the various stages so that we can pro-actively "align" things.

[cportele]

My general sentiment after a closer look at the STAC fields extension: The general mechanics are very similar to Part 6, but the fields extension is tightly coupled to the GeoJSON encoding (considering "id", "type", "geometry", "properties" as fields) while in Features the query parameter needs to be applicable for all feature encodings and as such the properties query parameter accepts properties from the logical schema (Part 5).

Where one would write properties=myid,mygeometry,foo according to the current draft of Part 6, using the fields extension it would be fields=id,geometry,properties.foo.

While the mechanics of including and excluding fields/properties in the response is quite similar, the semantics of what is a field / property are quite different. AFAICT, the fields concept is not applicable to APIs implementing Features, if the API supports any other encoding beside GeoJSON.

As a result, I think that we should continue to work on the "properties" query parameter in Part 6. We can discuss whether we drop "exclude-properties" and use a "-" prefix to exclude properties to align the mechanics.

@m-mohr, @pvretano, @jerstlouis - any thoughts on this?

[pvretano]

@cportele just an FYI, I kinda like having just the properties field and using +/- mechanics. However, keep in mind that if we decide to take this route, we may be blocking off a route that would allow expressions to be specified in the properties field like you can do in a SQL select statement.

properties=-temp+offset, cos(myOtherProperty),...

Of course to make this useful we would need to include an aliasing mechanism but I hope the idea is clear.

[cportele]

@pvretano - Good point. I also like the compact +/- notation that is also used in "sortby", but it would likely block #927.

[fmigneault]

One thing to consider (and which was raised in STAC a few times), is that the fields=id,geometry,properties.foo approach allows to potentially filter on other things than properties, which might be relevant, especially for the case of assets in STAC that are not under the properties field.

Therefore, while the properties=myid,mygeometry,foo is more compact, it is very limiting because it does not allow outer fields to be accessed. This will inevitably lead to many extra query parameters to filter on them, which could make the definition of CQL2 filters more complicated.

[pvretano]

@fmigneault property selection and filtering is based on using the list of property names obtained from the server's API document or the logical schema (if available) which allows access to the returnables and queryables which are different views of the logical schemas.

How these property names map to physical properties if entirely up to the server. So, a property named foo might internally map to properties.foo. Similarly a property named bob might map to assert.thumbnail.title. It's all up to the server. Of course the server may choose to simply name the logical properties properties.foo and assets.thumbnail.title.

One interesting aspect of this "logical schema" approach is the fact that the server can also create "synthetic properties". A relevance score is an example of a property that might be synthetic and would computed at runtime. Once defined by the server, thought, you could then use relevance in property selection or in a CQL2 filter (e.g. filter=relevance>0.95).

So, to summarize, I am not sure we are limited in the way you believe we are limited. Thoughts?

[fmigneault]

Ah yes! You're right. I completely forgot about that aspect.
It is so easy to confuse the properties query with the properties in GeoJSON responses.

I guess on that point, fields from STAC has the advantage to avoid the ambiguity, but I understand we have to live with many servers/specs already using properties.

No limitation then!

[jerstlouis]

To add to @pvretano 's comment, properties specifically references the returnables, whereas filter specifically references the queryables.