fix complex filtering expressions involving variables or field names that require escaping

[hallettj]

To filter results we use MongoDB's $match aggregation stage. This stage uses a shorthand "query document" syntax that looks generally like this:

{
  $match: {
    someField: { $eq: 1 },
    anotherField: { $gt: 2 }
  }
}

Query documents have some limitations:

• They cannot reference field names that contain dots (.) since that is interpreted as a nested field path.
• They cannot reference variables.
• They cannot reference another field on the RHS of a comparison (AFAICT)

There is more general MongoDB expression language, "aggregation expressions" that does not have these shortcomings. A $match stage can opt in to comparing using an aggregation expression instead of a query document using this syntax:

{
  $match: {
    $expr: aggregationExpression
  }
}

This switch must be made at the top-level of the $match argument so this is all-or-nothing.

The previous expression generation code made the switch to an aggregation expression in cases where it was necessary. But it did not correctly handle cases where an NDC expression that must be translated to an aggregation expression is embedded in a larger expression. In other words it did not handle complex expressions corrrectly.

This change fixes the problem by splitting expression generation into two independent functions.

• make_query_document builds the shorthand query document, but aborts if that is not possible for a given expression
• make_aggregation_expression builds aggregation expressions - this is the fallback if make_query_document fails

Along the way I implemented column-to-column comparisons in cases where the left operand (the target) is a column in a relation. But I left the case where the right operand is in a relation unimplemented. We probably don't need that anyway: NDC is moving away from allowing relationship paths in ColumnTarget, and is moving to explicit "exists" operations instead.

Fixes ENG-1020, ENG-942

[codedmart]

👍