query engine integration #2074
Open
query engine integration #2074
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
joshua-spacetime
force-pushed
the
joshua/query-engine-integration
branch
from
3acec8b to
3e4ae13
Compare
joshua-spacetime
force-pushed
the
joshua/query-engine-integration
branch
from
3e4ae13 to
eaa2861
Compare
Centril
reviewed
Dec 20, 2024
Comment on lines
+36
to
+39
| pub type Index<K> = multimap::MultiMap<K, RowPointer>; | ||
| pub type IndexIter<'a, K> = multimap::MultiMapRangeIter<'a, K, RowPointer>; | ||
| pub type UniqueIndex<K> = uniquemap::UniqueMap<K, RowPointer>; | ||
| pub type UniqueIndexIter<'a, K> = uniquemap::UniqueMapRangeIter<'a, K, RowPointer>; |
There was a problem hiding this comment.
Yeah not needed. At least for right now.
joshua-spacetime
force-pushed
the
joshua/query-engine-integration
branch
from
8a4e30b to
bace373
Compare
mamcx
reviewed
Dec 23, 2024
| impl<T: StateView> SchemaView for SchemaViewer<'_, T> { | ||
| fn table_id(&self, name: &str) -> Option<TableId> { | ||
| let AuthCtx { owner, caller } = self.auth; | ||
| self.tx |
There was a problem hiding this comment.
Suggested change
| self.tx | |
| // Get the schema from the in-memory state instead of fetching from the database for speed |
mamcx
reviewed
Dec 23, 2024
| @@ -836,16 +837,24 @@ impl ModuleHost { | |||
| } | |||
|
|
|||
| #[tracing::instrument(skip_all)] | |||
| pub fn one_off_query(&self, caller_identity: Identity, query: String) -> Result<Vec<MemTable>, anyhow::Error> { | |||
| pub fn one_off_query<F: WebsocketFormat>( | |||
| &self, | |||
There was a problem hiding this comment.
This is now dead code? Why still here?
mamcx
reviewed
Dec 24, 2024
| @@ -618,7 +743,7 @@ pub struct IxJoin { | |||
| /// The expression for computing probe values. | |||
| /// Values are projected from the lhs, | |||
| /// and used to probe the index on the rhs. | |||
| pub lhs_probe_expr: ProjectField, | |||
| pub lhs_field: TupleField, | |||
There was a problem hiding this comment.
Can the rhs_label turn into TupleField instead?
mamcx
reviewed
Dec 24, 2024
| @@ -575,25 +711,14 @@ pub enum Sarg { | |||
| Range(ColId, Bound<AlgebraicValue>, Bound<AlgebraicValue>), | |||
There was a problem hiding this comment.
In
pub enum Sarg {
Range(ColId, Bound<AlgebraicValue>, Bound<AlgebraicValue>),
Suggested change
| Range(ColId, Bound<AlgebraicValue>, Bound<AlgebraicValue>), | |
| Range(BinOp,ColId, Bound<AlgebraicValue>, Bound<AlgebraicValue>), |
(for the plan printer)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of Changes
This patch integrates the new query engine into the query code path. It does not integrate the new engine into incremental evaluation path.
querycrate which is the top level crate for invoking the query engine.Multi-column index scans have not been added to the optimizer yet, but they will be once we remove the old engine. We have a benchmark that will fail if the old engine is removed without adding multi-column index scans.
The old engine will be removed once incremental evaluation has been updated to use the new engine.
API and ABI breaking changes
n/a
Expected complexity level and risk
4
The query engine is responsible for implementing the subscription logic, which is the only way to read data out of SpacetimeDB. As such, its correctness and performance is paramount.
Testing
We have already added tests to ensure we preserve semantics, with more being worked on currently. This patch updates the applicable benchmarks to ensure we maintain performance characteristics.