1️⃣ Planner Redesign

[joocer]

Currently, the planner goes straight from AST to the physical plan.

AST -> Binder -> Physical Plan -> Optimizer

AST -> Logical Plan -> Binder -> Optimizer -> Physical Plan

This work should be done at the same time as the work to rewrite the operators for a pull-based execution.

Discovered Tasks

• Use the AST Rewriter to rewrite how CTEs are included in the plan - but need to solve how to get the CTEs out of the plan. The other option is to pass them to the logical planner for exchange, but that's what we do now that causes 🪲 CTEs cannot contain references to other CTEs #779

[joocer]

Step One

• write the framework for multi processing (a controller to issue work and receive responses)

• need to record the number of active items per query I'd, so we can tell if it's complete

• write a serial version too, it doesn't need to be performant, just needs to be an option for debugging

[joocer]

Logical planner should create plans like

PROJECT (name)
└─ LEFT JOIN (a=b)
   ├─ SUBQUERY
   │  └─ GROUP (planetId)
    |      ├─FILTER: (gm > 10)
   │     └─ SCAN ($satellites)
   └─  SCAN ($planets)

[joocer]

The logical planner is about 90% done - it plans most SELECT queries, although does have some edge cases it does not (relating to CAST and ARRAY_AGG functions), it also plans some other non-SELECT queries.

Attention is now on the binder.

[joocer]

BINDER TODO:

• bind '@@' and '@' variables
• to bind the columns we need to pass them upwards, morsels have these columns with these types with these aliases, this is the form of the data at this point. This includes when functions and aggregates are run, this column has this type
• we then need to prune columns, once we're at the head of the execution tree, work our way back down pruning columns (no point reading a column if it's not used)
• we need to do function type validation
• we need to do evaluation (e.g. a + b) type validation
• write schemas for GENERATE_SERIES, UNNEST and FAKE table constructors
• move fake, unnest and generate_series into their own modules, with additional functions to get the schema. This could be base classed to allow easier adding of new functions that create relations.
• copy the build() function which takes parts of the ast and builds trees into V2 and rewrite the parts that are changed for V2 (e.g. function detection, CAST and ARRAY_AGG)
• write out more functions into the V2 functions
• add support for show functions, so the can be used to test adding functions
• build a function harness for getting cost values for functions
• functions should have. str method to control how they are rendered
• change plus, subtract, multiply etc into function calls
• functions should have a time() method to get cost information and a test() method to unit test

[joocer]

Improvements to the binder should mean the old Columns model can be removed. Rather than work this information out at execution time, the planner/ binder can work this out. A fair amount of code has been stripped out which deals with working out the schema dynamically... Some of this will end up back in the binder but a lot will be avoided by having schema information.

Where schemas aren't stored centrally, they will be worked out from the data; single files and databases can be queried but multiple files will need to a) error, b) use the schema of the first file or c) have a schema in the folder structure.

[joocer]

Rethink approach to aggregates, currently split group and aggregate into two steps but

• group won't output a table, so breaks pattern

If done as

• aggregate, and
• group and aggregate

Would allow for specialism and would keep the in/out consistency.

Put code common to both of them in the aggregate module, group and aggregate could inherit do allow for reuse. (Similar inheritance model should be considered for the joins)

!!
Do I need tables anymore as I'm not using the metadata, can probably use chunks ( depends if actions like adding and removing columns works on chunks)

[joocer]

Most basic queries are working again with the sample datasets.

Moving to features like JOINs and subqueries next, and then onto external datasets.

Will start with the code that will run if there's no schemas in the catalogue;

• for SQL datasets ask the DB the schema
• for file-based datasets, get the schema of the last file.
• for collection datasets, get one record

File-based needs some consideration as to if fetching and opening a file will affect performance, especially if this file is then discarded and read again during the query execution.

[joocer]

Storage

Handlers need

• cache able
• oartionable

RegisterStore

• name
• prefix
• remove prefix
• connector
• cache
• partition
• connection string
• bucket/collection