feat(processing_engine): Runtime and Write-back improvements #25672
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes two major changes, running plugins in their own tasks and enabling writing back line protocol to the WriteBuffer.
Running Plugins as Separate Tasks
Each plugin now runs in its own task, started either when created through the API or when the system is started. It is connected to the rest of the system through two mechanisms:
QueryableBuffer
and whose receiver it polls from.Right now the only message being sent is
Arc<WalContents>
, but I expect the PluginEvent enum to be expanded both for other triggering events and control messages, e.g. Shutdown, Status, etc.Write-Back Line Protocol
The running pythons have a reference to the WriteBuffer, and this PR also adds a simple receiver for python to write line protocols to with the
insert_line_protocol
call. Everything is buffered per PluginEvent, and data is copied several times, so there is a lot of room for improvement.Example REST calls
Creating a Plugin:
Creating a Trigger: