How many DB could subscribe from one Kernel and Stream?

[Little-Wallace]

As I know, there may be many DB instance will execute on one kernel. But the interface new_update_writer of Kernel does not supply stream_name. It means that, if there are multiple Stream processing updates in the cluster, the db subscribe from the Kernel can not tell the order of updates between multiple Stream.

This is a consensus problem. I think it is enough for every engine process to subscribe only one Stream. Of course, it can consume part of the updates in this Stream. If you need to store the data from multiple Stream , you can keep multiple engine in one process. There is my suggestion for Kernal:

#[async_trait]
pub trait Kernel {
    type Memtable: Memtable
    type StreamReader: StreamReader;
    type StreamWriter: StreamWriter;
    type RandomReader: RandomRead;
    type SequentialWriter: SequentialWrite;

    /// Returns a Memtable to read and update data in memory
    async fn new_memtable(&self, memtable_name: &str) -> Result<Self::Memtable>;

    async fn new_stream_reader(&self, stream_name: &str) -> Result<Self::StreamReader>;

    async fn new_stream_writer(&self, stream_name: &str) -> Result<Self::StreamWriter>;

    async fn new_random_reader(
        &self,
        bucket_name: &str,
        object_name: &str,
    ) -> Result<Self::RandomReader>;

    async fn new_sequential_writer(
        &self,
        bucket_name: &str,
        object_name: &str,
    ) -> Result<Self::SequentialWriter>;
}
#[async_trait]
pub trait Memtable {
    async fn apply(&mut self, update: KernelUpdate) -> Result<Sequence>;

    async fn release(&mut self, sequence: Sequence) -> Result<()>;
}

[huachaohuang]

UpdateWriter and UpdateReader in Kernel is for metadata updates, not user data. Kernel only needs to manage persistent state for engines, so things like memory tables are not Kernel's concern. You can check #241 for a possible use case of multiple streams.

[Little-Wallace]

Thanks