Dynamically creating and deleting modules in Mir

[xosmig]

Problem description

In the current implementation of Mir, modules are created once, at the start of the node, and the set of modules remains static. The goal is to create an API to allow dynamically creating and deleting modules during the execution.

Use cases

1. Reconfiguration: The most straightforward way to reconfigure a system is to delete an instance of the protocol with the old configuration and instantiate a new instance with a new configuration.
2. Modular protocol design: Distributed protocols are often designed in a modular way, where one distributed protocol relies on a (not necessarily constant) number of other protocols. For example, a Distributed Key Generation protocol may rely on a number of instances of Consensus and each instance of Consensus may rely on a number of instances of a Reliable Broadcast protocol.
3. One-shot implementations: In the distributed computing literature, many protocols are described in the “one-shot” form (e.g., a typical Secret Sharing protocol allows sharing a single secret) and it is assumed that a practical implementation would use as many instances of the one-shot abstraction as necessary. It may not be the best practice for production-ready distributed systems (e.g., due to potential issues with performance and DOS attacks), but may be useful for prototyping.

Problems and potential solutions

1. How to communicate the new module to the core?

A naive solution would send the Module object itself through a Mir event to some special "system" module.

However, this event would need to be serializable (which is complicated because a module is an arbitrary object in Go that implements a certain interface).

Possible solution

The system passes a special handle with the method NewModule(name string, m Module) to the modules.
When this method is invoked, the system "registers" the module and notifies the "parent" through an event module_created(name).

One downside is that it will be impossible to replay a "child" module without replaying its "parent".

2. How to remove a module?

Note that the removal of a module needs to be ordered with respect to the incoming events of this module.
There are also some design decisions to make:

• Should we notify the module that it is being removed?
• Should we wait for a confirmation from the module that it is ready to be removed?

Possible solution

The module can only delete itself, through a method in a handle (similar to the one used for creating modules). Other modules can politely ask the module to delete itself (ideally, through a standardized event).

3. What to do if an event is intended for a non-existing module?

There can be many reasons for that:

1. it is possible that the module existed before, but was removed.
2. it is possible that the module just hasn't yet been created (e.g., in the debugging mode).
3. it is possible that the module was already created on another node, but not yet on this one.
4. it is possible that there is a typo in the module name :)
5. it is possible that a Byzantine node sends bogus messages with a bogus "destination module" field.

Note that we cannot buffer such events indefinitely if we want to operate on a finite memory.

Possible solution

Drop all events intended for non-existing modules and do not record them in the trace. In the debugging mode (i.e., when replaying a trace), buffer them and do not drop any events (those that should have been dropped did not make it to the trace).

Note that the third case (when the module was already created on another node, but not yet on this one) requires special care (see below).

4. How to route messages?

Scenario 1: If one node has already moved to a new configuration while the other is still in the old configuration, can we simply drop messages associated with the new configuration on the second node?

Scenario 2: Also, sometimes, a node needs to start participating in an instance of a distributed protocol only when it starts receiving messages associated with this instance.

Possible solution

We could specify a “fall-back” module in messages. If the target module does not exist, the message is routed to the “fall-back” module, which may create the target module and forward the message to it or simply drop the message.

However, this solution is rather complicated.

5. What if a module was created with a name that previously belonged to another (already deleted) module?

In such a case, the module may receive some unexpected events that were intended for its predecessor and either crash or behave in an unexpected way.

• Catching and prohibiting such situations is not trivial:
  • If ids are strings (which they are now), it would be impossible on finite memory (we would need to keep an ever-growing buffer of “tombstones”).
  • We could try to change the ids to integers and let the core assign them. However, in the current architecture, different nodes need to agree on the ids of corresponding modules so that they can exchange messages. This would not be trivial with system-generated ids.

Possible solution

Simply assume that it does not happen (i.e., make it the programmer's responsibility to use only unique module IDs). Note that accidentally violating this rule may lead to some subtle bugs and data races. We may want to keep the tombstones in the debugging mode in order to catch such situations at least in the tests.

One useful rule that we can enforce is to require that the "child" module name is prefixed with the name of the "parent".