External modules and multi-language Mir

[matejpavlovic]

Given that the interface of a module is extremely simple and only exchanges serializable data, modules could easily be written in other languages and called through RPCs. The Go process would run proxy modules that would send all events to a “module server” for processing, receive the resulting events and submit them back to the Go process. This can be very useful not only for enabling many other ways and languages for writing modules, but even maybe for “remote” modules that need to keep some data (e.g. keys) privage. E.g., in a deployment with particular security constraints, a signing module could be run on a separate server holding the signing key only locally.

[matejpavlovic]

Posting reaction by @xosmig from another document, from which the discussion is now ported here.

This sounds very interesting as Mir could actually become a multi-language framework. Features like DSL could also be ported to other languages relatively easily. Additional bonus is that all the suffering with protobufs could be not a waste of time after all as protobufs are relatively good for multi-language deployments (but they suck for single-language deployments, as we are currently using them). One thing I would be a bit concerned about in multi-language deployments is performance. If some low-level module (e.g., net or crypto) lives in one process and the protocol using them lives in a different process, could IPC become the bottleneck?

[matejpavlovic]

Yes performance is a very legitimate concern and IPC is surely not as efficient as having everything in a single process. But I can imagine scenarios where performance is either not the primary concern (e.g. when prototyping protocols) or the "external" modules happen to not be the performance bottlenecks (e.g. when they only need to be invoked from time to time).
But yes, if someone chooses to use Mir in such a "distributed" way, they might need to count on the performance potentially being impacted.