Plugin System Support

[majetideepak]

Some of the components in Velox such as connectors, and functions are extendable.
However, the current approach involves manually registering any new extension.
Supporting a plugin system will enable drop-in capabilities. The idea is to have similar capabilities as the Presto Plugin. https://prestodb.io/docs/current/develop/spi-overview.html#plugin.

CC: @paul-amonson, @tdcmeehan

[pedroerp]

Is the idea that we would read and register these extensions based on a configuration file, or that we would support extensions to be dynamically linked at execution time? Or both?

[pedroerp]

There's nothing preventing one from doing that today. I had a PR our sometime ago where I added support for doing that for scalar functions.

[majetideepak]

@pedroerp Can you share that PR when you get a chance? Thanks.

[pedroerp]

@majetideepak #1005

[aditi-pandit]

Thanks Pedro.

@majetideepak : Its good that we have this mechanism at the Velox level, and that using dlopen is fine with the Velox team.

Though I feel the plugin feature is more relevant at the Prestissimo (or other data engine) level. Looking at Prestissimo e2e for such a functionality :

• Presto plugins load functions in catalogs as documented https://prestodb.io/docs/current/develop/functions.html#plugin-implementation.
• We will need to extend the Plugin configuration to include metadata about the function libraries to load in Prestissimo worker. During Prestissimo worker startup, such catalogs libraries will need to be loaded with dlopen and hook the register function along with other registrations here https://github.com/prestodb/presto/blob/master/presto-native-execution/presto_cpp/main/PrestoServer.cpp#L754
• We do need some Java stub function code also to load the catalog and functions in the co-ordinator for function resolution in queries.
• What is still missing is how we do evaluation for constant folding optimization.

This is a fine first solution. But in general we should have fenced Remote UDF function support also. Pedro has done bunch of work on Remote functions already, but it would be great to wire that through Presto plugin framework as well.

[pedroerp]

Agreed, though I suspect we will eventually need support for both (fenced and unfenced), depending on the engine and use case. The PR above was a just a proof of concept for an internal ML integration, but we did not move forward with that approach at the time.