[ENH] - Investigate moving to a plugin based architecture

[soapy1]

Feature description

There are currently lots of tools in the python packaging/environment space including:

• conda
• mamba
• pixi

It would be nice to be able to swap out backends for doing actions like solving, locking and package management.

Value and/or benefit

Using a plugin based system conda store developers will be able to quickly experiment with new technologies to integrate into conda-store.

Anything else?

Start with a prototype of conda-store using plugins for locking. So, conda-store should have all the same functionality with a configurable locking mechanism.

[jaimergp]

Thanks! I'm not sure if we need a full plugin system (with discoverable entry points à la pluggy etc) or just an ad-hoc configurable backend where we swap different implementations of the same interface. Would be good to clarify that, but I'm partial to not using pluggy for now and instead have a base class that defines the interface, and then subclasses that implement the desired integration. Something like:

conda_store.backends.base:BaseBackend
conda_store.backends.conda:CondaBackend
conda_store.backends.conda_lock:CondaLockBackend
conda_store.backends.micromamba:MicromambaBackend
conda_store.backends.pixi:PixiBackend

Maybe we need several interfaces for each stage (solve, lockfile, link to disk), or maybe they are different methods of the same interface. To be seen.

[soapy1]

I started working on a prototype here main...soapy1:conda-store:plugin-demo if you want to take a look.

This set's up 2 plugins for conda lock (the implementation of the plugins is the same, except for their names and logging - so that I can test swapping between plugins).

Some notable things in the prototype (at this point):

• introduce a plugin manager - this holds all the registered plugins and provides a way to access them
• add a config parameter to choose a locking backend
• adds a locking plugin interface
• do locking from a plugin

Some things on my radar to look into:

• look into pluggy
• sorting out plugin discovery/configuration/init-int
• working out actions/tasks should interact with plugins
• try to pluginify some other pieces of the stack. I think storage is a good one, and maybe one other piece that interacts with actions

[soapy1]

I setup another little demo using pluggy as the plugin manager main...soapy1:conda-store:plugin-demo-pluggy.

I think this approach is also ok. It's nice that pluggy takes care of some things like enforcing a stricter adherence to a hookspec. However, pluggy will execute all registered plugins. Which means we need to sort out the issue of working out something like a "task context" for every task that gets run. The task context would be responsible for doing stuff like registering all the appropriate plugins (and then deregistering them if appropriate).

Some pros and cons of pluggy vs a home rolled plugin manager:

• pluggy
  • pros - fully thought out and featured plugin manger
  • cons - seems like it'll be a bigger lift to implement. Will need to rethink how we are running tasks and how to compose plugins
  • cons - not clear if you can call a plugin from within a plugin - seems like not possible
  • cons - conda-store doesn't really understand the concept of hooks, and would need to adopt a "hook"-centric way of running tasks
• home rolled plugin manager
  • pros - we have complete control over how it works
  • cons - will take more time to build out more advanced features (like, pluggy's strict adherence to hookspecs)

[soapy1]

Providing a little update here. I've pushed some more changes to main...soapy1:conda-store:plugin-demo-pluggy

Some notable things that this adds:

• locking and storage are both plugins
• pluggy is used for plugin registration + hooks
• we also have a plugin registry which collects all the plugins that are available for pluggy
• which plugin to use can be configured by name (eg. so before you had to import the S3Storage class to use the s3 storage backend. Now, you can select it with the plugin name "s3-storage"
• Introduces a "build context", this has some helpful bits for executing a build action, including access to a db session, namespace/environment specific settings, and a reference to the conda-store app. It also loads and build time specific plugins. There is an example of registering/unregistering the lock plugin at build time (in the future, users may be able to specify settings/plugins to load per namespace/environment)
• there is an example artifact plugin for building a conda env export. There is also an example of using this plugin directly in a build task (pulling from the plugin registry)

There are definitely ways this can be extended. Some other ideas that would be nice to explore

• support for user installed plugins/plugin discovery
• implement all actions that build artifacts as plugins
• a nice next plugin could be a log plugin so conda-store could log to multiple endpoints at a time (for building the log artifact type + collecting build logs for operators)

[jaimergp]

This is excellent @soapy1! So much progress! 🚀

support for user installed plugins/plugin discovery

Isn't this done by pluggy already via Python entry points? 🤔

[soapy1]

Isn't this done by pluggy already via Python entry points? 🤔

Kind of. Pluggy provides tools for registering user plugins pretty easy. But we don't want to load all available plugins all the time. For example, if we have 3 storage plugins, we usually only care about having one configured. Right now this is resolved with the addition of the plugin registry main...soapy1:conda-store:plugin-demo-pluggy#diff-7266bb101b347632218746795c39cf8a2917ecf1f87b72bc2a4e7dbd50133db4. We'll still need to do some work on the plugin registry side to discover user plugins. But I think we can borrow some ideas from pluggy to make this work.