Poetry and Databricks

[cm-halfspace]

Does anyone have any experience with Poetry and Databricks?

We want to use it at my company where we have a databricks cluster running.
Currently we use an init script to install the python packages which is basically:

/databricks/python/bin/pip install -r /dbfs/mnt/bronze/libraries/requirements.txt

We want to move over to Poetry, however, we are having trouble getting Poetry up and running in the init scripts. It is a big mono-repo with sub-projects:

[tool.poetry]
name = "CompanyX"
version = "0.1.0"
description = ""
authors = []
readme = "README.md"
packages = []

[tool.poetry.group.main.dependencies]
python = ">=3.10,<3.12"

[tool.poetry.group.package1.dependencies]
package1 = { path = "PythonPackages/Package1", develop = true}

[tool.poetry.group.package2.dependencies]
package2 = { path = "PythonPackages/Package2", develop = true}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Does anyone have any experience in this area? I get almost zero google results on experiences with Databricks and Poetry combined.

[miltad]

Any progress on this topic so far?

[cm-halfspace]

From our side, unfortunately no. Right now we use Poetry for dependency management and then export a requirements.txt file that Databricks can then install, because we couldn't get Databricks to work with Poetry (virtual environments kept crashing). Super annoying since it just adds an unnecessary extra layer of complexity and brittleness.

[saraelshawa]

Hi, I was wondering if you found a fix for this? My environment keeps crashing as well when I try to get Databricks to work with Poetry. Any help would be appreciated, thanks!

[abhiaagarwal]

My company is using poetry and databricks in production. Check this out. https://github.com/databricks/bundle-examples/tree/main/knowledge_base/python_wheel_poetry.

[saraelshawa]

Thank you @abhiaagarwal !

[laigner]

Hello, I don't know if I get you right but what about using GitHub actions to automatically build a python wheel using poetry. Afterwards, use another GitHub action to upload the wheel to a location to which your Databricks service has access to. Then you can install your package as an own library, cluster wide, and use its functionality.

[abhiaagarwal]

Seems there's some work currently being done on this end for the Databricks Asset Bundles fyi. https://github.com/databricks/bundle-examples/tree/python-wheel-poetry/knowledge_base/python_wheel_poetry

[simonrmonk]

I'm leveraging a poetry plugin to run poetry build commands that play nicely with a mono-repo. Any idea if it will be possible to using poetry build in a databricks asset bundle that leverage a plugin? The plugin I have in mind is here: https://github.com/gerbenoostra/poetry-plugin-mono-repo-deps/