Not able to every package with pip install lakehouse_engine

[alexiska]

I am trying to pip install lakehouse_engine in Databricks and get following error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
python-lsp-server 1.7.1 requires jedi<0.19.0,>=0.17.2, but you have jedi 0.19.1 which is incompatible.
databricks-sdk 0.1.6 requires requests<2.29.0,>=2.28.1, but you have requests 2.31.0 which is incompatible.
virtualenv 20.16.3 requires platformdirs<3,>=2.4, but you have platformdirs 3.11.0 which is incompatible.

Not sure if it is due to Databricks runtime version but I am using 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12) at the moment.

Any idea how to solve this or is this package not compatible with Databricks?

[PMRocha]

Hello @alexiska ,
This library is compatible with Databricks and, when I tried to install using pip con a cluster running DBR 13.3 I get no error.

How are you making the installation?

[PMRocha]

Hello,
the Lakehouse Engine was designed with S3 in mind but it should work for any backend except for some functions that were specifically designed for interacting with S3. The file manager is currently working only with S3 and the default behavior of the data quality checks is to store information in an S3 bucket, you can however use your local file system.

If you face any problems please open an issue in the repo.

[alexiska]

That's unfortunate, I am using azure cloud storage

[PMRocha]

Sorry if I wasn't clear in my previous message. The way the framework is designed you should be able to use any backend that spark can read from/write to. In the case of data quality you can use local file system instead of an S3 bucket. Currently there is no alternative for the functions of the file manager.

[alexiska]

is there any plan to add the other cloud provider like azure?

[PMRocha]

Currently there are no plans to add the support, but feel free to contribute.