[BUG] Install fails on python 3.12 due to databricks-cli's access to removed imp in setup.py

[jmahlik]

Issues Policy acknowledgement

• I have read and agree to submit bug reports in accordance with the issues policy

Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

MLflow version

• Client: 2.71.
• Tracking server: 2.7.1

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): all
• Python version: python 3.12
• yarn version, if running the dev UI: n/a

Describe the problem

mlflow depends on the databricks-cli which is using the removed module imp in the setup.py.

I was looking through the README for the databricks-cli and it mentions moving to the databricks-sdk-py.

I wasn't sure if it's worth pursuing moving to the python sdk or pursuing updates to the cli's setup.py.

Thoughts?

Tracking information

n/a doesn't install

Code to reproduce issue

python --version
Python 3.12.0
python -m pip install mlflow-skinny==2.7.1

Stack trace

python -m pip install mlflow-skinny==2.7.1

# redacted
...

databricks-cli-0.17.8.tar.gz (85 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "py312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "py312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "py312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "overlay\Lib\site-packages\setuptools\build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "pip-build-env-sltisind\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "\overlay\Lib\site-packages\setuptools\build_meta.py", line 507, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "overlay\Lib\site-packages\setuptools\build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 24, in <module>
      ModuleNotFoundError: No module named 'imp'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Other info / logs

n/a

What component(s) does this bug affect?

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/gateway: AI Gateway service, Gateway client APIs, third-party Gateway integrations
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

What language(s) does this bug affect?

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

What integration(s) does this bug affect?

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

[jmahlik]

Related databricks/databricks-cli#631

[harupy]

@jmahlik Thanks for reporting this issue!

[harupy]

It looks like databricks-cli doesn't provide a wheel for python 3?

https://pypi.org/project/databricks-cli/0.17.6/#files

[harupy]

I wasn't sure if it's worth pursuing moving to the python sdk.

This makes sense to me.

[jmahlik]

It looks like databricks-cli doesn't provide a wheel for python 3?

https://pypi.org/project/databricks-cli/0.17.6/#files

Yea... so we end up invoking the setup.py. I have a PR ready that'll ship universal wheels and fix the deprecated imports.

[harupy]

@jmahlik We're going to release databricks-cli 0.18. The imp module should be removed in this version :)

[harupy]

@jmahlik https://pypi.org/project/databricks-cli/0.18.0 was released, but mlflow installation on python 3.12 is still failing because pyarrow is not ready (apache/arrow#37880).