Default integration-test example does not work

[jmeidam]

I was scratching my head today about the integration-test that dbx provides in their examples (when you set up a project using dbx init). It suggests that you can run an integration test on Databricks using pytest.

In your project you will find an entrypoint (tests/entrypoint.py) that runs pytest.main on a directory (set in conf/deployment.yml).

Then in conf/deployment.yml there is a workflow with a spark_python_task, which refers to the entrypoint as python file.

In my project I listed all the typical test-related dependencies under my dev-dependencies and are therefore not listed in the wheel file (this is by design, we don't want these in a wheel). This results in my integration test failing (cannot import pytest). I tried running the test using poetry run dbx deploy followed by poetry run dbx launch <workflow name>.

My question: How can I run my integration test with the dev-dependencies, without having these be part of my wheel. I somehow need to let dbx know these also need to be installed on the cluster before running. Is there already such an option?

[j-4]

Hello,

we have the same issue: project.dependencies are installed from the wheel file, but the extra test dependencies are missing in the integration test workflow.

With dbx execute there is the option to pass extra dependencies:

dbx execute ... --pip-install-extras="test,other-extra,one-more-extra"

https://dbx.readthedocs.io/en/latest/guides/general/dependency_management/

However, with dbx deploy/launch as suggested for integration tests (https://dbx.readthedocs.io/en/latest/guides/python/integration_tests/, https://dbx.readthedocs.io/en/latest/guides/python/devops/package/) this does not seem to be possible.

[j-4]

We found a workaround for that issue, but official support by dbx would be preferred on the long run.

Add this cluster settings to deployment.yml (more infos here: https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsCreate):

tasks:
  - task_key: "test"
    new_cluster:
    ...
      spark_env_vars:
        WORKSPACE_DIST_DIR: "file:fuse://dist/.gitkeep"
      init_scripts:
        - dbfs:
            destination: "file://test-init-script.sh"
    ...

Add an empty dist/.gitkeep file to the repository
and also add a bash file to the root of the repository called test-init-script.sh with that content:

#!/bin/bash

WHEEL_FILE=$(ls ${WORKSPACE_DIST_DIR%/*}/*.whl)
echo "Installing test dependencies from $WHEEL_FILE[test]..."
/databricks/python/bin/pip install $WHEEL_FILE[test]

With the env var WORKSPACE_DIST_DIR a dummy .gitkeep file is uploaded to the assets which then serves to have the path to the wheel package in the init script. Suggestions to improve on that are more than welcome! (Any predefined env vars pointing to the assets dir?! Did not find anything here: https://docs.databricks.com/clusters/init-scripts.html#environment-variables)

Thus, the test extra dependencies in the setup.py or pyproject.toml are installed on the integration test cluster.

[bjtho08]

I was scratching my head today about the integration-test that dbx provides in their examples (when you set up a project using dbx init). It suggests that you can run an integration test on Databricks using pytest.

In your project you will find an entrypoint (tests/entrypoint.py) that runs pytest.main on a directory (set in conf/deployment.yml).

Then in conf/deployment.yml there is a workflow with a spark_python_task, which refers to the entrypoint as python file.

In my project I listed all the typical test-related dependencies under my dev-dependencies and are therefore not listed in the wheel file (this is by design, we don't want these in a wheel). This results in my integration test failing (cannot import pytest). I tried running the test using poetry run dbx deploy followed by poetry run dbx launch <workflow name>.

My question: How can I run my integration test with the dev-dependencies, without having these be part of my wheel. I somehow need to let dbx know these also need to be installed on the cluster before running. Is there already such an option?

I think this is an actual bug that should be reported as an issue rather than a discussion..

[copdips]

Another way to run integration test is to create a new job dedicated to integration test by dbx deploy, and then run it by dbx launch, write some tests in the end of the job itself. Dbx launch will fail if any error raised from the job. pytest and other dev requirements are not used here. Hope this could give you some ideas.