template-python-project

• template-python-project
  • Development environment
    • WARNING for conda users
    • IDE: Visual Studio Code
    • Project and dependency management
    • Manage Python versions
    • Virtual environment
      • Running your classical commands in a poetry virtual environment
  • Dependency management
    • Define main (mandatory) dependencies
    • Organize your dependencies
      • Define optional groups
    • Define extra dependencies
  • Installation
    • Install the package as a dependency for another project
      • From a repository
        • From PyPI
      • From a git repository
      • From a local directory
    • Install in editable mode and potentially contribute to the project
      • Editable install with poetry
        • Install group dependencies
      • Editable install with pip
        • Install extras dependencies
  • Testing
    • Run tests using VSCode
  • Debugging files and tests files using VSCode
  • Documentation
    • Install documentation dependencies
    • Configure the documentation
    • Generate the documentation locally
    • Build and publish the documentation online
      • GitHub pages
      • Read the Docs
  • Packaging and publishing
    • Build the package
    • Publish the package on a Package Index (PI)
      • TestPyPI
      • PyPI
  • Continuous integration
    • Test installation of the package and run tests
    • Code coverage with Codecov
  • Miscellaneous

This repository may serve as a template for scientific projects developed in Python using poetry.

Development environment

WARNING for conda users

If you have Anaconda or Miniconda installed,

1. DISABLE the auto-activation of the base environment

   conda config --set auto_activate_base false

2. ALWAYS DEACTIVATE your conda environment before running any of the following commands

   conda deactivate

IDE: Visual Studio Code

Visual Studio Code (VSCode) is recommended to simplify your coding experience.

The .vscode directory contains a list of suggested extensions together with the corresponding settings. You can place it at the root of your project workspace.

See also the vscode-workflow repository.

Project and dependency management

poetry Python Packaging And Dependency Management Made Easy

poetry>=1.2 is recommended for its simplicity to manage Python projects in many ways and make them meet the Python packaging standards.

1. Read WARNING for conda users,

2. See poetry's installation instructions,

3. Check your poetry version

   poetry --version

If you get an error, please check python-poetry/poetry/issues/507

Manage Python versions

To easily switch between Python versions, it is recommended to use pyenv or similar tools.

# cd REPOSITORY_NAME
pyenv install 3.9.8
pyenv local 3.9.8  # Activate Python 3.9.8 for the current project

See also the Real Python's tutorial "Intro to pyenv".

Virtual environment

A virtual environment is a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.

It is always good practice to work in a virtual environment, isolated from your other Python projects.

With poetry creating/activating a virtual environment is fairly simple with the poetry shell command

Please read WARNING for conda users)

# conda deactivate  # for conda users
poetry shell  # create/activate virtual environment (see poetry.toml)
exit  # to exit the virtual environment

Note

In this template project, as defined in the poetry.toml file, the command poetry shell will create a new/activate the virtual environment named (.venv) at the root of the project in a .venv folder See also poetry's documentation.

If you don’t want poetry to manage my virtual environments. Can I disable it?

Running your classical commands in a poetry virtual environment

Prepend poetry run to your classical commands (e.g. poetry run pytest) to make sure they are executed in the dedicated virtual environment.

See examples in the following sections

• Testing,
• Documentation.

Dependency management

poetry is recommended for its dependency management capabilities.

poetry makes use of a single file: the pyproject.toml file.

Nowadays, since PEP 517, PEP 518, PEP 660, PEP 621, the pyproject.toml file is used to define and manage your project metadata, build system, dependencies, packaging options, third party packages configurations, etc. In particular, it replaces the need for setup.py, setup.cfg, requirements.txt, ~~pytest.ini~~ files.

On top of installing a given dependency (like pip would do), poetry also resolves dependencies' version (like conda would do, but faster). The result of the resolution procedure is stored in the poetry.lock file. This guarantees full reproducibility of your project setup!

Anyone who sets up the project with poetry when the file poetry.lock is present uses the exact same versions of the dependencies that you are using.

Define main (mandatory) dependencies

Use the command poetry add and check the result in the section [tool.poetry.dependencies] of the pyproject.toml file

poetry add PACKAGENAME
# poetry add "PACKAGENAME[EXTRANAME1,EXTRANAME2]"
poetry remove PACKAGENAME

Organize your dependencies

With poetry you can organize your dependencies in groups to manage them in a more granular way

To do this, use the command poetry add PACKAGENAME --group GROUPNAME and check the result in the corresponding section [tool.poetry.group.GROUPNAME.dependencies] of the pyproject.toml file.

For example, to add pytest to the dev (development) group use

poetry add pytest --group dev
poetry remove pytest --group dev

Define optional groups

If you wish to make a group of dependencies optional, add the following to the pyproject.toml file

[tool.poetry.group.GROUPNAME]
optional = true

# [tool.poetry.group.GROUPNAME.dependencies]

See also install group dependencies.

Define extra dependencies

Single dependecies can also be made optional

poetry add PACKAGENAME1 --optional
poetry add PACKAGENAME2 --optional

and then combined to define package extra dependencies in the pyproject.toml.

[tool.poetry.extras]
EXTRANAME = ["PACKAGENAME1", "PACKAGENAME2"]

See also Install the package with extra dependencies.

Installation

Install the package as a dependency for another project

Use the command poetry add which, to some extend, is similar to pip install.

From a repository

By default, Poetry discovers and installs packages from PyPI. This represents most cases and will likely be enough for most users.

If the target package is available from a different source than PyPI, you can configure poetry and add this new source, see the documentation

From PyPI

When the target project is available on PyPI

poetry add PACKAGENAME
# pip install PACKAGENAME
poetry add "PACKAGENAME[EXTRANAME1,EXTRANAME2]"
# pip install "PACKAGENAME[EXTRANAME1,EXTRANAME2]"
poetry add --editable "PACKAGENAME[EXTRANAME1,EXTRANAME2]"
# pip install --editable "PACKAGENAME[EXTRANAME1,EXTRANAME2]"

From a git repository

When the target project is available online

poetry add git+https://github.com/USERNAME/REPOSITORY_NAME.git
# pip install git+https://github.com/USERNAME/REPOSITORY_NAME.git
poetry add --editable git+https://github.com/USERNAME/REPOSITORY_NAME.git
# pip install --editable git+https://github.com/USERNAME/REPOSITORY_NAME.git

From a local directory

When the target project is located in a specific directory or file poetry add --editable ./my-package/

poetry add PATH_TO_PACKAGE
# pip install PATH_TO_PACKAGE
poetry add --editable PATH_TO_PACKAGE
# pip install --editable PATH_TO_PACKAGE

Install in editable mode and potentially contribute to the project

By default, dependencies across all non-optional groups will be installed when executing poetry install.

Editable install with poetry

By default, the command poetry install installs the package in editable mode.

git clone https://github.com/USERNAME/REPOSITORY_NAME.git
cd REPOSITORY_NAME
poetry shell  # create/activate virtual environment (see poetry.toml)
poetry install  # main + non-optional group dependencies
poetry install --without GROUPNAME # main + non-optional - GROUPNAME dependencies

Install group dependencies

Optional group dependencies, see Define optional groups, can be installed using the --with, --only flags

poetry install --with GROUPNAME1 --without GROUPNAME2
poetry install --only GROUPNAME3

Editable install with pip

Consider using pip>=21.3.1, when installing packages defined by a pyproject.toml file.

git clone https://github.com/USERNAME/REPOSITORY_NAME.git
cd REPOSITORY_NAME
# activate your virtual environment  # e.g. conda activate ENVIRONMENT_NAME
pip install --editable .
# pip install --editable ".[EXTRANAME1, EXTRANAME2]"  # to install extra dependencies

See also the pip install optional commands.

Install extras dependencies

• With poetry, dependencies defined as extra dependencies in the [tool.poetry.extras] section of the pyproject.toml, simply run

  poetry install --extras "EXTRANAME1 EXTRANAME2"
  # poetry install -E EXTRANAME1 -E EXTRANAME2

  See also poetry's documentation.

• With pip, dependencies defined as extra dependencies in the [project.optional-dependencies] section of the pyproject.toml, simply run

  pip install ".[EXTRANAME1, EXTRANAME2]"

  See also pip install optional commands.

Testing

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing.

The unit tests of the package are declared in tests/test_*.py files as test_* functions with a simple assert statement.

The configuration of pytest is defined in the [tool.pytest.ini_options] section of the pyproject.toml file.

Run the package test suite with

# poetry shell  # create/activate virtual environment (see poetry.toml)
# poetry install --with dev # install main and development dependencies [tool.poetry.group.dev.dependencies]
poetry run pytest

Run tests using VSCode

See the Testing section of the VSCode documentation.

Debugging files and tests files using VSCode

The configuration file .vscode/launch.json contains two configurations for debugging

1. Python generic files
2. Python unit-test files

For more details, see the VSCode documentation

• Debugging,
• Debug Tests.

Documentation

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation.

Sphinx is in charge of building the documentation and generating HTML output, but also PDF, epub, ...

The source files of the documentation are simply .rst (reStructuredText) or .md (Markdown) files. However we suggest using the reST markup to keep the same syntax and format as used for writing Python docstings.

Install documentation dependencies

Install the documentation dependencies defined under [tool.poetry.group.docs.dependencies] in pyproject.toml

poetry shell  # create/activate virtual environment (see poetry.toml)
poetry install --with docs
# poetry install --only docs

Configure the documentation

The configuration file is located at docs/conf.py.

• Edit the metadata of the package documentation defined in docs/conf.py,
• See also the sphinx documentation

Generate the documentation locally

To generate the documentation locally, i.e., on your machine,

# poetry shell  # create/activate virtual environment (see poetry.toml)
# poetry install --with docs
poetry run sphinx-build -b html docs docs/_build/html

and navigate the documentation

open docs/_build/html/index.html

Important: Any change made in the source .py files or the docs/conf.py file require rebuilding the documentation.

Build and publish the documentation online

Choose either

• GitHub pages (default), or
• Read the Docs.

GitHub pages

The documentation can be built according to the GitHub workflow .github/workflows/docs.yml and deployed from the gh-pages branch via GitHub pages at https://USERNAME.github.io/template-python-project.

To make this work automatically, follow the simple steps below (only once!)

1. Push you latest changes

   • the gh-pages branch will be automatically created/updated once the GitHub workflow .github/workflows/docs.yml is triggered

2. On your GitHub repository

   • Go to Settings -> Pages -> Source
     • select branch: gh-pages
     • select folder: /root

Read the Docs

Read the Docs simplifies software documentation by automating building, versioning, and hosting of your docs for you.

After linking your project with Read the Docs, you can configure Read the Docs to build and deploy the documentation of the package at https://repository-name.readthedocs.io/en/latest/, either automatically or manually.

See also the .readthedocs.yaml configuration file.

Packaging and publishing

poetry is also of great help to simplify the packaging process.

Build the package

Before publication, poetry can package your project

poetry build

See also PyPA's build package.

Publish the package on a Package Index (PI)

TestPyPI

It is good practice to first publish on TestPyPI and check the results before publishing on the official PyPI, see PyPI section.

Let's configure poetry to publish your package on TestPyPI

• poetry documentation
• poetry configure credentials
• TestPyPI token

poetry config repositories.testpypi https://test.pypi.org/legacy/
poetry config http-basic.testpypi __token__ MY_TOKEN
# poetry build
poetry publish --repository testpypi

• Check the results https://test.pypi.org/project/packagename/

• Check your package installation from TestPyPI

  # create a new virtual environment and run
  # conda create -n test-install-testpypi python=3.8 pip
  # conda activate test-install-testpypi
  pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ packagename

See also PyPA twine package.

PyPI

PyPI is the official Python Package Index

By default, poetry is configured to use the PyPI repository, for package installation and publishing.is the default repository

• poetry publish to PyPI

• poetry configure credentials

• PyPI token

  poetry config pypi-token.pypi MY_TOKEN
  # poetry build
  poetry publish

• Check the results https://pypi.org/project/packagename/

• Check your package installation from PyPI

  # create a new virtual environment and run
  # conda create -n test-install-pypi python=3.8 pip
  # conda activate test-install-pypi
  pip install packagename

See also PyPA twine package.

Continuous integration

GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want.

GitHub workflows declared as .github/workflows/*.yml files.

Test installation of the package and run tests

See the corresponding sections in the .github/workflows/ci.yml

Code coverage with Codecov

• See the corresponding section in the .github/workflows/ci.yml
• codecov
• codecov-action

Miscellaneous

Consider reading

• Zen of Python
• Python naming algorithm