Dockerise testing for dev #1093

Scott-Simmons · 2024-11-14T14:01:15Z

Related: #994.

Easiest to go commit by commit. Uses 3 "flows"

(1) Test locally on specific python version Is useful for quick iteration.
(2) Test locally on the discoverable python versions Is a useful sanity check for ensuring new development works on multiple versions of python.
(3) Test in dedicated testing environment with guaranteed availability of python versions Is a good final sanity check. Takes longer, but creates a reproducible environment to test off and is a good 'source of truth'.

1 Test locally on specific python version

Testing time for one version is ~1 min. Note this includes the speedup from cached packages

2 Test locally on the discoverable python versions

Testing time across versions 3.7-3.11 is ~4 mins, but note that this speedup includes the speedup from cached packages.

Also note that python 3.7 fails on my machine due to some idiosyncratic issue related to my setup py37/lib/python3.7/site-packages/pip/_vendor/typing_extensions.py ... getting same error as here but not using vscode. Not an issue for me personally but shows an example of why testing within an isolated testing environment could be a good way to ensure reproducible version targeting.

3 Test in dedicated testing environment with guaranteed availability of python versions

Build time for testenv ~5 mins but this is a once-off build for the testenv.

Testing time across versions 3.7-->3.11 is ~6 mins.

Add image for testing tsfresh with multiple python versions. This can be a good final sanity check e.g. before a release.

Targets include 1. Building dockerised testing environment 2. Testing tsfresh within testing environment 3. Testing tsfresh locally with tox with available python versions 4. Testing tsfresh locally with current active python version 5. Cleaning build artifacts 6. Formatting. This is so that changes can be made before pushing the changes up the remote to trigger the formatting checks. TODO: pre-commit hooks should probs be added too.

Change docs to reflect the new testing invokations.

Makefile

Scott-Simmons · 2024-11-17T03:31:30Z

@nils-braun let me know If this is worth adding, hoping that it can helpful for the release process going forward

nils-braun

Thanks @Scott-Simmons - similarly to your last PR this is of very high quality. I do have a few suggestions around the single-line makefile commands but I am in general happy with this. I do not think all users will require it, but I do think that it can be a very good UX improvement for some users. Well done!

nils-braun · 2024-11-17T06:10:43Z

Makefile

+TEST_DOCKERFILE := Dockerfile.testing
+TEST_CONTAINER := tsfresh-test-container
+# >= 3.9.2 ---> https://github.com/dask/distributed/issues/7956
+PYTHON_VERSIONS := "3.7.12 3.8.12 3.9.12 3.10.12 3.11.0"


Do we really need to specify also the patch version number? I do not want to create the need to update this file whenever there is a new python version coming out.

3ddad59

I see all I need to do is specify major.minor like 3.10 and pyenv will interpret this as latest.

nils-braun · 2024-11-17T06:14:26Z

Makefile

+
+# Tests `PYTHON_VERSIONS`, provided they are also
+# specified in setup.cfg `envlist`
+test-all-testenv: clean build-testenv


Why do you need to call clean?

More sensible make layout which cleans after the tests are invoked in docker: 3ddad59

Calls to clean are required for local testing if test-all-testenv is run beforehand. Because those files will be owned by root and generated, it will brick the testing for local pytest and tox. see here:

However on reflection it makes more sense to move clean into executing after the docker tests run.

Makefile

nils-braun · 2024-11-17T06:17:44Z

Makefile

+# current context (e.g. global or local version of
+# python set by pyenv, or the python version in
+# the active virtualenv).
+test-local: clean


Same question: why do you need to clean?

Moved to cleaning within the docker testing target 3ddad59

nils-braun · 2024-11-17T06:21:50Z

docs/text/how_to_contribute.rst

@@ -64,21 +64,32 @@ you have to:

 .. code::

-    pytest
+    make test-local


With this, users will need to reinstall the pip packages for testing on very run of the test. I know, it should be quick as all packages are already there, but it does take some extra time nevertheless.

I would suggest to not add a makefile command for the local testing but to state that the users need to run pytest. The makefile command just replaced two commands and the installation command is explained on the page already anyways

Removed all the one-line makefile commands 3ddad59

nils-braun · 2024-11-17T06:23:51Z

Makefile

+clean:
+	rm -rf .tox build/ dist/ *.egg-info
+
+install-formatters:


Why is this command needed and the one below? We have pre-commit for exactly this, or?

Removed these targets 3ddad59

I was not aware that the precommit check runs locally with each commit, but I see that it is in the docs so we do not need this as you pointed out.

nils-braun · 2024-11-17T06:25:11Z

docs/text/how_to_contribute.rst



-`tox -r` will execute tests for the Python versions specified in `setup.cfg <https://github.com/blue-yonder/tsfresh/blob/1297c8ca5bd6f8f23b4de50e3f052fb4ec1307f8/setup.cfg>`_ using the `envlist` variable. For example, if `envlist` is set to `py37, py38`, the test suite will run for Python 3.7 and 3.8 on the local development platform, assuming the binaries for those versions are available locally. The exact Python microversions (e.g. `3.7.1` vs `3.7.2`) depend on what is installed on the local development machine.
+This will execute tests for the Python versions specified in `setup.cfg <https://github.com/blue-yonder/tsfresh/blob/1297c8ca5bd6f8f23b4de50e3f052fb4ec1307f8/setup.cfg>`_ using the `envlist` variable. For example, if `envlist` is set to `py37, py38`, the test suite will run for Python 3.7 and 3.8 on the local development platform, assuming the binaries for those versions are available locally. The exact Python microversions (e.g. `3.7.1` vs `3.7.2`) depend on what is installed on the local development machine.


Sorry, did not spot this in the last PR: can you make sure the link points to the latest HEAD/main version of the file?

Responded to feedback here 3ddad59

This might result in dead links in the future e.g. if we change the file and not a great way to trace them. If we pin to a specific commit even if the file changes, it will be more traceable as to what happened, instead of a 404.

nils-braun · 2024-11-17T06:27:27Z

docs/text/how_to_contribute.rst



 To test changes across multiple versions of Python, run:


 .. code::

-    tox -r
+    make test-all-local



Similar to above: I am not a big fan of replacing Single-Line commands with a makefile command. This adds another layer of indirection "advanced" users can be annoyed about (they know what tox is, so they want to potentially change options etc) and "new" users copy-paste what is in here anyways, so it does not matter.

Reverted the single line makefile commands + updated the docs 3ddad59

That makes sense. I was thinking that makefile is a useful place to act as a single point of entry for people new to the project to understand how to run it.

But since the docs fulfill that purpose, understandable why that is not desired. Will remove the syntactic sugar.

One advantage is that by encapsulating in a makefile, suppose tox -r evolves to tox -r -other_flag -another_flag etc... then using a makefile means the user only needs to think about make foobar instead of the flags. Without needing to update the docs. But this is probably not super crucial to have for the tsfresh project.

I do not know for tox, but many tools allow to set default options on the pyproject or setup.cfg or their own config file. This is typically better, because then the tool can do its own "schema" migration. For example, we so this for pytest in the setup.cfg.

- Remove the python PATCH version from makefile - Move calls to `make clean` to be part of the docker test invokation instead of within local test commands - Removed the one line makefile commands (the docs have enough info on the development process so will not bother using `make` for these. - Updated hyperlink to point to main instead of specific commit sha

Scott-Simmons

Hey @nils-braun, have responded to the feedback. Let me know if there is anything else

nils-braun

Looks good!

Scott-Simmons force-pushed the dockerise-testing-for-dev branch 2 times, most recently from 469f111 to 6773ae4 Compare November 14, 2024 21:54

Scott-Simmons marked this pull request as ready for review November 14, 2024 21:54

Scott-Simmons marked this pull request as draft November 14, 2024 21:58

Scott-Simmons added 4 commits November 15, 2024 16:17

Add a dockerfile for testing

14ba6d6

Add image for testing tsfresh with multiple python versions. This can be a good final sanity check e.g. before a release.

Nit: add document start on yaml file

b720398

Documentation updates

27f02b9

Change docs to reflect the new testing invokations.

Scott-Simmons force-pushed the dockerise-testing-for-dev branch from 6773ae4 to 27f02b9 Compare November 15, 2024 05:17

Scott-Simmons marked this pull request as ready for review November 15, 2024 05:18

Scott-Simmons commented Nov 17, 2024

View reviewed changes

Makefile Show resolved Hide resolved

nils-braun reviewed Nov 17, 2024

View reviewed changes

Scott-Simmons commented Nov 19, 2024

View reviewed changes

nils-braun approved these changes Nov 19, 2024

View reviewed changes

nils-braun merged commit b970a2e into blue-yonder:main Nov 19, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dockerise testing for dev #1093

Dockerise testing for dev #1093

Scott-Simmons commented Nov 14, 2024 •

edited

Loading

Scott-Simmons commented Nov 17, 2024

nils-braun left a comment

nils-braun Nov 17, 2024

Scott-Simmons Nov 18, 2024

nils-braun Nov 17, 2024

Scott-Simmons Nov 18, 2024

nils-braun Nov 17, 2024

Scott-Simmons Nov 18, 2024

nils-braun Nov 17, 2024

Scott-Simmons Nov 18, 2024

nils-braun Nov 17, 2024

Scott-Simmons Nov 18, 2024

nils-braun Nov 17, 2024

Scott-Simmons Nov 18, 2024

nils-braun Nov 17, 2024

Scott-Simmons Nov 18, 2024

nils-braun Nov 19, 2024

Scott-Simmons left a comment

nils-braun left a comment



		`tox -r` will execute tests for the Python versions specified in `setup.cfg <https://github.com/blue-yonder/tsfresh/blob/1297c8ca5bd6f8f23b4de50e3f052fb4ec1307f8/setup.cfg>`_ using the `envlist` variable. For example, if `envlist` is set to `py37, py38`, the test suite will run for Python 3.7 and 3.8 on the local development platform, assuming the binaries for those versions are available locally. The exact Python microversions (e.g. `3.7.1` vs `3.7.2`) depend on what is installed on the local development machine.
		This will execute tests for the Python versions specified in `setup.cfg <https://github.com/blue-yonder/tsfresh/blob/1297c8ca5bd6f8f23b4de50e3f052fb4ec1307f8/setup.cfg>`_ using the `envlist` variable. For example, if `envlist` is set to `py37, py38`, the test suite will run for Python 3.7 and 3.8 on the local development platform, assuming the binaries for those versions are available locally. The exact Python microversions (e.g. `3.7.1` vs `3.7.2`) depend on what is installed on the local development machine.

Dockerise testing for dev #1093

Dockerise testing for dev #1093

Conversation

Scott-Simmons commented Nov 14, 2024 • edited Loading

1 Test locally on specific python version

2 Test locally on the discoverable python versions

3 Test in dedicated testing environment with guaranteed availability of python versions

Scott-Simmons commented Nov 17, 2024

nils-braun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Scott-Simmons left a comment

Choose a reason for hiding this comment

nils-braun left a comment

Choose a reason for hiding this comment

Scott-Simmons commented Nov 14, 2024 •

edited

Loading