Testing & Tooling

[srikanthccv]

It's lot of frustrating to deal with out of the blue CI failures. Some times docs build fails for no reason; docker-tests that run successfully all the time suddenly fails; pypy3 tests fails occasionally; benchmark-tests failures show up once in a while even when the changes aren't related. And there are multiple linters & formatters.

Please share your observations and thoughts. Hopefully we will have consolidated action item to work on post rc1.

[Oberon00]

And there are multiple linters & formatters.

At least these don't cause spurious failures, do they? And they could all be ran easily with scripts/eachdist.py lint if someone would maintain eachdist.ini 😉

But I can definitely say that the build of odin-python is quite convoluted. Running tox takes ages, if you only want to run parts to cover your changes, you have to know what these are first, and the command can become quite long.

Also, the build does not work on Windows, and AFAIK this is not documented anywhere.

[codeboten]

Not addressing the bigger issue surrounding the builds, but the issue around the docs build was specifically triggered by tox cache causing issues w/ the docs build when there are changes in the api/sdk. I've updated the docs target in tox to always recreate its tox environment. https://github.com/open-telemetry/opentelemetry-python/pull/1560/files#diff-ef2cef9f88b4fe09ca3082140e67f5ad34fb65fb6e228f119d3812261ae51449R174

We should consider better strategies around tox caching where possible and maybe make the performance tests run separately in a job that is non blocking to the build. Performance testing on shared infrastructure is never very reliable...

[aabmass]

• Do we know how the tox env for docs got out of sync? Maybe the hash code needs to be more specific, I see it is missing the docs-requirements.txt file:

  opentelemetry-python/.github/workflows/test.yml

  Line 115 in 59f5bab

  key: tox-cache-${{ matrix.tox-environment }}-${{ hashFiles('tox.ini', 'dev-requirements.txt') }}-core

• We should probably add a "cache version" number to the key that can be easily bumped as a solution when CI is failing to "clear" the cache. So something like key: v${{cache-version}}-tox-cache-${{ matrix.tox-environment }}-${{ hashFiles('tox.ini', 'dev-requirements.txt') }}-core.
• We could try only caching the pip wheel cache and not the .tox folder. That way tox will recreate the envs every time in CI but shouldn't need to actually download anything unless the dep versions change

[aabmass]

Alternatively, ReadTheDocs can also do the docs CI (reports on the PR and everything) https://docs.readthedocs.io/en/stable/guides/autobuild-docs-for-pull-requests.html