Intermittent test suite freezing in GitHub Actions potentially linked to pytest-sentry conflict

[drew2a]

We had a weird bug in our repo. With quite a low probability, our test suite run in GitHub Actions would freeze until the GitHub action timeout was triggered. This is odd, as we use pytest-timeout, which should handle these situations.

After adding faulthandler, we got a stack trace suggesting that the problem might lie in pytest-sentry.

Thread 0x00007f71fe57d640 (most recent call first):
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 302 in wait
  File "/home/runner/.virtualenvs/.venv/lib/python3.8/site-packages/sentry_sdk/_queue.py", line 240 in get
  File "/home/runner/.virtualenvs/.venv/lib/python3.8/site-packages/sentry_sdk/worker.py", line 127 in _target
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 870 in run
  File "/home/runner/.virtualenvs/.venv/lib/python3.8/site-packages/sentry_sdk/integrations/threading.py", line 70 in run
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007f720057f640 (most recent call first):
  File "/home/runner/.virtualenvs/.venv/lib/python3.8/site-packages/sentry_sdk/monitor.py", line 50 in _thread
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 870 in run
  File "/home/runner/.virtualenvs/.venv/lib/python3.8/site-packages/sentry_sdk/integrations/threading.py", line 70 in run
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/threading.py", line 890 in _bootstrap

Full stack trace

I'm not fully convinced that the problem is related only to pytest-sentry, but after removing it from the dependencies, we never saw the problem again.

Perhaps pytest-sentry was conflicting with one of the libraries from the requirements:

-r requirements.txt

pytest==7.4.3
pytest-aiohttp==1.0.5
pytest-asyncio==0.21.1
pytest-randomly==3.15.0
pytest-timeout==2.2.0
pytest-rerunfailures==12.0
pytest-sentry==0.1.16


coverage==7.3.2
looptime==0.2 ; sys_platform != 'win32'

# for pyqt5 test
pytest-qt==4.2.0

https://github.com/Tribler/tribler/blob/4a4e6034f34afeaa49539d4002f6dc7bf8f1337a/requirements-test.txt#L1-L16

Environment

➜ pip list | grep pytest-sentry
pytest-sentry        0.1.16

[untitaker]

we got a stack trace suggesting that the problem might lie in pytest-sentry.

this is a dump of all thread stacks. you found a thread by the sentry SDK that is waiting for new events to send out to sentry.io. the SDK uses this one to send out data to sentry without blocking your application. the other thread by the SDK is for something related to the crons product, but should also not prevent your app from exiting.

there are only two things on the top of my mind that can prevent an app from exiting:

1. something is blocking your main thread (in your case this would be where the coverage.py CLI runs, /home/runner/.virtualenvs/.venv/bin/coverage)
2. there is a thread that runs with daemon=False and is not exiting (both SDK threads run with daemon=True)

since i don't see any other threads in your dump, i think you are stuck in:

  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/asyncio/base_events.py", line 1823 in _run_once

[untitaker]

Here are the two lines where daemon is set to True:

• https://github.com/getsentry/sentry-python/blob/4d10edfe7233d5adc2ceeeb984d8f93dfa3a29eb/sentry_sdk/monitor.py#L55 -- this one is for the "monitor" thread
• https://github.com/getsentry/sentry-python/blob/4d10edfe7233d5adc2ceeeb984d8f93dfa3a29eb/sentry_sdk/worker.py#L69 -- this one is for the thread you posted, the "transport" thread