Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redis - Losing Unacked Messages - WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM) #1836

Open
poirot-19 opened this issue Dec 20, 2023 · 0 comments
Open

Redis - Losing Unacked Messages - WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM) #1836

poirot-19 opened this issue Dec 20, 2023 · 0 comments

Comments

@poirot-19
Copy link

Versions:

  • Python 3.10
  • Ubuntu 22.04
  • flask-redis 0.4.0
  • celery 5.2.7

Using Redis as Broker for Celery, and have set below configs -

CELERY_BROKER_URL = redis://localhost:6379/0
CELERY_RESULT_BACKEND = redis://localhost:6379/0
CELERY_ACCEPT_CONTENT = ["json"]
CELERY_TASK_SERIALIZER = "json"
CELERY_RESULT_SERIALIZER = "json"
CELERY_REDIS_MAX_CONNECTIONS = 5
CELERY_BEAT_URL = redis://localhost:6379/2
CELERY_IGNORE_RESULT = True
CELERY_STORE_ERRORS_EVEN_IF_IGNORED = True
CELERYD_PREFETCH_MULTIPLIER = 1
CELERY_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True

Below is the celery task task.py-

@celery.task(queue="long_task")
def celery_task():
    try:
        #long running task - 
        print("will run for long time")
    except Exception as e:
        logger.exception(f"Uncaught Exception: {e}")

Command to start Celery -

celery worker -A task --concurrency=7 --max-tasks-per-child=1  -Q long_task

I am running the above process on Ubuntu 22.04 and while this worker is executing multiple tasks parallely, I am stopping the service - sudo systemctl stop celery-worker.service
Occasionally its sending back the un-acked messages, which I want -

[2023-12-20 13:00:38] [CRITICAL] [f:beat.py] [l:111] [n:celery.beat] beat raised exception <class 'redis.exceptions.LockError'>: LockError('Cannot release an unlocked lock')
Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 635, in start
    time.sleep(interval)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 154, in _sync
    service.sync()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
    self.scheduler.close()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
    self.lock.release()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 226, in release
    self.do_release(expected_token)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 232, in do_release
    raise LockNotOwnedError("Cannot release a lock"
redis.exceptions.LockNotOwnedError: Cannot release a lock that's no longer owned

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 109, in start_scheduler
    service.start()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 641, in start
    self.sync()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
    self.scheduler.close()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
    self.lock.release()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 224, in release
    raise LockError("Cannot release an unlocked lock")
redis.exceptions.LockError: Cannot release an unlocked lock
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] --- Logging error ---
[2023-12-20 13:00:38] [ERROR] [f:tasks.py] [l:163] [n:background.tasks] Shutting down worker immediately, pushing back the currently executing tasks to redis queue
[2023-12-20 13:00:38] [ERROR] [f:tasks.py] [l:163] [n:background.tasks] Shutting down worker immediately, pushing back the currently executing tasks to redis queue
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Traceback (most recent call last):
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 635, in start
    time.sleep(interval)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 154, in _sync
    service.sync()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
    self.scheduler.close()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
    self.lock.release()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 226, in release
    self.do_release(expected_token)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 232, in do_release
    raise LockNotOwnedError("Cannot release a lock"
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] redis.exceptions.LockNotOwnedError: Cannot release a lock that's no longer owned
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] During handling of the above exception, another exception occurred:
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Traceback (most recent call last):
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 109, in start_scheduler
    service.start()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 641, in start
    self.sync()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
    self.scheduler.close()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
    self.lock.release()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 224, in release
    raise LockError("Cannot release an unlocked lock")
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] redis.exceptions.LockError: Cannot release an unlocked lock
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] During handling of the above exception, another exception occurred:
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Traceback (most recent call last):
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/usr/lib/python3.10/logging/__init__.py", line 1104, in emit
    self.flush()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/usr/lib/python3.10/logging/__init__.py", line 1084, in flush
    self.stream.flush()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 154, in _sync
    service.sync()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
    self.scheduler.close()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
    self.lock.release()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 224, in release
    raise LockError("Cannot release an unlocked lock")
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] redis.exceptions.LockError: Cannot release an unlocked lock
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Call stack:
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/usr/local/bin/celery", line 8, in <module>
    sys.exit(main())
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/__main__.py", line 16, in main
    _main()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 322, in main
    cmd.execute_from_commandline(argv)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 491, in handle_argv
    return self.execute(command, argv)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 415, in execute
    return cls(
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 308, in run_from_argv
    return self.handle_argv(prog_name,
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 393, in handle_argv
    return self(*args, **options)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 253, in __call__
    ret = self.run(*args, **kwargs)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/beat.py", line 109, in run
    return beat().run()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 81, in run
    self.start_scheduler()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 111, in start_scheduler
    logger.critical('beat raised exception %s: %r',
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Message: 'beat raised exception %s: %r'
Arguments: (<class 'redis.exceptions.LockError'>, LockError('Cannot release an unlocked lock'))
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Traceback (most recent call last):
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 635, in start
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] time.sleep(interval)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 154, in _sync
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] service.sync()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.scheduler.close()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.lock.release()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 226, in release
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.do_release(expected_token)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 232, in do_release
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] raise LockNotOwnedError("Cannot release a lock"
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] redis.exceptions
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] .
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] LockNotOwnedError
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] :
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Cannot release a lock that's no longer owned
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] During handling of the above exception, another exception occurred:
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Traceback (most recent call last):
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/usr/local/bin/celery", line 8, in <module>
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] sys.exit(main())
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/__main__.py", line 16, in main
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] _main()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 322, in main
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] cmd.execute_from_commandline(argv)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] super(CeleryCommand, self).execute_from_commandline(argv)))
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return self.handle_argv(self.prog_name, argv[1:])
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 491, in handle_argv
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return self.execute(command, argv)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 415, in execute
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return cls(
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 308, in run_from_argv
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return self.handle_argv(prog_name,
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 393, in handle_argv
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return self(*args, **options)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 253, in __call__
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] ret = self.run(*args, **kwargs)
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/beat.py", line 109, in run
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return beat().run()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 81, in run
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.start_scheduler()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 109, in start_scheduler
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] service.start()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 641, in start
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.sync()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.scheduler.close()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.lock.release()
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 224, in release
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] raise LockError("Cannot release an unlocked lock")
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] redis.exceptions
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] .
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] LockError
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] :
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Cannot release an unlocked lock
[2023-12-20 13:00:38] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Restoring 7 unacknowledged message(s)

But occasionally, its not sending back the currently running tasks, and is exiting with this message, resulting in loss of tasks -

[2023-12-20 13:22:44] [CRITICAL] [f:beat.py] [l:111] [n:celery.beat] beat raised exception <class 'redis.exceptions.LockError'>: LockError('Cannot release an unlocked lock')
Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 635, in start
    time.sleep(interval)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 154, in _sync
    service.sync()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
    self.scheduler.close()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
    self.lock.release()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 226, in release
    self.do_release(expected_token)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 229, in do_release
    if not bool(self.lua_release(keys=[self.name],
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/client.py", line 3731, in __call__
    return client.evalsha(self.sha, len(keys), *args)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/client.py", line 2829, in evalsha
    return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/client.py", line 839, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/client.py", line 853, in parse_response
    response = connection.read_response()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/connection.py", line 699, in read_response
    response = self._parser.read_response()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/connection.py", line 309, in read_response
    response = self._buffer.readline()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/connection.py", line 241, in readline
    self._read_from_socket()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/connection.py", line 183, in _read_from_socket
    data = recv(self._sock, socket_read_size)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/_compat.py", line 58, in recv
    return sock.recv(*args, **kwargs)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 154, in _sync
    service.sync()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
    self.scheduler.close()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
    self.lock.release()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 224, in release
    raise LockError("Cannot release an unlocked lock")
redis.exceptions.LockError: Cannot release an unlocked lock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 109, in start_scheduler
    service.start()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 641, in start
    self.sync()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
    self.scheduler.close()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
    self.lock.release()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 224, in release
    raise LockError("Cannot release an unlocked lock")
redis.exceptions.LockError: Cannot release an unlocked lock
[2023-12-20 13:22:44] [INFO] [f:or_routing.py] [l:500] [n:core.or_routing] [completed routing] created_routes:4, unassigned_nodes:0
[2023-12-20 13:22:44] [INFO] [f:tasks.py] [l:103] [n:background.tasks] Notifying WD of route completion for request: wd-f9ddad9f8c5345f182b7-c64bf4f3fb4b
[2023-12-20 13:22:44] [INFO] [f:trace.py] [l:125] [n:celery.app.trace] Task background.tasks.routing_task[eb6fbba7-9c83-42d8-a089-a8da991ae18e] succeeded in 28.762498654425144s: None
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Traceback (most recent call last):
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 635, in start
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] time.sleep(interval)
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 154, in _sync
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] service.sync()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.scheduler.close()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.lock.release()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 226, in release
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.do_release(expected_token)
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 229, in do_release
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] if not bool(self.lua_release(keys=[self.name],
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/client.py", line 3731, in __call__
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return client.evalsha(self.sha, len(keys), *args)
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/client.py", line 2829, in evalsha
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/client.py", line 839, in execute_command
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return self.parse_response(conn, command_name, **options)
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/client.py", line 853, in parse_response
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] response = connection.read_response()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/connection.py", line 699, in read_response
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] response = self._parser.read_response()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/connection.py", line 309, in read_response
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] response = self._buffer.readline()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/connection.py", line 241, in readline
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self._read_from_socket()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/connection.py", line 183, in _read_from_socket
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] data = recv(self._sock, socket_read_size)
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/_compat.py", line 58, in recv
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return sock.recv(*args, **kwargs)
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 154, in _sync
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] service.sync()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.scheduler.close()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.lock.release()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 224, in release
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] raise LockError("Cannot release an unlocked lock")
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] redis.exceptions
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] .
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] LockError
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] :
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Cannot release an unlocked lock
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] During handling of the above exception, another exception occurred:
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Traceback (most recent call last):
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/usr/local/bin/celery", line 8, in <module>
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] sys.exit(main())
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/__main__.py", line 16, in main
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] _main()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 322, in main
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] cmd.execute_from_commandline(argv)
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] super(CeleryCommand, self).execute_from_commandline(argv)))
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return self.handle_argv(self.prog_name, argv[1:])
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 491, in handle_argv
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return self.execute(command, argv)
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/celery.py", line 415, in execute
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return cls(
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 308, in run_from_argv
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return self.handle_argv(prog_name,
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 393, in handle_argv
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return self(*args, **options)
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/base.py", line 253, in __call__
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] ret = self.run(*args, **kwargs)
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bin/beat.py", line 109, in run
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] return beat().run()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 81, in run
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.start_scheduler()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/beat.py", line 109, in start_scheduler
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] service.start()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 641, in start
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.sync()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/beat.py", line 644, in sync
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.scheduler.close()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redbeat/schedulers.py", line 463, in close
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] self.lock.release()
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/redis/lock.py", line 224, in release
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] raise LockError("Cannot release an unlocked lock")
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] redis.exceptions
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] .
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] LockError
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] :
[2023-12-20 13:22:44] [WARNING] [f:log.py] [l:235] [n:celery.redirected] Cannot release an unlocked lock
[2023-12-20 13:22:44] [ERROR] [f:request.py] [l:544] [n:celery.worker.request] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).')
Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/worker.py", line 208, in start
    self.blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/asynchronous/hub.py", line 308, in create_loop
    events = poll(poll_timeout)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/utils/eventio.py", line 84, in poll
    return self._epoll.poll(timeout if timeout is not None else -1)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/worker.py", line 285, in _handle_request
    raise exc(exitcode)
celery.exceptions.WorkerShutdown: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
    raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
[2023-12-20 13:22:44] [ERROR] [f:request.py] [l:544] [n:celery.worker.request] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).')
Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/worker.py", line 208, in start
    self.blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/asynchronous/hub.py", line 308, in create_loop
    events = poll(poll_timeout)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/utils/eventio.py", line 84, in poll
    return self._epoll.poll(timeout if timeout is not None else -1)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/worker.py", line 285, in _handle_request
    raise exc(exitcode)
celery.exceptions.WorkerShutdown: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
    raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
[2023-12-20 13:22:44] [ERROR] [f:request.py] [l:544] [n:celery.worker.request] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).')
Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/worker.py", line 208, in start
    self.blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/asynchronous/hub.py", line 308, in create_loop
    events = poll(poll_timeout)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/utils/eventio.py", line 84, in poll
    return self._epoll.poll(timeout if timeout is not None else -1)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/worker.py", line 285, in _handle_request
    raise exc(exitcode)
celery.exceptions.WorkerShutdown: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
    raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
[2023-12-20 13:22:44] [ERROR] [f:request.py] [l:544] [n:celery.worker.request] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).')
Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/worker.py", line 208, in start
    self.blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/asynchronous/hub.py", line 308, in create_loop
    events = poll(poll_timeout)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/utils/eventio.py", line 84, in poll
    return self._epoll.poll(timeout if timeout is not None else -1)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/worker.py", line 285, in _handle_request
    raise exc(exitcode)
celery.exceptions.WorkerShutdown: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
    raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
[2023-12-20 13:22:44] [ERROR] [f:request.py] [l:544] [n:celery.worker.request] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).')
Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/worker.py", line 208, in start
    self.blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/asynchronous/hub.py", line 308, in create_loop
    events = poll(poll_timeout)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/utils/eventio.py", line 84, in poll
    return self._epoll.poll(timeout if timeout is not None else -1)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/worker.py", line 285, in _handle_request
    raise exc(exitcode)
celery.exceptions.WorkerShutdown: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
    raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
[2023-12-20 13:22:44] [ERROR] [f:request.py] [l:544] [n:celery.worker.request] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).')
Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/worker.py", line 208, in start
    self.blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/asynchronous/hub.py", line 308, in create_loop
    events = poll(poll_timeout)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/kombu/utils/eventio.py", line 84, in poll
    return self._epoll.poll(timeout if timeout is not None else -1)
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/celery/apps/worker.py", line 285, in _handle_request
    raise exc(exitcode)
celery.exceptions.WorkerShutdown: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tkpd/pathfinder-worker/venv/lib/python3.10/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
    raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@poirot-19