Using celery with flask application factory pattern

[capymind]

Hello, everyone.
First of all, thanks a lot for maintaining this beautiful and invaluable open source project🎉

I know that there are already many "tutorial" explaining using celery in flask (even in documentation).
BUT, almost(if not all) just start with app=Flask(__name__) , which in production nobody use directly initialization of Flask instance.

So... what I want to achieve is to interagte flask application factory with celery instance including flask context.

The official flask docs say the below code

from celery import Celery

def make_celery(app):
    celery = Celery(app.import_name)
    celery.conf.update(app.config["CELERY_CONFIG"])

    class ContextTask(celery.Task):
        def __call__(self, *args, **kwargs):
            with app.app_context():
                return self.run(*args, **kwargs)

    celery.Task = ContextTask
    return celery

That's great!!. But the point I want to mention is below

from flask import Flask

flask_app = Flask(__name__)  # <=🔴😢  I want to use a kind of application factory pattern the docs say
flask_app.config.update(CELERY_CONFIG={
    'broker_url': 'redis://localhost:6379',
    'result_backend': 'redis://localhost:6379',
})
celery = make_celery(flask_app)

@celery.task()
def add_together(a, b):
    return a + b

If flask app is created like

from flask import Flask
def create_app():
    app = Flask()
    return app

How to integrate the celery app with flask app with correct way?

I know that flask and celery is launched decades ago,
I guess that ALREADY there are solutions(or best practices) about that in online.
But I cannot find solution (maybe my tech skills and searching ability is below-average... sorry about that)

Thanks a lot
Happy new year in advance. 🙏 🎉
Best regards.

[jessielw]

I've spent some time messing around with this over the course of the last couple weeks on and off. What I have came up with, by looking at documentation and looking around online in something like this.

├── example_app
│   ├── static
│   │   ├── static stuff
│   ├── tasks
│   │   ├── random_task.py
│   ├── templates
│   ├── __init__.py   
│   ├── utils.py  
├── stuff here if you need it

(Not very good at making a tree, sorry)

utils.py

from celery import Celery


def make_celery(app):
    celery = Celery(app.import_name)
    celery.conf.update(app.config["CELERY_CONFIG"])

    # add beat tasks if you have them here
    # celery.config_from_object(celery_config)

    class ContextTask(celery.Task):
        def __call__(self, *args, **kwargs):
            with app.app_context():
                return self.run(*args, **kwargs)

    celery.Task = ContextTask
    return celery

random_task.py

from celery import shared_task
from time import sleep

@shared_task()
def long_task():
    for x in range(5):
        sleep(1)
        print(x)
    return "done"

init.py

from celery import Celery
from flask import Flask

celery_app = Celery()


def create_app():
    app = Flask(__name__)

    # import all utils
    from example_app.utils import make_celery

    # celery
    celery_app.conf.update(app.config["CELERY_CONFIG"])

    # import all task modules before init app
    from example_app.tasks import random_task

    celery = make_celery(app)
    celery.set_default()

    # register blueprints
    from example_app.home.routes import home
    from example_app.users.routes import users

    app.register_blueprint(home)
    app.register_blueprint(users)

    return app, celery


app, celery = create_app()

Good luck!

[jessielw]

When you start your celery worker be sure to call it with celery -A example_app.celery worker

[capymind]

Really appreciate it, I will give a shot Thanks a lot!!

[jessielw]

Really appreciate it, I will give a shot Thanks a lot!!

No problem man

[davidism]

Flask's official documentation on Celery was rewritten recently. It shows how to use the app factory with both Flask and Celery. https://flask.palletsprojects.com/en/2.2.x/patterns/celery/

[capymind]

Oh..! Really appreciate it. It is helpful a lot. Thanks!