Flask-CeleryExt¶
About¶
Flask-CeleryExt is a simple integration layer between Celery and Flask.
Installation¶
Flask-CeleryExt is on PyPI so all you need is:
pip install flask-celeryext
Usage¶
Simple integration layer between Celery and Flask.
This extension adds a small integration layer between Celery and Flask based on https://flask.pocoo.org/docs/0.10/patterns/celery/. In addition to support execution of tasks in a Flask application context, the extension also supports execution of task in a test request context (e.g. to ensure that before first request functions have been executed).
Extension initialization¶
Initialize the extension like this:
>>> from flask import Flask
>>> from flask_celeryext import FlaskCeleryExt
>>> app = Flask('testapp')
>>> ext = FlaskCeleryExt(app)
or alternatively using the factory pattern:
>>> app = Flask('testapp')
>>> app.config.update(dict(
... CELERY_ALWAYS_EAGER=True,
... CELERY_RESULT_BACKEND='cache',
... CELERY_CACHE_BACKEND='memory',
... CELERY_EAGER_PROPAGATES=True))
>>> ext = FlaskCeleryExt()
>>> ext.init_app(app)
The extension will create a Celery application. Configuration is by default loaded from the Flask application:
>>> celery = ext.celery
Defining tasks¶
You can now define tasks using the Celery application. By default the tasks
will be run inside a Flask application context, and thus have access to e.g.
current_app:
>>> from flask import current_app
>>> @celery.task
... def apptask():
... return current_app.name
>>> r = apptask.delay()
>>> r.result
'testapp'
If you need to run tasks inside a Flask request context, simply change the task base class:
>>> from flask import has_request_context, request
>>> from flask_celeryext import RequestContextTask
>>> @celery.task(base=RequestContextTask)
... def reqtask():
... return has_request_context()
>>> r = reqtask.delay()
>>> r.result
True
Application factory¶
The Celery application is created by a default application factory, which you can also use separately:
>>> from flask_celeryext import create_celery_app
>>> app = Flask('testapp')
>>> celery = create_celery_app(app)
>>> @celery.task
... def appctx():
... return 'test'
It’s also possible to provide a custom Celery application factory to the extension:
>>> from celery import Celery
>>> from flask_celeryext import AppContextTask
>>> def make_celery(app):
... celery = Celery(app.import_name)
... celery.flask_app = app
... celery.Task = AppContextTask
... return celery
>>> app = Flask('testapp')
>>> ext = FlaskCeleryExt(app, create_celery_app=make_celery)
Larger applications¶
In a front-end Flask application you will usually only need a minimal Celery application configured in order for Celery to know which broker to use etc.
Note
There is a difference with the Celery tutorial in Flask documentation.
One should use BROKER_URL configuration option instead of
CELERY_BROKER_URL.
This minimal application however does not need to load all tasks upfront, as especially for larger applications loading many tasks can cause startup time to increase significantly. The background worker on the other hand usually needs to load tasks upfront in order to know which tasks it can handle.
Testing¶
Testing your celery tasks is rather easy. First ensure that you Celery is configured to execute tasks eagerly and stores results in local memory:
>>> app = Flask('testapp')
>>> app.config.update(dict(
... CELERY_ALWAYS_EAGER=True,
... CELERY_RESULT_BACKEND='cache',
... CELERY_CACHE_BACKEND='memory',
... CELERY_EAGER_PROPAGATES=True))
>>> celery = create_celery_app(app)
You can now create your task:
>>> from celery import current_task
>>> @celery.task(name='testapp.test_name')
... def test():
... return current_task.name
And finally execute your task:
>>> r = test.delay()
>>> r.result
'testapp.test_name'
Changes¶
Version 0.4.0 (released 2022-03-15)
Adds support for Python 3.9.
Adds support for new Celery configuration variable naming.
Drops support for Python 2.7 and 3.6 due to end of life.
Drops support for Celery 3 and 4.
Version 0.3.4 (released 2020-02-17)
Adds support for Python 3.8
Fixes pin for Celery on Python <3.7.
Version 0.3.3 (released 2020-02-13)
Fix celery version for Python < 3.7
Version 0.3.2 (released 2019-06-25)
Uses correct Celery version for Python 3.7.
Prevents multiple creation and pushing of Flask application contexts.
Version 0.3.1 (released 2018-03-26)
Accounts for non-strict Celery versions.
Version 0.3.0 (released 2017-03-24)
Adds support for Celery v4.
Version 0.2.2 (released 2016-11-07)
Forces celery version to v3.1-4.0 due to problem with 4.x.
Version 0.2.1 (released 2016-07-25)
Improved features
Improves documentation structure and its automatic generation.
Version 0.2.0 (released 2016-02-02)
Incompatible changes
Changes celery application creation to use the default current celery application instead creating a new celery application. This addresses an issue with tasks using the shared_task decorator and having Flask-CeleryExt initialized multiple times.
Version 0.1.0 (released 2015-08-17)
Initial public release
Contributing¶
Bug reports, feature requests, and other contributions are welcome. If you find a demonstrable problem that is caused by the code of this library, please:
Search for already reported problems.
Check if the issue has been fixed or is still reproducible on the latest master branch.
Create an issue with a test case.
If you create a feature branch, you can run the tests to ensure everything is operating correctly:
$ ./run-tests.sh
You can also test your feature branch using Docker:
$ docker-compose build
$ docker-compose run --rm web /code/run-tests.sh
License¶
Flask-CeleryExt is free software; you can redistribute it and/or modify it under the terms of the Revised BSD License quoted below.
Copyright (C) 2015 CERN.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.
