Learn about Python's scalability, concurrency, and distribution via CPU scaling, event loops, queue-based distribution, and building REST APIs to optimize and deploy high-performance applications.

python.tar.gz

Chapter2

Chapter7

Chapter5

Chapter6

Chapter3

Chapter7-without-etcd

Chapter8-with-etcd

Chapter9

Chapter9-gabbi

Chapter10-heroku

Chapter10-aws_eb

Chapter11_Tox_integration

Chapter11_Tox_integration-pifpaf

Chapter11_pifpaf_fixtures

Chapter12_caching

Chapter13_Performance

Chapter13_Performace_KCacheGrind

Chapter5-rq-dashboard

Chapter5-celery-flower

Chapter10-gcloud

Chapter2-zsh

Chapter13_Performace_KCacheGrind-copy

Chapter11_Tox_integration-pifpaf-postgresql

Celery-Project1_Sol

Celery-Project1_Sol2

Python, in all of its greatness, is often dismissed when needing to write performant and distributed applications. It’s considered to be slow and not suited for the task. In this course, you will find that with the right implementation of Python, you can write applications that scale horizontally, perform well, and are distributed. 

To kick things off, you’ll learn about CPU scaling, concurrency, and event loops, all of which are crucial for implementing a distributed system. After that, you’ll move on to queue-based distribution, lock management, and group memberships, which are fundamental concepts for maintaining availability and consistency in distributed applications.

In the last half, you’ll learn how to build a REST API, how to deploy your app to a PaaS, and how to optimize your application's performance.

By the end of this course, you’ll be more productive with Python, you’ll be able to write faster applications that are distributed, and you’ll have a great new skill that is highly coveted.

The Hacker's Guide to Scaling Python

_Celery_ is another queue management system. In contrast to RQ, it is broker agnostic and can use various software as a broker, such as Redis, RabbitMQ or Amazon SQS. Moreover, if you are brave enough, it is possible to write your own driver.

# Celery backend
_Celery_ also needs a **backend** for storing the results of the job. It supports a variety of solutions, such as Redis, MongoDB, SQL databases, ElasticSearch, files, etc. Just like for brokers, you can also write your own.

Celery implements its own serialization format for its jobs. However,
this format is not specific to Python. That means it is possible to implement job creators or consumers in different languages. There are already clients in PHP and Javascript.

> **Note:** The example in this lesson assumes that you have a **Redis server**. We already run a Redis server as a daemon when **Run** button is clicked on a widget so you do not need to worry about starting the server yourself.

# Celery task execution: Worker and queues
In Celery, tasks are functions that can be called asynchronously. When called, Celery puts them in the broker queue for execution. Remote workers then execute the tasks, putting the task results into the backend.

When called, a task returns a `celery.result.AsyncResult` object. This object offers a principal method, `get`, which returns the result as soon as it is available, blocking the program in the meantime.
> **Note:** To run the application below, click **Run** and enter `python celery-task.py` in the below application to run the `celery-task.py`.

> If you want to update the source code in coding playgrounds, you need to click**Run** again after changing the code and repeat the instructions to run the application.

Celery

Scaling

CPU Scaling

Event Loops

Functional Programming

Queue-Based Distribution

Designing for Failure

Create a Web Crawler

Project Walkthrough

Lock Management

Group Membership

REST Interfaces

Build a REST API in Flask

Deploying on PaaS

Testing Distributed Systems

Caching

Performance

Conclusion

Celery

Celery backend