Gain insights into Python's scalability, concurrency, and distribution. Learn about CPU scaling, event loops, queue-based distribution, and building REST APIs to optimize and deploy high-performance applications.

python.tar.gz

Chapter2

Chapter7

Chapter5

Chapter6

Chapter3

Chapter7-without-etcd

Chapter8-with-etcd

Chapter9

Chapter9-gabbi

Chapter10-heroku

Chapter10-aws_eb

Chapter11_Tox_integration

Chapter11_Tox_integration-pifpaf

Chapter11_pifpaf_fixtures

Chapter12_caching

Chapter13_Performance

Chapter13_Performace_KCacheGrind

Chapter5-rq-dashboard

Chapter5-celery-flower

Chapter10-gcloud

Chapter2-zsh

Chapter13_Performace_KCacheGrind-copy

Chapter11_Tox_integration-pifpaf-postgresql

Celery-Project1_Sol

Celery-Project1_Sol2

Python, in all of its greatness, is often dismissed when needing to write performant and distributed applications. It’s considered to be slow and not suited for the task. In this course, you will find that with the right implementation of Python, you can write applications that scale horizontally, perform well, and are distributed. 

To kick things off, you’ll learn about CPU scaling, concurrency, and event loops, all of which are crucial for implementing a distributed system. After that, you’ll move on to queue-based distribution, lock management, and group memberships, which are fundamental concepts for maintaining availability and consistency in distributed applications.

In the last half, you’ll learn how to build a REST API, how to deploy your app to a PaaS, and how to optimize your application's performance.

By the end of this course, you’ll be more productive with Python, you’ll be able to write faster applications that are distributed, and you’ll have a great new skill that is highly coveted.

The Hacker's Guide to Scaling Python

# Background

Python 3.2 introduced the `concurrent.futures` module, which provides an easy way to schedule asynchronous tasks. 

The module is also available in Python 2 as it has been back-ported. It can easily be installed by running `pip install futures`.

The `concurrent.futures` module is pretty straightforward to use. First, one
needs to pick an executor. An executor is responsible for scheduling and running
asynchronous tasks. It can be seen as a type of engine for execution. The module
currently provides two kinds of executors: `concurrent.futures.ThreadPoolExecutor`
and `concurrent.futures.ProcessPoolExecutor`. As one might guess, the first one is
based on threads and the second one on processes.

As outlined in the first chapter, the process-based executor is going to be much more efficient for long-running tasks that benefit from having an entire CPU available. The threading executor suffers from the same limitation as the `threading` module, which was covered earlier.



# `concurrent.futures.Future` abstraction

So what is interesting about the `concurrent.futures` module is that it provides an easier-to-use abstraction layer on top of the `threading` and `multiprocessing` modules. It allows one to run and parallelize code in a straightforward way, providing an abstract data structure called a `concurrent.futures.Future` object.

Each time a program schedules some tasks to execute in threads or processes, the
`concurrent.futures` module returns a `Future` object for each of the tasks scheduled. This `Future` object owns the promise of the work to be completed. Once that work is achieved, the result is available in that `Future` object. So in the end, it does represent the future and the promise of a task to be performed. That is why it is called "Future" in Python, and sometimes "promise" in other languages.

> To run the following code snippet, press the Run button and wait until all the commands finish running. Then, enter the command `time python futures-threads-worker.py`.

Using Futures

Scaling

CPU Scaling

Event Loops

Functional Programming

Queue-Based Distribution

Designing for Failure

Create a Web Crawler

Project Walkthrough

Lock Management

Group Membership

REST Interfaces

Deploying on PaaS

Testing Distributed Systems

Caching

Performance

Conclusion

Using Futures

Background