Gain insights into Python's scalability, concurrency, and distribution. Learn about CPU scaling, event loops, queue-based distribution, and building REST APIs to optimize and deploy high-performance applications.

python.tar.gz

Chapter2

Chapter7

Chapter5

Chapter6

Chapter3

Chapter7-without-etcd

Chapter8-with-etcd

Chapter9

Chapter9-gabbi

Chapter10-heroku

Chapter10-aws_eb

Chapter11_Tox_integration

Chapter11_Tox_integration-pifpaf

Chapter11_pifpaf_fixtures

Chapter12_caching

Chapter13_Performance

Chapter13_Performace_KCacheGrind

Chapter5-rq-dashboard

Chapter5-celery-flower

Chapter10-gcloud

Chapter2-zsh

Chapter13_Performace_KCacheGrind-copy

Chapter11_Tox_integration-pifpaf-postgresql

Celery-Project1_Sol

Celery-Project1_Sol2

Python, in all of its greatness, is often dismissed when needing to write performant and distributed applications. It’s considered to be slow and not suited for the task. In this course, you will find that with the right implementation of Python, you can write applications that scale horizontally, perform well, and are distributed. 

To kick things off, you’ll learn about CPU scaling, concurrency, and event loops, all of which are crucial for implementing a distributed system. After that, you’ll move on to queue-based distribution, lock management, and group memberships, which are fundamental concepts for maintaining availability and consistency in distributed applications.

In the last half, you’ll learn how to build a REST API, how to deploy your app to a PaaS, and how to optimize your application's performance.

By the end of this course, you’ll be more productive with Python, you’ll be able to write faster applications that are distributed, and you’ll have a great new skill that is highly coveted.

The Hacker's Guide to Scaling Python

As you might already know, [tox](https://tox.readthedocs.io/) is a tool that automates building virtual Python environments. Those environments are isolated and straightforward to rebuild while having support for multiple Python versions at the same time in different environments.

Most of the time, _tox_ is used to directly run unit tests of a program, with a file as
simple as:

Such a file makes sure that the tests are run in a clean Python environment with
the right dependencies installed, as specified by your packaging tools.

Since it is possible to run any command by setting the `commands` parameter, it is easy to envision a system where you could start other services before running your tests. Therefore, a simplistic technique to run integration tests with `memcached`, for
example, would be:

[testenv]
commands=memcached -p 12345 &
        nosetests

The application would then have to set itself up to use _memcached_ on TCP port
12345, and the tests could run using that instance of _memcached_.

Unfortunately, this approach is overly naive: first, there is no way to know when
_memcached_ is ready to accept connections. That could make the first tests fail for no good reason. This is exactly the kind of false negative result that you want to avoid
at all costs, as they undermine the usefulness of the test suite.

Moreover, after the tests are done, _memcached_ would still be running. The second run of _tox_ will spawn a new _memcached_ instance, which would fail to start because the TCP port is already being used. Also, the second test run would connect to the previous _memcached_ daemon, possibly using old entry and keys stored in it, again yielding potential false-negatives.

This short example demonstrates that unfortunately, the orchestration of external dependency for integration testing is much more complicated than one might hope. It does not mean that it is impossible to orchestrate a whole deployment of resources before running your tests, but to do that, you need to take care of the following
before running the tests:

1. Prepare any directory or resources needed by external services.
2. Launch those external services.
3. Make sure those external services are ready to be used.

Then, after the tests are run you need to:
1. Shut down the external service (correctly, if possible).
2. Clean everything the service might have left behind (e.g. temporary data).

Taking care of this is not trivial, but once done well, it makes sure that you can comfortably run _tox_ and run integration testing once and for all with a simple
command: `tox -e py38-integration`.


Learn how to set up a testing environment with Tox using Python.

Scaling

CPU Scaling

Event Loops

Functional Programming

Queue-Based Distribution

Designing for Failure

Create a Web Crawler

Project Walkthrough

Lock Management

Group Membership

REST Interfaces

Deploying on PaaS

Testing Distributed Systems

Caching

Performance

Conclusion

Setting Up Environments with tox