Introduction

We’ve discussed how many of Python’s built-ins and idioms seem, at first blush, to fly in the face of object-oriented principles, but are actually providing access to real objects under the hood. In this chapter, we’ll discuss how the for loop, which seems so structured, is actually a lightweight wrapper around a set of object-oriented principles. We’ll also see a variety of extensions to this syntax that automatically create even more types of object. We will cover the following topics:

• What design patterns are
• The iterator protocol—one of the most powerful design patterns
• List, set, and dictionary comprehensions
• Generator functions, and how they build on other patterns

The case study for this chapter will revisit the algorithms for partitioning sample data into testing and training subsets to see how the Iterator design pattern applies to this part of the problem. We’ll start with an overview of what design patterns are and why they’re so important.

Design patterns in brief

When engineers and architects decide to build a bridge, or a tower, or a building, they follow certain principles to ensure structural integrity. There are various possible designs for bridges (suspension and cantilever, for example), but if the engineer doesn’t use one of the standard designs, and doesn’t have a brilliant new design, it is likely the bridge they design will collapse.

Why do we need design patterns?

Design patterns are an attempt to bring this same formal definition for correctly designed structures to software engineering. There are many different design patterns to solve different general problems. Design patterns are applied to solve a common problem faced by developers in some specific situation. The design pattern is a suggestion as to the ideal solution for that problem, in terms of object-oriented design. What’s central to a pattern is that it is reused often in unique contexts. One clever solution is a good idea. Two similar solutions might be a coincidence. Three or more reuses of an idea and it starts to look like a repeating pattern.

The challenge of proper implementation

Knowing design patterns and choosing to use them in our software does not, however, guarantee that we are creating a correct solution. In 1907, the Quebec Bridge (to this day, the longest cantilever bridge in the world, just short of a kilometer long) collapsed before construction was completed, because the engineers who designed it grossly underestimated the weight of the steel used to construct it. Similarly, in software development, we may incorrectly choose or apply a design pattern, and create software that collapses under normal operating situations or when stressed beyond its original design limits.

Any one design pattern proposes a set of objects interacting in a specific way to solve a general problem. The job of the programmer is to recognize when they are facing a specific version of such a problem, then to choose and adapt the general pattern to their precise needs.

In this chapter, we’ll look deeply at the Iterator design pattern. This pattern is so powerful and pervasive that the Python developers have provided multiple syntaxes to access the object-oriented principles underlying the pattern. We will be covering other design patterns in the next two chapters. Some of them have language support and some don’t, but none of them are so intrinsically a part of the Python coder’s daily life as the Iterator pattern.