Matrix Math

Overview of basic matrix operations

To upgrade our system to multiple variables, we’ll need to know about two operations on matrices. We do not want to introduce those operations at the last moment, while we are busy coding. Instead, we’ll take a few moments to understand them now.

A matrix is a bi-dimensional array. For example, a (4,3) matrix. It means that it has four rows and three columns:

1. We’ll cover two operations in this chapter: Matrix multiplication
2. Matrix transpose

Both are ubiquitous in MLMachine Learning, and each deserves its own section.

Multiply matrices

Did we ever wonder why ML is usually done on those big racks of GPUs? That’s because ML systems spend most of their time doing one operation that happens to be particularly fast on a GPU, and they multiply big matrices.

To introduce matrix multiplication, let’s describe the golden rule first: we can multiply two matrices if (and only if) the second matrix has as many rows as the first matrix has columns, like this:

$M1$ is (4, 3) and $M2$ is (3, 2). Can we multiply them? To answer this question, write down the operation like this:

Look at the inner dimensions in the multiplication. If they are equal, as in this case, then the multiplication is valid, and the result will be a matrix that has the outer dimensions,in this case, (4, 2):

Let’s look at concrete cases, starting with a simple one. A matrix with a single row, multiplied by a matrix with a single column, like this:

First, let’s check whether this multiplication is a valid one. The first matrix is (1, 3), and the second is (3, 1). By the golden rule, the multiplication (1, 3) ⋅ (3, 1) is legit, and it will return a (1, 1) matrix. It’s a matrix that contains only one element.

To calculate that one element, we multiply each element of $M1$ with the corresponding element of $M2$. The first with the first, the second with the second, and so on. Then we add them all together:

2 * 2.5 + 3 * 4 + 5 * 1 = 22

So, the result of the ...