What is the mean average precision in information retrieval?

We use the mean average precision (mAP) to measure the accuracy of information retrieval models.

The mAP is a value between $0–1$, with higher scores representing a more accurate model.

We describe it by the following formula:

In the above formula, $N$ is the total number of queries, and$AP_i$ is the average precision of query_i. In simple terms, mAP is the average of average precisions across all queries.

To understand the calculation of mAP, we must first explore precision, precision at $k$, and average precision.

Precision

Precision is the ratio of correctly-identified positive valuesPredicted values that match the actual values to the total number of predicted positive values.

We must slightly modify the formula for precision to use it with an information retrieval model, as follows:

The $relevant\ documents$ set refers to the ground truth values. In other words, these are the expected documents that should be received for the given query.

The $retrieved\ documents$ set includes the documents selected by the information model based on similarity scoresScores that represent how relevant the retrieved document is to our query. This set includes all relevant and non-relevant documents that the model picks. Hence, it is equal to $TP + FP$ from the usual precision formula.

The intersection size of $relevant\ documents$ and $retrieved\ documents$ can be considered the true positive count ($TP$).

Precision at $k$

Usually, precision takes all the retrieved documents into account. However, another way to calculate precision involves a cut-off rankAn integer that determines the number of documents we want to take into account, $k$. Through this method, we calculate precision only for the top $k$ documents. This metric is called precision at $k$, or $P(k)$.

To understand $P(k)$, let's consider an information retrieval model that accepts a query and returns documents that are similar to that query.

Average precision (AP)

Average precision is the area under the precision-recall curve. For information retrieval models, where recall is a less critical metric, we can calculate AP using the following formula:

Where $RD$ is the number of relevant documents for the query, $n$ is the total number of documents, $P(k)$ is the precision at $k$, and $r(k)$ is the relevance of the $k$^th retrieved document (0 if not relevant, and 1 if relevant).

Let's continue with the previous example. We can calculate the AP of the information retrieval model as follows:

Let's consider another model that assigns the highest scores to the relevant documents.

As we can see, more accurate models—those that assign higher scores to the relevant documents—have greater average precision.

Mean average precision (mAP)

We repeat the calculation for AP for each query. The mAP across this set of queries is the average of the calculated APs.