Applying Complex Row-by-Row Operations

Often in a data science context, we’ll read some data and then need to add new data columns based on the existing data. For instance, we’re creating an additional column to classify fish as healthy, underweight, or overweight based on a formula that uses weight and length data already provided in the input data. Or if we’re working with grade data for students, maybe we need to add a column for maximum and minimum grades by student.

However, most tidyverse functions, like cor or mean, are intended to aggregate across rows of data; they’re column-wise aggregations. And that idea is consistent with the fact that we’re working with tidy data, so most of our aggregations will be column-wise, across our rows (observations) and not our columns (variables).

When working with tidy data, row-wise aggregations are most common in the data cleaning stages rather than the actual analytics. The functions of tidyverse are designed to work best when rows represent observations and columns represent variables. So, we’ll typically perform row-wise aggregations to create columns (a.k.a. variables) representing latent measurements—things we indirectly observed based on directly measuring other variables.