Data Classification

Splitting the data

In effect, splitting the data into two subsets can be defined around some filter functions. We’ll avoid Python for a moment and focus on the conceptual math to make sure we have the logic completely correct before diving into code. Conceptually, we have a pair of functions, e($s_i$) and r($s_i$), that decide if a sample, ($s_i$), is for testing, $e$, or training, $r$. These functions are used to partition the samples into two subsets. (If training and testing didn’t both begin with t, we’d have an easier time finding names.) It might help to think about e($s_i$) for evaluation and testing, and r($s_i$) for running a real classification.)

It’s simpler if these two functions are exclusive $e(s_i)$ = $¬r(s_i)$. (We’ll use ¬ instead of the longer not.) If they are proper inverses of each other, this means we only need to define one of the two functions: