Logits

Dive into the inner layers of a neural network and understand the importance of logits.

Chapter Goals:

  • Build a single fully-connected model layer
  • Output logits, AKA log-odds

A. Fully-connected layer

Before we can get into multilayer perceptrons, we need to start off with a single layer perceptron. The single fully-connected layer means that the input layer, i.e. self.inputs, is directly connected to the output layer, which has output_size neurons. Each of the input_size neurons in the input layer has a connection to each neuron in the output layer, hence the fully-connected layer.

In TensorFlow, this type of fully-connected neuron layer is implemented using tf.keras.layers.Dense, which takes in a neuron layer and output size as required arguments, and adds a fully-connected output layer with the given size to the computation graph.

In addition to the input layer neurons, tf.keras.layers.Dense adds another neuron called the bias, which always has a value of 1 and has full connections to the output layer.

The bias neuron helps our neural network produce better results, by allowing each fully-connected layer to model a true linear combination of the input values.

B. Weighted connections

The forces that drive a neural network are the real number weights attached to each connection. The weight on a connection from neuron A into neuron B tells how strongly A affects B as well as whether that effect is positive or negative, i.e. direct vs. inverse relationship.

The diagram above has three weighted connections:

A → B: Direct relationship.
A → C: No ...