Logits
Dive into the inner layers of a neural network and understand the importance of logits.
We'll cover the following...
Chapter Goals:
- Build a single fully-connected model layer
- Output logits, AKA log-odds
A. Fully-connected layer
Before we can get into multilayer perceptrons, we need to start off with a single layer perceptron. The single fully-connected layer means that the input layer, i.e. self.inputs
, is directly connected to the output layer, which has output_size
neurons. Each of the input_size
neurons in the input layer has a connection to each neuron in the output layer, hence the fully-connected layer.
In TensorFlow, this type of fully-connected neuron layer is implemented using tf.keras.layers.Dense
, which takes in a neuron layer and output size as required arguments, and adds a fully-connected output layer with the given size to the computation graph.
In addition to the input layer neurons, tf.keras.layers.Dense
adds another neuron called the bias, which always has a value of 1 and has full connections to the output layer.
The bias neuron helps our neural network produce better results, by allowing each fully-connected layer to model a true linear combination of the input values.
B. Weighted connections
The forces that drive a neural network are the real number weights attached to each connection. The weight on a connection from neuron A into neuron B tells how strongly A affects B as well as whether that effect is positive or negative, i.e. direct vs. inverse relationship.
The diagram above has three weighted connections:
A → B: Direct relationship.A → C: No ...