Perceptrons: A Brain in a Function

The simplest neural network architecture—the perceptron was inspired by biological research to understand the basis of mental processing in an attempt to represent the function of the brain with mathematical formulae.

From tissues to TLUs

The recent popularity of AI algorithms might give the false impression that this field is new. Many recent models are based on discoveries made decades ago that have been reinvigorated by the massive computational resources available in the cloud and customized hardware for parallel matrix computations, such as Graphical Processing Units (GPUs), Tensor Processing Units (TPUs), and Field-Programmable Gate Array (FPGAs).

If we consider research on neural networks to include their biological inspiration as well as computational theory, this field is over a hundred years old. Indeed, one of the first neural networks described appears in the detailed anatomical illustrations of 19th-century scientist Santiago Ramón y Cajal.López-Muñoz F., Boya J., Alamo C. (2006). Neuron theory, the cornerstone of neuroscience, on the centenary of the Nobel Prize award to Santiago Ramón y Cajal. Brain Research Bulletin. 70 (4–6): 391–405. https://pubmed.ncbi.nlm.nih. gov/17027775/ illustrations, based on experimental observations of layers of interconnected neuronal cells, inspired the Neuron Doctrine—the idea that the brain is composed of individual, physically distinct, and specialized cells rather than a single continuous networkRamón y Cajal, Santiago (1888). Estructura de los centros nerviosos de las aves. . The distinct layers of the retina observed by Cajal were also the inspiration for particular neural network architectures such as CNN, which we’ll discuss later.

This observation of simple neuronal cells interconnected in large networks led computational researchers to hypothesize how mental activity might be represented by simple, logical operations that, combined, yield complex mental phenomena. The original “automata theory” is usually traced to a 1943 article by Warren McCulloch and Walter Pitts of the Massachusetts Institute of TechnologyMcCulloch, W.S., Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5, 115–133. https://doi. org/10.1007/BF02478259. They described a simple model known as the Threshold Logic Unit (TLU), in which binary inputs are translated into a binary output based on a threshold:

where $I$ is the input values, $W$ is the weights with ranges from $(0, 1)$ or $(-1, 1)$, and $f$ is a threshold function that converts these inputs into a binary output depending upon whether they exceed a thresholdRashwan M., Ez R., reheem G. (2017). Computational Intelligent Algorithms For Arabic Speech Recognition. Journal of Al-Azhar University Engineering Sector. 12. 886-893. 10.21608/auej.2017.19198. http://wwwold.ece.utep.edu/ research/webfuzzy/docs/kk-thesis/kk-thesis-html/node12.html $T$:

Visually and conceptually, there is some similarity between McCulloch and Pitts’ model and the biological neuron that inspired it. Their model integrates inputs into an output signal, just as the natural dendrites (short, input “arms” of the neuron that receive signals from other cells) of a neuron synthesize inputs into a single output via the axon (the long “tail” of the cell, which passes signals received from the dendrites along to other neurons). We might imagine that, just as neuronal cells are composed into networks to yield complex biological circuits, these simple units might be connected to simulate sophisticated decision processes.

Indeed, using this simple model, we can already start to represent several logical operations. If we consider a simple case of a neuron with one input, we can see that a TLU can solve an identity or negation function.

For an identity operation that simply returns the input as output, the weight matrix would have $1$s on the diagonal (or be simply the scalar $1$), for a single numerical input:

Similarly, for a negation operation, the weight matrix could be a negative identity matrix, with a threshold at $0$ ...