Deep Neural Network Development and Tensorflow

First, let’s start by diving more into the details of TensorFlow, the library we have used to develop models throughout the rest of this course. What problem does TensorFlow solve for neural network model development? What approaches does it use? How has it evolved over the years? To answer these questions, let’s review some of the history behind deep neural network libraries that led to the development of TensorFlow.

A deep neural network, in essence, consists of matrix operations (addition, subtraction, multiplication), nonlinear transformations, and gradient-based updates computed by using the derivatives of these components.

In the world of academia, researchers have historically often used efficient prototyping tools such as MATLABMATLAB, Natick, Massachusetts: The MathWorks Inc. https://www. mathworks.com/products/matlab.html to run models and prepare analyses. While this approach allows for rapid experimentation, it lacks elements of industrial software development, such as object-oriented (OO) development, that allow for reproducibility and clean software abstractions that allow tools to be adopted by large organizations.

These tools also had difficulty scaling to large datasets and could carry heavy licensing fees for such industrial use cases. However, prior to 2006, this type of computational tooling was largely sufficient for most use cases. However, as the datasets being tackled with deep neural network algorithms grew, groundbreaking results were achieved, such as:

• Image classification on the ImageNet datasetKrizhevsky A., Sutskever I., & Hinton G E. ImageNet Classification with Deep Convolutional Neural Networks. https://papers.nips.cc/paper/4824- imagenet-classification-with-deepconvolutional-neural-networks.pdf.

• Large-scale unsupervised discovery of image patterns in YouTube videosDean J., Ng A., (2012, Jun 26). Using large-scale brain simulations for machine learning and A.I.. Google | The Keyword. https://blog.google/technology/ ai/using-large-scale-brain-simulations-for/.

• The creation of artificial agents capable of playing Atari video gamesMnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M. (2013). Playing Atari with Deep Reinforcement Learning. arXiv:1312.5602. https://arxiv.org/abs/1312.5602 and the Asian board game GO with human-like skillSilver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A, Chen Y, Lillicrap T, Hui F, Sifre L, van den Driessche G, Graepel T, Hassabis D. (2017) Mastering the game of Go without human knowledge. Nature. 550(7676):354-359. https://pubmed.ncbi. nlm.nih.gov/29052630/.

• State-of-the-art language translation via the BERT model developed by GoogleDevlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. https://arxiv.org/abs/1810.04805.

The models developed in these studies exploded in complexity, along with the size of the datasets they were applied to (see the table below to get a sense of the immense scale of some of these models). As industrial use cases required robust and scalable frameworks to develop and deploy new neural networks, several academic groups and large technology companies invested in the development of generic toolkits for the implementation of deep learning models. These software libraries codified common patterns into reusable abstractions, allowing even complex models to be often embodied in relatively simple experimental scripts.