activation_function

Gain insights into deep learning constructs like LSTM networks and autoencoders. Delve into modeling, prediction strategies, and TensorFlow implementation for rare event prediction in real-life scenarios.

Understanding Deep Learning Application in Rare Event Prediction.png

DLAA.tar.gz

key1

UDL1

UDL2

UDL1-copy

This course aims to provide a practical understanding of the key constructs of deep learning, including Multi-layer Perceptrons, Long-Short-Term Memory (LSTM) networks, convolutional neural networks, and autoencoders, which focus on rare event prediction.

Through this course, you’ll gain hands-on experience developing solutions for rare event prediction. You’ll start by modeling rare events that occur infrequently. Next, you’ll explore machine and deep learning solutions for imbalanced data. You’ll then learn about rare event “prediction,” its statistical foundations, and prediction strategies. Finally, you’ll implement real-life prediction models using TensorFlow, a crucial framework for building and training models.

By the end of this course, you’ll have a strong grasp of advanced machine and deep learning techniques in rare event prediction. These exercises will not only help you solve complex issues but also become a firm footing for further study and innovation in this field.

Understanding Deep Learning Applications in Rare Event Prediction

# Structure
This section exemplifies the structure of a convolutional network. The illustration below shows an elementary convolutional network. 


The components of the network are as follows:
* **Grid-like input**: Convolutional layers take grid-like inputs. The input in the illustration is like an image, that is, it has two axes and three channels each for $\color{blue}{\text{blue}}$, $\color{red}{\text{red}}$, and $\color{green}{\text{green}}$.

* **Convolutional layer**: A layer comprises filters. A filter is made up of kernels. It has one kernel for each input channel. The size of a layer is essentially the number of filters in it which is a network configuration. Here, five illustrative filters, such as diagonal stripes, horizontal stripes, diamond grid, shingles, and waves, are shown in the convolutional layer. Each of them has $\color{blue}{\text{blue}}$, $\color{red}{\text{red}}$, and $\color{green}{\text{green}}$ channels to match the input.

* **Convolutional output**: A filter sweeps the input to tell the presence/absence of a pattern and its location in the input. The outputs corresponding to each filter are shown with a black square of the same pattern in the figure. It must be noted that the colored channels in the input are absent in the layer’s output. This is because the information across the channels is aggregated during the convolution operation. Consequently, the original channels in the input are relinquished. Instead, the output of each filter becomes a channel for the next layer.

* **Pooling layer**: A convolutional layer is conjoined with a pooling layer. The pooling layer summarizes the spatial features, which are the horizontal and vertical axes in the illustration.

* **Pooling output**: Pooling reduces the sizes of the spatial axes due to a data summarization along the axes. This makes the network invariant to minor translations and robust to noisy inputs. It’s important to note that the pooling occurs only along the spatial axes. Therefore, the number of channels remains intact.

* **Flatten**: The feature map so far is still in a grid-like structure. The flatten operation vectorizes the grid feature map. This is necessary before passing the feature map onto a dense output layer.

* **Dense (output) layer**: Ultimately, a dense layer maps the convolution-derived features with the response.

A convolutional network's purpose is to automatically learn predictive filters from data. Multiple layers are often stacked to learn from low- to high-level features. For instance, a face recognition network could learn the edges of a face in the lower layers and the shape of eyes in the higher layers.

> **Note**: The purpose of a convolutional network is to learn the filters automatically.

## `Conv1D`, `Conv2D`, and `Conv3D`


In Tensorflow, the convolutional layer can be chosen from `Conv1D`, `Conv2D`, and `Conv3D`. The three types of layers are designed for inputs with one, two, or three spatial axes, respectively. Let’s look at when each of them is applicable and their interchangeability.

Convolutional networks work with grid-like inputs. Such inputs are categorized based on their axes and channels. The table below summarizes them for a few grid-like data, such as time series, image, and video.






Axes and Channels in Grid-Like Inputs to Convolutional Networks

align-center

A **univariate time series** has a single spatial axis corresponding to time. If it is **multivariate**, then the features make the channels. Irrespective of the number of channels, a time series is modeled with `Conv1D` as it has only one spatial dimension.


Images, on the other hand, have two spatial axes along their height and width. Videos have an additional spatial axis oxymoronically along time. `Conv2D` and `Conv3D` are, therefore, applicable to them, respectively. The channels in them are the palette colors such as $\color{red}{\text{red}}$, $\color{green}{\text{green}}$, and $\color{blue}{\text{blue}}$.

> **Note**: `Conv1D`, `Conv2D`, and `Conv3D` are used to model inputs with one, two, and three spatial axes, respectively.


The `Conv‘x’D` selection is independent of the channels. There could be any number of channels of arbitrary features. The `Conv‘x’D` is chosen based on the number of spatial axes only. 

Inputs to `Conv1D`, `Conv2D`, and `Conv3D` are structured as N-D tensors of the following shapes, respectively: 
* `(samples, time_steps, features)`

* `(samples, height, width, channels)`

 * `(samples, height, width, channels)` 

The first axis is reserved for samples for almost every layer in TensorFlow.

The shape of a sample is defined by the rest of the axes (shown in the illustrations above and below).

Among them, the last axis corresponds to the channels (by default) in any of the `Conv‘x’D` #concept=Layercon#layers#concept#
and the rest are the spatial axes.


The `kernel_size` argument in `Conv‘x’D` determines the spatial dimension of the convolution kernel. The argument is a tuple of integers. Each element of the tuple corresponds to the kernel’s size along the respective spatial dimension. The depth of the kernel is fixed and equal to the number of channels. The depth is, therefore, not included in the argument.

> **Note**: Conv layers are agnostic to the number of channels. They differ only by the shape of the input’s spatial axes.

Besides, we might observe that a `Conv2D` can be used to model the inputs of `Conv1D` by appropriately reshaping the samples. For example, a time series can be reshaped as `(samples, time, 1, features)`.

Similarly, a `Conv3D` can be used to model the inputs of both `Conv1D` and `Conv2D` by reshaping: 

* A time series as `(samples, 1, 1, time, features)`

* An image as `(samples, height, width, 1, colors)`

Essentially, a universal `Conv3D` layer could be made to work with various inputs due to their interchangeability. The three variants are still provided in TensorFlow for convenience.

Additionally, it is worth learning that a network can be formulated differently by moving the features on the channels to a spatial axis. For example, a multivariate time series can be modeled like an image with a `Conv2D` by reshaping it from `(samples, time, features)` to `(samples, time, features, 1)`. Similarly, the channels of an image can be put on a spatial axis as `(sample, height, width, colors, 1)` and modeled with `Conv3D`.

In short, the input types and the convolutional layer variants are “not” rigidly interlocked. Instead, it is upon the practitioner to formulate the problem as appropriate.

> **Note**: The choice of `Conv1D`, `Conv2D`, and `Conv3D` are not tightly coupled with specific input types. They
can be used interchangeably based on a problem.

# Convolution layer output size

It is essential to understand the size of a convolutional layer output in a different scenario to construct a network. A typical unpadded convolutional layer, for example, downsizes the feature map. Without keeping track of output downsizing, constructing deep networks may result in an ineffective network.

The output size of a filter in a convolutional layer is:

$$o=\bigg\lfloor \frac{i − k − (k − 1)(d − 1) + 2p}{s}\bigg\rfloor+1$$


where

* $\textbf{\textit{i}}$: Size of input’s spatial axes
* $\textbf{\textit{k}}$: Kernel size
* $\textbf{\textit{s}}$: Stride size
* $\textbf{\textit{d}}$: Dilation rate
* $\textbf{\textit{p}}$: Padding size

Each parameter in the equation is a tuple of the same size as the number of spatial axes in the input. For instance, in the convolution example between $G$ and $F_{s, a}$ in the previous lesson:

* $\textbf{\textit{i}}\space \space \space =$ $(5, 5)$, size of the spatial axes of input $G$
* $\textbf{\textit{k}}\space \space \space =$ $(3, 5)$, size of the kernel $F_·$
* $\textbf{\textit{s}}\space \space \space =$ $(1, 1)$, the default single stride
* $\textbf{\textit{d}}\space \space \space =$ $(1, 1)$, the default no dilation
* $\textbf{\textit{p}}\space \space \space =$ $(0, 0)$, the default no padding

The output size is computed as:

$$o =\bigg\lfloor \frac{(5, 5) − (3, 5) − ((3, 5) − 1)((1, 1) − 1) + 2*(0, 0)}{(1, 1)}\bigg\rfloor+1=(3, 1).$$

Furthermore, a convolutional layer with $l$ filters has an $(o, l)$ tensor as the output, where $l$ corresponds to the channels. Extending the example in the previous lesson, suppose a convolutional layer has the semicircle, $F_s$, and angle, $F_a$, filters. The output will then be a $(3, 1, 2)$ tensor where the last tuple element $2$ corresponds to channels—one from each filter.

It is worth noting that the original channels in the input are not included in the output size computation. As mentioned previously, the convolution operation aggregates the channels due to which the original channels are lost in the output. This also implies that a network construction by tracking the output downsizing is unrelated to the input channels.

# Pooling layer output size
Similar parameters govern the output size of the pooling layer as the convolutional. Analogous to the kernel size, a pooling layer has a pool size. Strides and padding are also choices in pooling. Dilation, however, is not available in pooling.

Based on these, the pooling output size for a channel is:

$$o=\bigg\lfloor \frac{i − k\ + 2p}{s}\bigg\rfloor+1$$

where,

* $\textbf{\textit{i}}$ : Size of input’s spatial axes
* $\textbf{\textit{k}}$ : Pool size
* $\textbf{\textit{s}}$ : Stride size
* $\textbf{\textit{p}}$ : Padding size

If there are $l$ channels, the pooling output is an $(o,l)$ tensor. Essentially, the pooling happens independently for every channel, that is, the values in the channels are not merged.

Moreover, note that “same” padding does not result in downsampling. Still, the pooling operation brings the invariance attribute to the network and is, therefore, useful.

# Parameters

The number of parameters in a Conv layer can be simply expressed as:
$$((\text{filter volume}) + 1) * (\text{number of filters})$$

A filter has weight parameters equal to its volume plus a bias parameter. The volume of a filter is the product of the spatial axes $(\text{kernel size}, \textbf{\textit{k}})$ and the depth (the number of channels, $c$, in the input). For example, in the previous lesson, the kernel shape $\textbf{\textit{k}}$ is $(3, 5)$, and the number of channels $c$ for the grayscale input is $1$. Therefore, the filter volume is $3 \times 5 \times 1 = 15$. If the input was in RGB, that is, $c = 3$, the filter volume becomes $3 \times 5 \times 3 = 45$.

The number of filters is equal to the parameter filters set in the definition of a `Conv` layer.

Unlike the `Conv` layer, most pooling layers, such as `MaxPool` and `AveragePooling` in TensorFlow do not have any trainable parameters.













 





Explore the structure of convolutional networks, from grid-like inputs to the dense layer output.

Getting Started

Rare Event Prediction

Multi-Layer Perceptrons (MLPs)

Long Short-Term Memory (LSTM) Networks

Convolutional Neural Networks (CNNs)

Autoencoders

Conclusion

Architecture of Convolutional Networks

Structure