activation_function

Gain insights into deep learning constructs like LSTM networks and autoencoders. Delve into modeling, prediction strategies, and TensorFlow implementation for rare event prediction in real-life scenarios.

Understanding Deep Learning Application in Rare Event Prediction.png

DLAA.tar.gz

key1

UDL1

UDL2

UDL1-copy

This course aims to provide a practical understanding of the key constructs of deep learning, including Multi-layer Perceptrons, Long-Short-Term Memory (LSTM) networks, convolutional neural networks, and autoencoders, which focus on rare event prediction.

Through this course, you’ll gain hands-on experience developing solutions for rare event prediction. You’ll start by modeling rare events that occur infrequently. Next, you’ll explore machine and deep learning solutions for imbalanced data. You’ll then learn about rare event “prediction,” its statistical foundations, and prediction strategies. Finally, you’ll implement real-life prediction models using TensorFlow, a crucial framework for building and training models.

By the end of this course, you’ll have a strong grasp of advanced machine and deep learning techniques in rare event prediction. These exercises will not only help you solve complex issues but also become a firm footing for further study and innovation in this field.

Understanding Deep Learning Applications in Rare Event Prediction

It’s always advisable to begin with a baseline model. A restricted stateless LSTM network is taken as a baseline. In such a network, every LSTM layer is stateless, and the final layer has a restricted output, that is:
```python
LSTM(..., stateful=False, return_sequences=False)
```

We’ll now proceed to build the baseline model, outlining each step in the process. 

# Input layer

The input layer in LSTM expects three-dimensional inputs. The input shape should be: 

$$\textit{(batch size, time-steps, features)}$$

A stateless LSTM doesn’t require to specify the batch size explicitly. Therefore, the input shape is defined as follows in the code below.

model = Sequential () 
model.add(Input(shape=(TIMESTEPS , N_FEATURES),
                name='input'))


The input shape can also be provided as an argument to the first LSTM layer defined next. However, this is explicitly defined for clarity.

# LSTM layer

As a general princple, two hidden LSTM layers are stacked in the baseline model. The `recurrent_activation` argument is left as its default `sigmoid` while the output `activation` is set to `relu`. The `relu` activation came into existence after LSTMs. Therefore, they’re not in the legacy LSTM definitions but can be used on the output.

model.add( 
     LSTM(units=16,
          activation='relu', 
          return_sequences=True, 
          name='lstm_layer_1'))
model.add( 
     LSTM(units=8,
          activation='relu', 
          return_sequences=False, 
          name='lstm_layer_2'))

When we run the above code, it shows `Succeeded`, which means that the layers are created successfully.

The first LSTM layer has `return_sequences` set to `True`. This layer, therefore, yields the hidden outputs for every timestep. Consequently, the first layer output is $\textit{(batch size, time-steps, } 16)$. Here, $16$ is the layer size.
Since this is a restricted LSTM network, the last LSTM layer is set with `return_sequences` to `False`. Therefore, it returns the output from only the last time-step. Therefore, the layer output is of the shape $\textit{(batch size, }8)$. Here, $8$ is the layer size.

# Output layer

The output layer should be a dense layer in an LSTM network and most other networks, in general.

**Why?** The output layer should be dense because it performs an affine transformation on the output of the ultimate hidden layer. The purpose of complex hidden layers, such as LSTM and convolutional, is to extract predictive features. However, these abstract features do not necessarily translate to the model output $y$. A dense layer’s affine transformation puts together these features and translates them to the output $y$.

> **Note**: The output layer should be a dense layer for most deep learning networks.

We add a `Dense` layer of size 1. The size is based on the number of responses. For a “binary classifier,” like here, the size is one with `sigmoid` activation. If we have a “multiclass classifier,” the number of labels with `softmax` activation should be the size.







Learn to build and analyze a restricted stateless LSTM network as your baseline model for time series analysis.

Getting Started

Rare Event Prediction

Multi-Layer Perceptrons (MLPs)

Long Short-Term Memory (LSTM) Networks

Convolutional Neural Networks (CNNs)

Autoencoders

Conclusion

Restricted Stateless LSTM Network for Baseline Modeling

Input layer