activation_function

Gain insights into deep learning constructs like LSTM networks and autoencoders. Delve into modeling, prediction strategies, and TensorFlow implementation for rare event prediction in real-life scenarios.

Understanding Deep Learning Application in Rare Event Prediction.png

DLAA.tar.gz

key1

UDL1

UDL2

UDL1-copy

This course aims to provide a practical understanding of the key constructs of deep learning, including Multi-layer Perceptrons, Long-Short-Term Memory (LSTM) networks, convolutional neural networks, and autoencoders, which focus on rare event prediction.

Through this course, you’ll gain hands-on experience developing solutions for rare event prediction. You’ll start by modeling rare events that occur infrequently. Next, you’ll explore machine and deep learning solutions for imbalanced data. You’ll then learn about rare event “prediction,” its statistical foundations, and prediction strategies. Finally, you’ll implement real-life prediction models using TensorFlow, a crucial framework for building and training models.

By the end of this course, you’ll have a strong grasp of advanced machine and deep learning techniques in rare event prediction. These exercises will not only help you solve complex issues but also become a firm footing for further study and innovation in this field.

Understanding Deep Learning Applications in Rare Event Prediction

MLP Models Comparison

align-center

The baseline model has higher accuracy measures but increasing validation loss indicating potential overfitting. Dropout resolves the overfitting, but the accuracy goes to zero. Class weights boosted the accuracy but also had a high false-positive rate. The `selu` activation attempted had significantly lower f1-score than the baseline.

The custom activation `telu` performed relatively better than the others. It has a non-increasing validation loss and is close to the baseline accuracy. Therefore, the `telu` activated model is selected as the final model.

This does not mean `telu` will be the best choice in other cases, too. The process of building multiple models and selection should be followed for every problem.

The final selected model is evaluated using the `model.evaluate()` function. The `evaluate` function applies the trained model on test data and returns the performance measures defined in the `model.compile()` function.

The evaluation and test results are shown in the code below.



# Final model
loss,accuracy,recall,f1_score,fpr = model.evaluate( x=X_test_scaled ,
                                                    y=y_test , batch_size=128,
                                                    verbose =1)

print(' Loss: ',loss, '\n Acuuracy: ',accuracy, 
        '\n recall_5: ',recall, '\n f1_score: ',f1_score,
        '\n false_positive_rate: ',fpr)
# loss: 0.0796 - accuracy: 0.9860 -
# recall_5: 0.0755 - f1_score: 0.1231 - # false_positive_rate: 0.0020

In practice, there are many choices, but they could be overwhelming. Let's take a look at them.


## Number of layers and nodes
* **Number of layers:**
 Start with two hidden layers, excluding the output layer. 

* **Number of nodes of intermediate layers:** This is a number from a geometric series of $2: 1, 2, 4, 8, 16, 32, ...$. The first layer should be around half of the number of input data features. The next layer size is half of the previous and so on.

* **Number of nodes of the output layer**: 

  * **Classification:** If it’s binary classification, the size is one.
For a multiclass classifier, the size is the number of classes.
  * **Regression:** If it’s a single response, the size is one. For multi-response regression, the size is the number of responses.
### Activation and dropout
* **Activation**:
  * **Intermediate layers:** Use `relu` activation.

  * **Output layer:** Use `sigmoid` for binary classification, `softmax`
for a multiclass classifier and linear for regression.

* **Dropout:** Don’t add dropout in a baseline MLP model. It should be added to a large or complex network. Dropout should certainly not be added to the input layer.
### Data preparation
* **Data Preprocessing:** Make the features $X$ numeric by converting any categorical columns into one-hot-encoding. Then, perform feature scaling. Use `StandardScaler` if the features are un- bounded and `MinMaxScaler` if bounded.

* **Splitting the data:** Split the data to train, valid, and test datasets using `train_test_split` from `sklearn.model_selection` to do this.
### Data imbalance and optimization
* **Class weights**: Should be used with caution and better avoided for extremely imbalanced data. If used in a binary classifier, the weights should be: 

  * For class $0$, the following formula should be used: 
  $$\frac{\text{number of}\space 1\text{s}} {\text{data size}}$$

  * For class $1$, the following formula should be used : 
  $$\frac{\text{number of}\space 0\text{s}} {\text{data size}}$$

* **Optimizer:** Use `Adam` with its default learning rate.
### Loss functions and evaluation
* **Loss:**
  * **Classification:** For binary classification, use `binary_crossentropy`. For multiclass, use `categorical_crossentropy` if the labels are one-hot-encoded; otherwise, use
`sparse_categorical_crossentropy` if the labels are integers.
  * **Regression:** Use `mse`.

* **Metrics:**
  * **Classification:** Use `accuracy` that shows the percent of correct classifications. For imbalanced data, include `Recall`, `FalsePositiveRate`, and `F1Score`.
  * **Regression:** Use `RootMeanSquaredError()`.
### Model fitting
* **Epochs:** Set it as 100 and see if the model training shows decreasing loss and any improvement in the metrics over the epochs.

* **Batch size**: Choose the batch size from the geometric progression of 2. Imbalanced data sets have larger values, like 128; otherwise, start with 16.

### Advanced parameters

* **Oscillating loss:** If the oscillating loss is encountered upon training, there’s a convergence issue. Try reducing the learning rate and/or changing the batch size.

* **Oversampling and undersampling:** 
Random sampling will work better for multivariate time series than synthesis methods like `SMOTE`.

* **Curve shifting:**
 If the time-shifted prediction is required, for example, in an early prediction, use curve shifting.

* `selu` **activation:**
`selu` activation has been deemed better for large networks. If you’re using `selu` activation then use `kernel_initializer=’lecun_normal’` and `AlphaDropout()`. In `AlphaDropout`, use the rate as `0.1`.









Learn the process of selecting the best neural network model based on validation results. 

Model	Validation
Model		Loss	f1-score	Recall	fpr
Baseline	Increasing	0.13	0.08	0.001
Dropout	Non-increasing	0.00	0.00	0.000
Class weights	Non-increasing	0.12	0.31	0.102
`selu`	Nonincreasing	0.04	0.02	0.001
`telu` (custom)	Non-increasing	0.12	0.08	0.001

Getting Started

Rare Event Prediction

Multi-Layer Perceptrons (MLPs)

Long Short-Term Memory (LSTM) Networks

Convolutional Neural Networks (CNNs)

Autoencoders

Conclusion

Optimizing and Evaluating the Final Neural Network Model

MLP Models Comparison