activation_function

Gain insights into deep learning constructs like LSTM networks and autoencoders. Delve into modeling, prediction strategies, and TensorFlow implementation for rare event prediction in real-life scenarios.

Understanding Deep Learning Application in Rare Event Prediction.png

DLAA.tar.gz

key1

UDL1

UDL2

UDL1-copy

This course aims to provide a practical understanding of the key constructs of deep learning, including Multi-layer Perceptrons, Long-Short-Term Memory (LSTM) networks, convolutional neural networks, and autoencoders, which focus on rare event prediction.

Through this course, you’ll gain hands-on experience developing solutions for rare event prediction. You’ll start by modeling rare events that occur infrequently. Next, you’ll explore machine and deep learning solutions for imbalanced data. You’ll then learn about rare event “prediction,” its statistical foundations, and prediction strategies. Finally, you’ll implement real-life prediction models using TensorFlow, a crucial framework for building and training models.

By the end of this course, you’ll have a strong grasp of advanced machine and deep learning techniques in rare event prediction. These exercises will not only help you solve complex issues but also become a firm footing for further study and innovation in this field.

Understanding Deep Learning Applications in Rare Event Prediction

# Metrics customization

Looking at suitably chosen metrics for a problem tremendously increases the ability to develop better models. Although a metric does not directly improve model training, it helps in a better model selection.
Several metrics are available outside TensorFlow, such as in `sklearn`. However, they can’t be used directly during model training in TensorFlow. This is because the metrics are computed while processing batches during each training epoch.

Fortunately, TensorFlow provides the ability for this customization. The custom-defined metrics `F1Score` and `FalsePositiveRate` are provided in the user-defined `performancemetrics` library. Learning the programmatic context for the customization is important and, therefore, is elucidated here.
The TensorFlow official guide shows the steps for writing a custom metric. It instructs to create a new subclass inheriting the `Metric` class and work on the following definitions for the customization:

* `__init__()`**:** All the state variables should be created in this method by calling the `self.add_weight()` method, e.g.,
  ```python 
  self.var = self.add_weight(...)
  ```
* `update_state()`**:** All updates to the state variables should be done as    
  ```python
   self.var.assign_add(...)
  ```
* `result()`**:** The final result from the state variables is computed and returned in this function.

Using these instructions, the `FalsePositiveRate()` custom metric is defined in the code below. Note that the `FalsePositives` metric is already present in TensorFlow. However, drawing the false positive rate from it during training is not direct, so, the `FalsePositiveRate()` is defined.



class FalsePositiveRate(tf.keras.metrics.Metric): 
  def __init__(self, name='false_positive_rate',
    **kwargs):
    super(FalsePositiveRate , self). 
    __init__(name=name, **kwargs) 
    self.negatives = self.add_weight(name='negatives', 
        initializer='zeros')
    self.false_positives = self.add_weight( 
      name='false_negatives', 
      initializer='zeros')

  def update_state(self,
    y_true , y_pred , sample_weight=None):

  ''' Arguments:
      y_true The actual y. 
             Passed by default to Metric classes.
      y_pred The predicted y. Passed
             by default to Metric classes.
      '''

    # Compute the number of negatives.
    y_true = tf.cast(y_true , tf.bool)

    negatives = tf.reduce_sum(tf.cast( 
      tf.equal(y_true , False), self.dtype))

    self.negatives.assign_add(negatives)
    
    # Compute the number of false positives.
    y_pred = tf.greater_equal( 
      y_pred , 0.5
    ) # Using default threshold of 0.5 to
      # call a prediction as positive labeled.

    false_positive_vector = 
      tf.logical_and(tf.equal(y_true , False),
    tf.equal(y_pred , True)) 
    false_positive_vector = tf.cast(false_positive_vector , 
       self.dtype)
    if sample_weight is not None: 
      sample_weight = tf.cast(sample_weight ,
        self.dtype)
      sample_weight = tf.broadcast_weights(
        sample_weight , values) 
      values = tf.multiply(false_positive_vector , 
        sample_weight)

    false_positives = tf.reduce_sum(false_positive_vector)

    self.false_positives.assign_add( false_positives)

  def result(self):
      return tf.divide(self.false_positives , 
        self.negatives)

## Working of metrics computation and customization

Model training is done iteratively. The iterations happen at multiple levels. The iteration levels for Multi-layer Perceptron (MLP) training are laid out in the illustration below.

The topmost iteration level is _epochs_. Within an epoch, a model is trained iteratively over randomly selected batches. The _batches_ are the second level of iteration.
Multiple samples are present within a batch. The model can be trained by processing one _sample_ at a time. But they’re processed together as a batch for computational efficiency. Therefore, the sample is grayed in the illustration, indicating it’s a logical iteration instead of an actual one.

All the model parameters—weights and biases—are updated during batch processing. Simultaneously, the states of a metric are also updated. The estimated parameters and computed metrics are returned upon processing all the batches in an epoch. 

> **Note:** All these operations are enclosed within an epoch, and no values are communicated between two epochs.

The meaning of the metrics state, metrics computation, and the programmatic logic in the code for defining `FalsePositiveRate` are enumerated below.

* The class `FalsePositiveRate()` is inheriting the metric class in TensorFlow. As a result, it automatically has the `__init__()`, `update_state()`, and `result()` definitions. These definitions will be overwritten during the customization.
* `__init__()` is the entry gate to a metric class. It initializes the **state variables,** which are used for metric computation.
* The false-positive rate is the ratio of false positives over the negatives. Therefore, `false_positives` and `negatives` become the state variables.
* The state variables are prefixed with a `self` variable, which can be accessed and updated from any definition in the class.
* The states—false positives and negatives—are initialized to zero in `__init__()`. At the onset of an epoch, all the states are reset to their initial values. In some problems, the initial value can be other than zero.
* After `__init__()`, the program enters `update_state()`. This function can be imagined as the metric processing unit. It is run for every batch in an epoch. Inside this function, the metric states are updated from the outcome derived from a batch.
*  `update_state()` has the actual labels `y_true` and the predictions `y_pred` for the samples in the batch as default arguments.
* **Lines 23–28** in the function, compute and update the `self.negatives` state variable. The original labels `y_true` in the inputted data are numeric $0$ and $1$.
  * They’re first converted into boolean on **line 23**.   

  * The `negatives` in the batch are then computed by finding all the `False` in the boolean `y_true` vector followed by converting it back to numeric $0/1$ and summing the now numeric vector.
  * The `negatives` computed on **line 25** is the number of negative samples in the batch under process. `negatives` is a local variable in its enclosing function `update_state()`. It is added to the metric state variable `self.negatives` (a class variable) on **line 28**.

* The possibility of a not `None` sample weight is accounted for on **lines 26–29**.

* Similarly, the false positives in the batch are computed on **lines 31-51**. False positives are derived by comparing the predictions `y_pred` with the actuals `y_true`.

  * To get to the false positives, the first step is to convert `y_pred` to boolean. `y_pred` is the output of a _sigmoid_ activated output layer. So, it’s a continuous number in $(0, 1)$.

  * `y_pred` is converted to boolean using a default threshold of $0.5$. The predictions in `y_pred` greater than the threshold are made `True` and `False`, otherwise.

  * A false positive is when the actual label `y_true `is `False`, but the prediction `y_pred` is `True`. That is, the model incorrectly predicted a negative sample as positive. This logical comparison is done on **line 36**.

  * The output of the logical comparison is a boolean vector `false_positive_vector`, which is converted to numeric $0/1$ on **line 39**.

  * The `false_positives` in the batch is computed by summing the vector on **line 49**.

  * Same as before, the `self.false_positives` state is then updated on **line 51**.

* After the epoch has iterated through all the batches, the metric state variables `self.negatives` and `self.false_positives` will have stored the totals for the entire data set.

  * The state `self.negatives` has the total negative samples in the data set. This is a constant and, therefore, `self.negatives` will remain the same in all epochs. `self.false_positives`, on the other hand, is a result of the goodness of model fitting. And so, it changes (ideally reduces) over successive epochs.

* Finally, the program goes to `result()` and yields the metric’s
value. This is the “exit” gate of the metric class. The false-positive rate is computed by dividing the metric states `self.false_positives` and `self.negatives`.

> **Note**: The state in metric customization should be additive only.

A metric state should be additive only. This is because the `update_state()` can only increment or decrement the state variable. For instance, if the false-positive rate—a ratio and so, non-additive—was created as the state variable and updated in `update_state()`, it would yield:


$$ \sum_{i\space∈\space\text{batches}}\frac{\text{false positives}_{i}}{\text{negatives}_i} $$

instead of the desired

$$ \frac{\sum_{i\space∈\space\text{batches}}\text{false positives}_{i}}{\sum_{i\space∈\space\text{batches}}\text{negatives}_i}. $$

In general, any non-additive computation should, therefore, be done in `result()`.

Similarly, the `F1Score` custom metric is defined in the code below.







Discover how to implement custom metrics to refine model evaluation and selection.

Getting Started

Rare Event Prediction

Multi-Layer Perceptrons (MLPs)

Long Short-Term Memory (LSTM) Networks

Convolutional Neural Networks (CNNs)

Autoencoders

Conclusion

Enhancing Model Selection with Custom Metrics

Metrics customization