Predictive Patterns and Rare Event Prediction
Learn how predictive patterns from data can transform decision-making and the role of AI in interpreting datasets.
We'll cover the following
Data is important to draw inferences and predictions. As obvious as it may sound today, it has been a collective work over the centuries that has brought us this far. Francis Bacon proposed a Baconian method to propagate this “concept” in the 16th century. His book “Novum Organum” advanced Aristotle’s “Organon” to advocate data collection and analysis as the basis of knowledge, laying the groundwork for empirical research that underpins modern data science.
Data collection and analysis have unarguably become an essential part of most processes. As a result,
Without this key, data by itself is a dump. At the same time, drawing valuable predictive patterns from this dump is a challenge that we are facing today.
Note: E.O. Wilson states in his book, Consilience: The Unity of Knowledge, “We are drowning in information while starving for knowledge. In the future, the world will be run by synthesizers, people able to put together the correct information at the right time, think critically about it, and make important choices wisely.”
Learning from patterns
This synthesis is crucial in data science, achieved through adept pattern recognition and predictive modeling. Humans inherently learn patterns from data. For example, as children grow, they learn that touching a hot cup will burn. They come to know this after doing it a few times (collecting data) and realizing the touch burns (a pattern). Over time, individuals learn several other ways that assist them in making decisions.
However, as problems become more complex, humans’ abilities become limited. For example, we might foretell today’s weather by looking at the morning sun but cannot predict it for the rest of the week. This is where artificial intelligence (AI) comes into the picture. AI enables automatic derivation of predictive patterns. Sometimes the patterns are interpretable, and sometimes not. Regardless, these automatically drawn patterns are usually quite predictive.
Challenge of predicting rare events
In this course, we’ll go a little further than these patterns to understand the constructs of deep learning. A rare event prediction problem will be also solved side-by-side to learn the application and implementation of the constructs.
Rare event prediction is a particular problem with profound importance. Rare events are the events that infrequently occur. Statistically, if an event constitutes less than 5% of the dataset, it’s categorized as a rare event. In this course, even rarer events that ”occur less than 1%” are discussed and modeled.
Despite being so rare, when these events occur, their consequences can be pretty dramatic and often adverse. Due to this, such problems are sometimes referred to as adverse event predictions. Rare event prediction problems have been categorized under various umbrellas, such as mining needle in a haystack, chance discovery, exception mining, and so on. However, the rare event prediction problems discussed in this course are distinct from many other categories. They involve anticipating rare events in advance. One illustrative example is predicting a tsunami before it hits the shore. While this example helps illustrate the concept, it’s important to note that the course covers a range of rare event prediction scenarios beyond this specific instance.