What is complex event processing?

Complex event processing (CEP) is an approach to searching patterns in a real-time stream of events. In CEP systems, there is a stream of events generated by the users or devices, and queries for patterns in the events are created in a high-level declarative query language. These queries are run against the event stream in real-time.

For example, suppose we are building a fraud detection system for a bank that can detect fraudulent transactions in real-time.

Event Stream

The card transaction data is the event stream on which we want to apply different rules to detect fraudulent transactions.

Pattern Query

Suppose our business analysts come up with a pattern that causes the system to report a transaction as fraudulent if a user performs that transaction more than five times in a window of 15 minutes.

CEP systems provide a high-level declarative query language (similar to SQL). For example, the query below is for the event described above.

FROM transaction_data_stream#window.time(5 mins)
SELECT userid, count(userid) as TransactionCount
GROUP BY userid
HAVING TransactionCount >3
INSERT INTO fraudulent_transaction_stream;

Result

The query is parsed and run on the stream of events by an event processing framework during runtime, while matched events are written to a different output stream. In our case, a notification service can subscribe to the output stream and send an email to the user to confirm that the transactions were done by them and are not fraudulent.

Complex stream processing systems are completely opposite, in some ways, to classic database query systems where data is stored permanently, queries are run against the data, and then the query is forgotten.

However, in CEP systems, queries are stored long-term and are run against an ever-changing streaming data to look for patterns.