Introduction to Kafka

Problem statement

Most big companies gather data, analyze it, and make decisions depending on the results. Delivering data from a source to a destination in real time in streams is crucial for many Internet companies. This task is called event streaming. It involves collecting data from different sources like user activity and click information of a page, log messages, credit card transactions, smart appliances, and software applications and storing data efficiently for retrieval by consumer applications later.

For many use cases, it is critical to collect and process individual events in near real time. One such example is to alert a credit-card user about a suspicious transaction and to take the user's consent to proceed. Not delivering such events quickly to the user or a user unable to pay because the system hasn’t yet processed the consent provided on the mobile phone can cause a range of issues, such as potential financial losses and consumer annoyance.