Kafka Connect Source Connectors

Previously, we studied that Kafka Connect Source connectors are used to pipe data from source systems into Kafka topics. Kafka Connect has connectors for a wide variety of systems, including databases.

In this lesson, we will learn how to build a data pipeline from PostgreSQL to Kafka. PostgreSQL is a powerful, open-source relational database that provides robust data integrity and scalability. It is known for its advanced features such as support for complex queries, transactional processing, and data analysis. PostgreSQL is widely used in enterprise-level applications, data warehousing, and web-based systems. One of the most popular Source connectors in the Kafka ecosystem is the Debezium connector for PostgreSQL.

Here is a high-level representation of the solution we will implement. Let’s dive into its individual building blocks and learn how they work.