Data Integrity Pattern
Learn the Data Integrity design pattern and its usage.
Intent
This pattern is used to maintain the data consistency and business integrity of a system with dispersed data sources in the case of a processing failure. The Data Integrity pattern is also known as the Transactional Service pattern and the Atomic Service Transactions.
Context and problem
Integration applications typically process data from disparate data sources such as a relational database, key-value store, message queue, or file system. Maintaining data integrity in an environment where each data source has a different transaction model can be challenging. If a service has to mutate two or more data sources, there is a chance that these data sources will get out of sync in the case of a failure of the mutating service.
Forces and solution
Depending on the data consistency requirements, there are different approaches for dealing with failure scenarios:
-
Some systems do not offer transactional behavior at all. The data consistency has to be ensured through additional mechanisms such as:
- Data locking before modifying.
- Committing changes when the operation completes successfully.
- The undoing modifications when there are failures.
-
All changes in a strongly consistent system are atomic (indivisible), consistent, isolated, and durable (ACID). This model relies on using locks within a single data source and impacts the availability and scalability of the system. If a service has to modify multiple data sources, it has to be wrapped in a global transaction, and then this transaction will ensure that either all changes are successful or that no changes are carried out.
-
In an eventually consistent model, the data sources are partitioned across processes, and it is impossible to use locking to ensure consistency. If a service has to modify multiple data sources, it requires additional coordination logic to guarantee the overall system integrity.
In this lesson, we will look at a nontransactional example and a few strongly consistent use cases.
The eventually consistent model will be covered later as part of the Saga pattern.
Mechanics
In the Camel world, the Data Integrity pattern is implemented in four different ways:
No transactions
The main goal of the Data Integrity pattern is to ensure that the data that is managed by a service is mutated and persisted according to ACID principles. The integration applications do support a variety of protocols, and not all of them support transactions. In these situations, these are used, whichever means are available for mutating data in an ACID-compliant way. Let’s look at a simple and still popular way of exchanging information, the file transfer, and its potential issues.
Get hands-on with 1400+ tech skills courses.