Gain insights into Camel Enterprise Integration Patterns, delve into foundational, Error Handling, Deployment, and Configuration patterns, and learn to implement them in varied scenarios with real-world examples.

The Camel Enterprise Integration Patterns (EIP) provide the interfaces to communicate with large component-based distributed systems. When designing the best integration solution for message-oriented applications, these patterns help save time and effort.

In this course, you'll learn about all four foundational patterns. You'll be introduced to Error Handling patterns, Deployment patterns, and Configuration patterns. Each pattern is discussed in detail to be used in distinct settings. You'll also gain a deep understanding of how each pattern works under different circumstances, and implement them appropriately when necessary using real-world examples to illustrate the design specifics. Camel EIPs offer advantages over traditional frameworks such as REST or SOAP regarding integration problems within an organization.

By the end of this course, you'll have hands-on practice with valuable interfaces and the simplification processes for integrating applications across an organization's services.

Using Apache Camel with Enterprise Integration Patterns

## Intent

This pattern enforces resource partitioning and damage containment to preserve partial functionality in the case of a failure.The Bulkhead pattern is also known as the **Failure Containment Principle** and the **Damage Control Principle**.

## Context and problem

The Titanic disaster has been well studied over the years, and there are many lessons we can learn from it in the IT industry. Among the many reasons why it sank, a few of them are as follows:

 - Design flaws (watertight compartments did not reach high enough in order to allow more living space in first class).
 - Implementation/construction faults (the three million rivets used to hold different parts of the Titanic together were found to be made from substandard quality iron, and the collision with the iceberg badly impacted them).
- Operational failures (the iceberg notice was given too late, and the ship was traveling too fast to react to any warning). 

We can identify similar issues in software projects today too, but luckily they have not cost so many human lives so far, and as a consequence, we learn more slowly than other industries. However, that does not mean software failures have no consequences. Plenty of examples (such as the Therac-25 machine) caused human death because of software defects.

From an architectural point of view, the ship was designed to cope with four compartments being flooded, but the bulkheads that isolated the compartments did not reach the deck above and weren’t watertight. As a result, the water spread to more compartments and sank the ship. Similar cascading failures exist in software systems too. A failure in one component can exhaust all available resources, and it can spread to other components until the whole system is down. Avoiding similar failure scenarios requires resource partitioning and capacity isolation at all levels, from data centers to individual thread pools.


## Forces and solution

The Bulkhead pattern in software systems works by the same principle as in ships. By partitioning a system into separate components and isolating the resources, failures cannot cascade and bring the whole system down. This pattern enforces the damage containment principle and improves the system’s resilience. Implementing the Bulkhead pattern can be done at many different granularity levels depending on the type of faults we want to protect the system from.

* The most common way to apply the Bulkhead pattern is through physical redundancy. In this age of cloud computing and virtual machines, the only way to ensure that two hosts are on separate hardware (compute, storage, or networking) is by ensuring they are on two separate data centers (for example, AWS Availability Zones). Having this separation, hardware failure in one processing unit cannot affect the other.


* Virtualization is still the most common way for resource isolation and scalability. So the next level for applying the Bulkhead pattern is to use redundant hosts (virtual machines).

* The next level of the Bulkhead pattern is having isolated processes on the hosts. The processes can be CPU-bound with technologies such as cGroups or Docker containers. It is also possible to dedicate a certain amount of memory, limit disk size, put constraints on other resources to each process, and prevent resource monopoly and contention.

* The Bulkhead pattern can be applied to ensure thread pool isolation at the application level. So, rather than having a single thread pool for all tasks which can be exhausted, using separate thread pools for critical tasks can ensure thread leakage containment. Threads are granular software primitives that map to hardware resources (CPU time). Ensuring isolated thread pools for critical parts of the application is essential.

   Each of these isolation boundaries addresses different concerns and mitigates a different set of risks. Thread pool isolation will ensure partial functionality and graceful degradation of different parts of the application. For example, it can ensure that admin functionality is still working or the request handling endpoint is still functioning while the database connection pool is exhausted. Having multiple service instances as separate processes will ensure that a hung process does not bring down all instances. Using a separate host will provide further capacity isolation and easier scalability. Moreover, physical redundancy will allow disaster recovery or transparent failover as a last resort.


## Mechanics

The Bulkhead pattern is one implementation of the failure containment principle. Applying it requires using many other patterns and principles at different levels. Let's see what should be considered to implement this pattern when looking from an application development point of view.

### Hardware redundancy

With its “Write once, run anywhere (WORA)” promise, the Java platform makes sure the applications behave the same way even on different hardware. Still, keep in mind that there might be some differences if the underlying hardware architecture differs. For example, a 32-bit architecture will allow only a 2GB effective heap, whereas a 64-bit one will consume more memory and hit the Java garbage collector harder. Having a distributed system deployed on geographically dispersed hardware will also increase the chances of potential issues caused by any of the eight distributed computing fallacies.

### Host redundancy

This category is also mostly taken care of by the Java platform's ability to run on different operating systems, but there are a few more things to be cautious about. Different hosts may run different operating systems, and as such, when writing an application, do not make assumptions about the line endings (Windows will use two symbols, whereas Unix will use only one), file separators, disk structures, file locking capabilities, or command line calls. If we have made any assumptions about these, or about the Java version, or assumed localhost to be the only hostname when implementing a service, it will limit the available hosts to deploy a service.

### Process isolation

This principle relies on having multiple instances of the service as separate processes (whether that is on the same host or not is irrelevant). So, implementing the Service Instance pattern requires many considerations described in the chapter with the same name. For example, if we have used an in-process communication mechanism such as Camel VM or direct-VM components for service communication, it will not be possible to split these two services into separate processes without refactoring both services.


Using multiple processes for the same service is also popular in messaging. There is a scaling technique in ActiveMQ called client-side partitioning. With this technique, the message queues are distributed into multiple message broker instances rather than having one message broker instance for all services. The messaging clients are configured to connect to the right broker instances—hence the pattern’s name. Then we can locate queues for critical services in a broker instance with more redundancy or allocate a separate broker instance for important customers. This way, if the broker process dies, it will only affect the services using that broker instance.


### Thread pool isolation

At this most granular level, applying the Bulkhead pattern is a part of the individual service design and development. Assuming that failures will happen, this pattern aims to isolate and control failures in part of the application from spreading and affecting the whole application. We can use whatever patterns, principles, and practices are available to achieve this. Let's see a few that are very common:

* **Parallel pipeline pattern**: This allows breaking a processing flow into independent sub-flows. When the interaction style is fire-and-forget where no response is expected, a failure in one sub-flow will not affect the rest of the pipeline. The other sub-flows can continue accepting and processing incoming requests.

* **Load Levelling pattern**: Using queues in general for temporal decoupling will help isolate different parts of the application.

* **Other related patterns**: Using the following stability patterns does not contribute to the implementation of bulkheads, but these patterns will improve the system's stability, making it less likely to fail and need bulkheads. These are patterns such as throttling, circuit breakers, timeouts, even CQRS.

* **Thread pool segregation**: This is the most popular way to implement bulkheads at the application level. Let’s briefly cover some key areas to configure in Camel regarding thread segregation: Camel uses multithreading to perform concurrent tasks at many different levels. There are components, EIPs, and error handlers that offer parallel processing features. Usually, whenever Camel offers parallelism, it is also possible to tune and customize it with a custom thread pool. By default, Camel thread pools are based on the implementations provided by the `java.util.concurrent` package and described through Camel Thread Pool Profiles. Profiles are a declarative way of describing the various thread pool configurations and behaviors (such as `poolSize, maxPoolSize, keepAliveTime, maxQueueSize, and rejectedPolicy`) through configuration. By default, a profile (called the default thread pool profile) is used as a base for creating thread pools. For the record, as of this writing it has the following default values:` poolSize=10, maxPoolSize=20, keepAliveTime=60s, maxQueueSize=1000, rejectedPolicy=CallerRuns`. So, if we do not customize the default profile and do not use a different profile, all thread pools will have the aforementioned configuration, which might not be ideal for our application. There are some exceptions—internal thread pools created for aggregating messages in Recipient List, splitter, and multicast ignore profiles and create unbounded cached thread pools to ensure the aggregate on-the-fly task will always be assigned a thread as soon as it is submitted. All other thread pools in Camel will be based on the default thread pool profile.

* **Consumers**:  Usually, polling consumers (as opposed to event-driven consumers) use a separate thread to perform the polling. Some of the components (such as JMS, SEDA, Kestrel, Hazelcast, and Scheduler) provide `concurrentConsumers` the option to specify the number of threads used for polling. So, before using a component, one must check whether the component offers any parallelism and whether the default thread pool configurations are feasible.


* **Single-threaded components**: Some components, such as the file consumer, are single-threaded. If the only worker thread gets blocked, the whole route will stop functioning. To make it a little more resilient and scale such a consumer, we can use the Threads DSL in Camel to process files in parallel. Threads DSL can turn a synchronous route into an asynchronous one by processing the messages using new threads from the Threads DSL point forward.


* **Custom components**: When creating a custom component that uses threads, use `ExecutorServiceManager` from Camel rather than creating a custom pool manually. This class will ensure that the thread profiles are honored and do some resource management for us. Similar concepts also exist in other projects like JBoss WildFly, where the whole thread management is delegated to the application server.


* **Multithreaded EIPs**: The following EIPs support parallel processing and allow custom thread pools. When an EIP is configured for parallel processing, it creates a thread pool based on the default thread pool profile (so bulkhead is already applied), but make sure the default configuration makes sense for this use case. The EIPs are delayer, multicast, Recipient List, splitter, threads, Throttler, WireTap, polling consumer, ProducerTemplate, and OnCompletion.

* **Error handler**: The error handler in Camel can also perform asynchronous tasks, like delayed retries. If no thread pool is provided, all the error handlers in the `CamelContext` will share the same thread pool. So it is a potential area where asynchronous redelivery in one route consumes the threads and blocks the redelivery thread in another route, as the thread pool is shared for the whole `CamelContext`. Make sure to either tune the global `ErrorHandlerExecutorService` or specify separate thread pools for some error handlers.

* **Camel Hystrix Component**:  Hystrix is a fault tolerance library designed to stop cascading failures and enable resilience in distributed systems. As of this writing, there is no official Camel connector for Hystrix yet, but it is a work in progress and should be available soon. The Hystrix component can also provide thread and semaphore isolation as a bulkhead.

This is an example application with the Bulkhead pattern applied that is fully symmetrical. In the real world, we should analyze the loss of a capability and its impact on the business and let this drive the isolation boundary.


# Intent

This pattern enforces resource partitioning and damage containment to preserve partial functionality in the case of a failure.The Bulkhead pattern is also known as the **Failure Containment Principle** and the **Damage Control Principle**.

# Context and problem

The Titanic disaster has been well studied over the years, and there are many lessons we can learn from it in the IT industry. Among the many reasons why it sank, a few of them are as follows:

 - Design flaws (watertight compartments did not reach high enough in order to allow more living space in first class).
 - Implementation/construction faults (the three million rivets used to hold different parts of the Titanic together were found to be made from substandard quality iron, and the collision with the iceberg badly impacted them).
- Operational failures (the iceberg notice was given too late, and the ship was traveling too fast to react to any warning). 

We can identify similar issues in software projects today too, but luckily they have not cost so many human lives so far, and as a consequence, we learn more slowly than other industries. However, that does not mean software failures have no consequences. Plenty of examples (such as the Therac-25 machine) caused human death because of software defects.

From an architectural point of view, the ship was designed to cope with four compartments being flooded, but the bulkheads that isolated the compartments did not reach the deck above and weren’t watertight. As a result, the water spread to more compartments and sank the ship. Similar cascading failures exist in software systems too. A failure in one component can exhaust all available resources, and it can spread to other components until the whole system is down. Avoiding similar failure scenarios requires resource partitioning and capacity isolation at all levels, from data centers to individual thread pools.


# Forces and solution

The Bulkhead pattern in software systems works by the same principle as in ships. By partitioning a system into separate components and isolating the resources, failures cannot cascade and bring the whole system down. This pattern enforces the damage containment principle and improves the system’s resilience. Implementing the Bulkhead pattern can be done at many different granularity levels depending on the type of faults we want to protect the system from.

* The most common way to apply the Bulkhead pattern is through physical redundancy. In this age of cloud computing and virtual machines, the only way to ensure that two hosts are on separate hardware (compute, storage, or networking) is by ensuring they are on two separate data centers (for example, AWS Availability Zones). Having this separation, hardware failure in one processing unit cannot affect the other.


* Virtualization is still the most common way for resource isolation and scalability. So the next level for applying the Bulkhead pattern is to use redundant hosts (virtual machines).

* The next level of the Bulkhead pattern is having isolated processes on the hosts. The processes can be CPU-bound with technologies such as cGroups or Docker containers. It is also possible to dedicate a certain amount of memory, limit disk size, put constraints on other resources to each process, and prevent resource monopoly and contention.

* The Bulkhead pattern can be applied to ensure thread pool isolation at the application level. So, rather than having a single thread pool for all tasks which can be exhausted, using separate thread pools for critical tasks can ensure thread leakage containment. Threads are granular software primitives that map to hardware resources (CPU time). Ensuring isolated thread pools for critical parts of the application is essential.

   Each of these isolation boundaries addresses different concerns and mitigates a different set of risks. Thread pool isolation will ensure partial functionality and graceful degradation of different parts of the application. For example, it can ensure that admin functionality is still working or the request handling endpoint is still functioning while the database connection pool is exhausted. Having multiple service instances as separate processes will ensure that a hung process does not bring down all instances. Using a separate host will provide further capacity isolation and easier scalability. Moreover, physical redundancy will allow disaster recovery or transparent failover as a last resort.


# Mechanics

The Bulkhead pattern is one implementation of the failure containment principle. Applying it requires using many other patterns and principles at different levels. Let's see what should be considered to implement this pattern when looking from an application development point of view.

## Hardware redundancy

With its “Write once, run anywhere (WORA)” promise, the Java platform makes sure the applications behave the same way even on different hardware. Still, keep in mind that there might be some differences if the underlying hardware architecture differs. For example, a 32-bit architecture will allow only a 2GB effective heap, whereas a 64-bit one will consume more memory and hit the Java garbage collector harder. Having a distributed system deployed on geographically dispersed hardware will also increase the chances of potential issues caused by any of the eight distributed computing fallacies.

## Host redundancy

This category is also mostly taken care of by the Java platform's ability to run on different operating systems, but there are a few more things to be cautious about. Different hosts may run different operating systems, and as such, when writing an application, do not make assumptions about the line endings (Windows will use two symbols, whereas Unix will use only one), file separators, disk structures, file locking capabilities, or command line calls. If we have made any assumptions about these, or about the Java version, or assumed localhost to be the only hostname when implementing a service, it will limit the available hosts to deploy a service.

## Process isolation

This principle relies on having multiple instances of the service as separate processes (whether that is on the same host or not is irrelevant). So, implementing the Service Instance pattern requires many considerations described in the chapter with the same name. For example, if we have used an in-process communication mechanism such as Camel VM or direct-VM components for service communication, it will not be possible to split these two services into separate processes without refactoring both services.


Using multiple processes for the same service is also popular in messaging. There is a scaling technique in ActiveMQ called client-side partitioning. With this technique, the message queues are distributed into multiple message broker instances rather than having one message broker instance for all services. The messaging clients are configured to connect to the right broker instances—hence the pattern’s name. Then we can locate queues for critical services in a broker instance with more redundancy or allocate a separate broker instance for important customers. This way, if the broker process dies, it will only affect the services using that broker instance.


## Thread pool isolation

At this most granular level, applying the Bulkhead pattern is a part of the individual service design and development. Assuming that failures will happen, this pattern aims to isolate and control failures in part of the application from spreading and affecting the whole application. We can use whatever patterns, principles, and practices are available to achieve this. Let's see a few that are very common:

* **Parallel pipeline pattern**: This allows breaking a processing flow into independent sub-flows. When the interaction style is fire-and-forget where no response is expected, a failure in one sub-flow will not affect the rest of the pipeline. The other sub-flows can continue accepting and processing incoming requests.

* **Load Levelling pattern**: Using queues in general for temporal decoupling will help isolate different parts of the application.

* **Other related patterns**: Using the following stability patterns does not contribute to the implementation of bulkheads, but these patterns will improve the system's stability, making it less likely to fail and need bulkheads. These are patterns such as throttling, circuit breakers, timeouts, even CQRS.

* **Thread pool segregation**: This is the most popular way to implement bulkheads at the application level. Let’s briefly cover some key areas to configure in Camel regarding thread segregation: Camel uses multithreading to perform concurrent tasks at many different levels. There are components, EIPs, and error handlers that offer parallel processing features. Usually, whenever Camel offers parallelism, it is also possible to tune and customize it with a custom thread pool. By default, Camel thread pools are based on the implementations provided by the `java.util.concurrent` package and described through Camel Thread Pool Profiles. Profiles are a declarative way of describing the various thread pool configurations and behaviors (such as `poolSize, maxPoolSize, keepAliveTime, maxQueueSize, and rejectedPolicy`) through configuration. By default, a profile (called the default thread pool profile) is used as a base for creating thread pools. For the record, as of this writing it has the following default values:` poolSize=10, maxPoolSize=20, keepAliveTime=60s, maxQueueSize=1000, rejectedPolicy=CallerRuns`. So, if we do not customize the default profile and do not use a different profile, all thread pools will have the aforementioned configuration, which might not be ideal for our application. There are some exceptions—internal thread pools created for aggregating messages in Recipient List, splitter, and multicast ignore profiles and create unbounded cached thread pools to ensure the aggregate on-the-fly task will always be assigned a thread as soon as it is submitted. All other thread pools in Camel will be based on the default thread pool profile.

* **Consumers**:  Usually, polling consumers (as opposed to event-driven consumers) use a separate thread to perform the polling. Some of the components (such as JMS, SEDA, Kestrel, Hazelcast, and Scheduler) provide `concurrentConsumers` the option to specify the number of threads used for polling. So, before using a component, one must check whether the component offers any parallelism and whether the default thread pool configurations are feasible.


* **Single-threaded components**: Some components, such as the file consumer, are single-threaded. If the only worker thread gets blocked, the whole route will stop functioning. To make it a little more resilient and scale such a consumer, we can use the Threads DSL in Camel to process files in parallel. Threads DSL can turn a synchronous route into an asynchronous one by processing the messages using new threads from the Threads DSL point forward.


* **Custom components**: When creating a custom component that uses threads, use `ExecutorServiceManager` from Camel rather than creating a custom pool manually. This class will ensure that the thread profiles are honored and do some resource management for us. Similar concepts also exist in other projects like JBoss WildFly, where the whole thread management is delegated to the application server.


* **Multithreaded EIPs**: The following EIPs support parallel processing and allow custom thread pools. When an EIP is configured for parallel processing, it creates a thread pool based on the default thread pool profile (so bulkhead is already applied), but make sure the default configuration makes sense for this use case. The EIPs are delayer, multicast, Recipient List, splitter, threads, Throttler, WireTap, polling consumer, ProducerTemplate, and OnCompletion.

* **Error handler**: The error handler in Camel can also perform asynchronous tasks, like delayed retries. If no thread pool is provided, all the error handlers in the `CamelContext` will share the same thread pool. So it is a potential area where asynchronous redelivery in one route consumes the threads and blocks the redelivery thread in another route, as the thread pool is shared for the whole `CamelContext`. Make sure to either tune the global `ErrorHandlerExecutorService` or specify separate thread pools for some error handlers.

* **Camel Hystrix Component**:  Hystrix is a fault tolerance library designed to stop cascading failures and enable resilience in distributed systems. As of this writing, there is no official Camel connector for Hystrix yet, but it is a work in progress and should be available soon. The Hystrix component can also provide thread and semaphore isolation as a bulkhead.

This is an example application with the Bulkhead pattern applied that is fully symmetrical. In the real world, we should analyze the loss of a capability and its impact on the business and let this drive the isolation boundary.


Learn the Bulkhead design pattern and its usage.

About This Course

Foundational Patterns

Error Handling Patterns

Deployment Patterns

Conclusion

Bulkhead Pattern

Intent

Context and problem

Forces and solution