Increasing Network Latency

We saw how we can deal with network failures. To be more precise, we saw one possible way to simulate network failures and one way to solve the adverse outcomes it produces. However, it’s not always going to be that easy. Sometimes the network does not fail, and requests do not immediately return 500 response codes. Sometimes there is a delay. Our applications might wait for responses for milliseconds, seconds, or even longer. How can we deal with that?

Let’s see what happens if we introduce a delay to requests’ responses. Is our application, in its current state, capable of handling this well and without affecting end users?

Inspecting the definition of network-delay.yaml

Let’s take a look at yet another chaos experiment definition.

This time, there are more than a few lines of changes.

We’re keeping the part that aborts requests, and we added delays. So, instead of creating a separate experiment to deal with delays, we will do both at the same time. We’re keeping the abort failures, and we’re adding delays. We’re spicing it up so it’s not so far from the “real world” situation. When network failures happen, some other requests might be delayed.

Checking the difference between network-rollback.yaml and network-delay.yaml

Let’s take a look at the differences between this and the previous definition.

The output is as follows.

2,3c2,3
< title: What happens if we abort responses
< description: If responses are aborted, the dependant application should retry and/or timeout requests
---
> title: What happens if we abort and delay responses
> description: If responses are aborted and delayed, the dependant application should retry and/or timeout requests
20c20
<       timeout: 5
---
>       timeout: 15
53a54,69
> - type: action
>   name: delay
>   provider:
>     type: python
>     module: chaosistio.fault.actions
>     func: add_delay_fault
>     arguments:
>       virtual_service_name: go-demo-8
>       fixed_delay: 15s
>       routes:
>         - destination:
>             host: go-demo-8
>             subset: primary
>       percentage: 50
>       version: networking.istio.io/v1alpha3
&g