Chaos engineering in real projects

[vadzim-z]

Where would you recommend using Polly Chaos Engineering in real-case scenarios?
After investigating a topic, I found that Chaos Engineering is often applied along with regression testing before release. But it doesn't work so well with Polly. It looks like, Dev Proxy tool would be more suitable here.
Is it a good idea to use Polly+Simmy with Integration tests? Do you run these tests together with Unit/Integration tests in CI/CD?
How do you use it in commercial projects?

[peter-csala]

cc: @vany0114

[vany0114]

@vadzim-z Regression and unit testing have different purposes and they are usually used in the development lifecycle of your application such as in CI/CD pipelines as you mentioned, regression testing is meant to detect bugs after changes, ensuring that the new code hasn't adversely affected existing functionalities, Chaos engineering, however, is about ensuring the system's robustness against unpredictable events.

On the other hand, tools specialized in network conditions or external service simulation like "Dev Proxy" are indeed great where external dependencies are a factor. However, chaos engineering does not come down to only networking or external dependency failures, how about faults in a given service's data layer? or when the database server is rebooting because got overwhelmed? or when a Redis instance is down? etc, Simmy is still useful in these contexts by simulating failures, latency, or even any behavior you could imagine (like the ones just mentioned) at a more granular level and in a controlled manner.

Regarding integration tests, that's totally valid as well, you could have a "Resilience Testing Stage" in your CI/CD pipeline where Simmy injects faults into the system, it would help ensure that the resilience logic built with Polly is effective under adverse conditions, but that's more a SRE good practice than Chaos Engineering itself IMHO.

I always advise before starting with Chaos Engineering to make sure the system is stable enough, that it is built thinking about pessimistic scenarios, with basic resilience strategies, and with good SRE good practices, then you can start from there by running controlled experiments in a production/near-production environments to find weaknesses and keep improving the reliability of your system.

Allow me to refer to this link, it's old because it is the article I wrote when we first released Simmy but still, very valid IMHO where you can find a practical example that emulates a kind of real-life project and walks you through how to use Polly and Simmy together and how to automate the chaos.

[vadzim-z]

Thank you, @vany0114 for such a detailed explanation!