Having worked in complex solutions that had Unit Tests and Integration Test in the CI/CD pipeline, I recall having a tough time with tests that failed randomly (either due to random values being injected or because of the async nature of the process being tested - that from time to time resulted in some weird racing condition). Anyway, having this random behavior in the CI pipeline was not a good experience, we could never say for sure the change a developer committed was really causing the build issue.
I was recently introduced to AutoFixture, which helps in the creation of tests, by randomly generating values - surprisingly I was the only one who did not feel it was a great idea to introduce it in all tests of our CI pipeline.
I mean, I understand fuzz testing, monkey testing, etc but I believe this should be done out of the CI/CD pipeline - which is the place I want to ensure my business requirements are being met by having sturdy, solid and strict to the point tests. Non linear behavior tests like this (and load testing, black box, penetration, etc) should be done outside of the build pipeline - or at least should not be directly linked to code changes.
If these side tests ever find a behavior that is not expected, a fix should be created and a new concrete and repeatable test case should be added to avoid going back to the previous state.
Am I missing something?
AutoFixture
at least supports property-based testing, which is what it looks like you're looking for.QuickCheck
popularized this approach, which allows you to test classes of inputs rather than using random inputs.