At our company, we've neglected our "staging environment" for quite some time, and are feeling the pain. It is hurting our iteration speed, and even though we do rigorous unit and integration testing, undefined (or badly-defined) business logic still ends up causing problems in production. It has resulted in a culture where (client-)developers and our support team create "fake" accounts in our production environment to validate customer complaints, or to test newly built customer flows.
It has to be said that our systems are pretty stable, and the amount of bugs we ship is minimal, but the real pain is in the time it takes to validate whatever you are currently working on in an end-to-end environment, outside of your unit-tested world.
We are designing a plan to make real-world testing easier for our different teams, and asked them what their biggest pain points were:
- Missing real-world data to mimic production-usage
- Missing (or broken) key systems in staging, causing missing functionality
- Difficult to start testing from a particular customer state onwards
- Sometimes you really want to test the actual payment flow of customers, without using the sandbox environment of a PSP
We've been thinking on possible solutions, but wanted to survey the land first, and see how other companies (of different sizes) tackle these problems.
I'm interested in your thoughts on this topic, how do you handle testing functionality outside of using unit and integration tests? Do you maintain two (or more) environments? Do you allow testing on your production environment, and if so, how do you model such a system to keep garbage data from impacting other systems?
