I recently a part of a production deployment. It is not like a deployment of the early 2000s where everything went up and you expected it to work. No. It was deployed in parts because, well, we forgot stuff and we discovered stuff.
We deployed the product and started it up. We discovered something we didn’t expect. In the early 2000s, we would have executed the back out plan and deployed another day (a delay in business value). However, this is 2019. We made a correction in production and tried again. We did this several times until we finally had the entire product operating as expected. Early reviews of its functions say that the product was even operating successfully (read: providing business value now). I love testing in production!
Could we have had a better plan for deployment? Yes! Based on this experience, our next deployment will be better. Ain’t that being agile?
Did we deploy the product that was approved by multiple interested parties? Hell no! We deployed something that demonstrated excellent functionality in a non-production environment. When we deployed to production, we solved issues like forgotten configurations, forgotten files, and an unexpected data format. That is, we LEARNED a lot about our product in another environment, corrected our mistakes and misses, and got it working. When it started working, then the real learning started!
We discovered some things our product could not do because of data quality (by the way, our error handling caught these!). These will be addressed in a future deployment. We discovered some functionality that needs tweaking. This will be addressed in a future deployment.
Going forward, we deploy small and deploy often. We will learn as we did above because it is soooo valuable! Testing in production is not a crime, it is a learning experience! If you can, go forth and learn!