In this guide, you will learn about the process I follow when pushing a release into production. Why listen to me? I've been deploying websites for large enterprise-level organizations for over 15 years. I've worked with some of the biggest companies in the UK. I've worked within a large CMS vendor where I helped clients ship code on a near-daily basis. I've worked in some of the UKs biggest digital agencies. I've also worked client-side, where the cost of getting a release wrong could be up to 50 thousand pounds an hour, so mistakes were costly.
In that time, I've learned a few basic rules of how to increase the chances that your release will go smoothly that I will share below. Stressful releases mean late nights, angry team members and going home at the end of the night venting why your job is so shit. Life is too short to stress. By following some simple practices and processes you can make your life a lot easier. Below outline 7 simple steps you should follow that will reduce the chance of you bumping into issues during a release.
1. Be Prepared 🍳
Before starting any deployment process to production, you need to know exactly what you're doing. Trying to figure it out on the day is a recipe for disaster. Trust me.
Other professional professions have known this fact for years. Take flying. In order to become a pilot, you need hundreds of hours of practice flying. Before a pilot takes off they have to go through a number of pre-flight checklists. Hundreds of hours are put into creating the checklists the pilots need to follow. Thousands of plane crashes have been researched and the mistakes that resulted in the crashes have been included in the checklists to avoid future incidents. In aviation, they learn from their mistakes as the stakes are so high.
On each flight you take, there will be a co-pilot to ensure that all the preparation steps are followed. When we ship software though, no one in your company is likely to enforce this on you. When you don't have a clear picture of what you need to do, you significantly increase the chances that things will go wrong.
2. Check the code works 💙
Before releasing anything, it needs to work. I can't count the number of times that some small last-minute change becomes the root cause for weeks worth of careful planning and preparation to be reverted. Never trust the words, 'it won't break anything' just before a deployment!
It is a common pressure in most companies to ship last-minute code changes quickly, untested, rather than ensure it works. If the code you're shipping is wrong, no matter how good your deployment process the end result will still be a failure, no matter how hard you try, or how good your deployment process is.
You get paid by a company to act like a professional. You wouldn't be too impressed if you went into life-threatening surgery and the doctor turned up late, with no clue what was wrong with you, or what he needed to do in order to fix you. The same principles are true for releasing software. 'winging it' is not a scalable approach, especially when you work in a team and you have people relying on you. You may find documenting stuff is boring, but it needs to be done. That's why you get paid the big bucks.
3. Document your release process ☂️
When I started my career I thought I knew it all when it came to shipping code. Granted, I made very few mistakes, as I knew what I was doing I didn't see the point of following a checklist, or, documenting what I was doing. As I was doing the work, it didn't affect anyone else. Documenting stuff I knew felt inefficient, why waste the time? The problem with this type of thinking is that the people on the team didn't know what was going on. Project work is teamwork. Even if you find it a chore, documenting what you are doing helps the team. The better you can communicate in the team and the wider company the more people will appreciate you.
If an issue occurs during a release, a documented process makes the project managers and the business owners feel all warm and fuzzy. When people in the team are unsure of what to do, they panic. To get around these issues, document your steps. In times of crisis, explaining things to people increases stress. Most issues that you encounter during a release can be considered ahead of time and a rollback and contingency plan can be documented.
4. Follow the process 🏁
After you have done 100 releases of the same website, the steps that guaranteed you success, don't seem too important... the releases usually go smoothly, so you skip them. Then something goes wrong. If you don't believe how important it is to follow the steps that guarantee your success, I recommend reading The Checklist Manifesto. The book documents numerous case studies on why checklists are essential in order for complex processes to go smoothly 🧈🧈🧈
5. Get the right people 👑
The next thing you need to ensure is that you have access to all the right people. Do you have all the right permissions, or, do you need an infrastructure person to flick any switches? Have you got a tester lined up? Is the client online to confirm it is working? Not having the right person in a key step can cause delays. Simply checking in with everyone before a release and confirming they are available is something you should do on every release.
6. Agreed communication channels defined 📍
Before you release any code, you need to ensure the right people know. If you don't do this, you'll likely get in trouble with someone. In these instances, if you let everyone who cares know you might be doing something that could affect them, you've done your part. How you let people know will vary depending on where you work, usual candidates are:
Slack/teams, etc..
Email - Ensure you use distribution groups, rather than individual email addresses. This will avoid issues when people leave jobs etc...
3rd party portals, support tickets, etc...
The best rule to follow around release communication.. the more the better until you get told differently. If your change can break someone else's system, as long as they know about it beforehand, no one can blame you. If you break another team's system and you didn't tell them, they then waste time trying to figure out what went wrong, people tend to get mad. In short... it's your fault. if you tell everyone beforehand, there are no excuses. In large companies with hundreds of teams, this is a vital step to success!
7. Automation 🥊
Ideally, your deployment process should be automatic and triggered by a single click of a button. If you manually need to copy any files from one place to another place, your process is wrong. A general observation that I have made in life, is that the companies who manual deploy code, tend to have more bugs and waste more time fire-fighting compared to those who invest some upfront time creating a CI/CDS pipeline. Strangely, the main excuse that I encounter when companies skip building a CI/CD pipeline is time. The companies who invest in automation usually have a little more overhead at the beginning, but, save exponentially so much more time over the long term.
You may not be able to move to a Cid pipeline today, however, if you perform regular releases start to scream and shout about it. If the release goes wrong because you mess up a manual step in the process, it's not your fault. It's only your fault when you give up trying to move the team towards automation. Any manual steps should be automated. Any mistakes made while messing up a step in the manual workflow is a process issue, not a personal issue. In a manual process, you need to have all the release steps documented and agreed upon. Documenting a manual release process is an essential release task!
8. Clear the cache on each release! 🧂
I can't count the number of times that after a release has been made, a tester has flagged there's an issue, however, after a period of time, the issue magically solves itself. This is the beauty of caching. After deploying code into production, make sure you kill whatever cache is needed. Ensure testers kill their browser cache, you clean the output cache, the Redis cache, the CDN cache, the WAF cache, mongo cache, etc... if you don't understand how your website is cached. Stop and figure it out. Remember to also document this as a step!
9. Document the regression pack 🥴
Whenever you release you need to test. Ideally, in a perfect world... all testing is done via automation and your tests coverage is so high, no bugs ever make it to production. In reality, most projects I've worked on have good test coverage but still need some manual checks. Get your tester to go through the site. They should have some test scripts that cover the most important journey on the website. Journeys like someone buying something etc... Ensure that this regression testing process is documented and that everyone on the team can access it. If the tester is not around, not being able to test the site is a massive risk. Just because it is the QAs job to run the scripts, that doesn't mean the developers should be ignorant to it!
10. End communications 🔴
After a release has been made and signed off, do not forget to tell everyone! Anyone you told at the start of your deployment, needs to be informed when the process has ended. It doesn't matter if it's a successful deployment, or, not, as long as you have told people, your responsibility is over!
I hope the steps in this article has given you some inspiration on how you can level up your release process. If you follow these steps I can guarantee you will have less stress when deploying than you do now. Over the years, I've easily done over 750 live releases. There is definitely a common theme when it comes to releases. The teams who follow the basics religiously have more successful releases than the companies that wing it. Most of the steps I've listed above will sound obvious, however, most companies tend to forget to follow them and then act surprised when things fall over.
Having a living document that lists every step in your release process will instantly give you smoother releases. After documenting everything, if you notice that there are too many steps to list, then you know your process is wrong. Start automating things. If you continue to improve you will have pain free releases eventually! Happy Coding 🤘
