The One Big-Company Practice That Benefits Everyone: Continuous Delivery

There has been a lot of talk over the last year that your company isn’t Google, isn’t Amazon, and isn’t Meta, so you probably aren’t going to need the solutions that they need, because their scale is entirely different from yours.

Microservices and Serverless get a lot of scrutiny as architectures and approaches that were adopted by smaller firms and ended up not delivering on their promise. GraphQL is another technology that solves very specific problems that your company might not have.

Many of these technologies are trying to solve one of two problems: how do I get the most out of my servers, and how do I loosely couple different parts of the business to make it easier for disparate teams to coordinate and collaborate. The goal of this loose coupling is faster delivery, with less slack time as teams wait for other teams.

But if you don’t have hundreds of engineers all working on essentially the same product, and you don’t have hundreds of servers that are mostly standing idle, you don’t have the problem that these technologies are trying to solve.

In fact, adopting them might force you to increase delivery time and server costs, since you are adopting a complex solution that has to be maintained forever — even if you suddenly have a lot fewer people to do so.

After working with dozens of teams and consulting with many companies and tech leaders, there is one thing that these big companies do that I think can benefit almost everyone — and that is Continuous Delivery.

Often combined with Continuous Integration as CI/CD, Continuous Delivery means you can securely deploy your code at any time, with very little friction and ideally, very little downtime.

Every project and company can benefit from being able to smoothly and seamlessly deploy their code to production. Long gone are the days of FTP’ing code up to the server from your personal machine, or logging into the production server and doing a git pull.

There are so many ways now for any project to have a CD pipeline that there really isn’t an excuse for even the smallest of projects not to implement it.

But I’m writing this because I keep encountering projects where there is no CD — even large and complex projects — and these projects are missing out on major benefits.

Benefits of Continuous Delivery

It might seem like too much work to set up a CD pipeline when you are first working on a project; after all, you have so many other things to focus on. But there are so many advantages, regardless of your project scale, that I believe it is absolutely one of the first things you should set up.

The key benefits are:

• Release without fear — If you can release with extremely high confidence that it will work, you will release more often.
• Enable anyone to securely deploy — If only a few people have the power to do the release, eventually you won’t be able to release.
• Make tiny changes the norm — If you can make a 3–4 line change and deploy it, your overall site stability will increase, since you will be able to test that change much more completely.
• Fail fast, fix fast — If a change does break something, you can roll it back or fix it quickly without having to roll back dozens of changes.

Release Without Fear

A project I work on had over 30 steps to get a build to production. While I could do all those steps myself, it still took 30+ minutes, and every time I made a slight error, or a step wouldn’t quite work, I’d have to repeat something.

Still, it didn’t feel too bad — after all, I could still release whenever I wanted. Compared to my telco days of four releases a year, it was a huge increase in delivery speed.

I slowly started automating the steps using GitHub Actions, and eventually I got to the point where only the last few steps were manual. Again, I felt like it was a huge step forward, but I still made errors almost every time, as I did it irregularly enough to forget some nuances, and it still required multiple logins and cut-and-paste of complex paths.

I left it that way for years — just mostly automated. And I came to dread every release, because even with my checklist, it seemed like something would go wrong. And I had almost no indication when things went right, so I would have to go confirm everything downloaded correctly and worked on multiple devices every time I released. Each release took at least an hour of my dedicated time.

So I released fewer and fewer changes, and eventually didn’t release at all for a while. The fear of doing releases was making me dread working on the project, and it took me a long time to realize it. To be fair, those last steps were quite hard to automate at the time, and it wasn’t until a while later that it became a no-brainer to build it.

That’s when I realized that the deployment process was stopping me from working on the project at all. Especially since I was the only one on the project with the keys to all the places that had to be touched for deployment, so I was the only release manager as well. I never liked being the bottleneck, but many of the platforms involved either didn’t have group-based permissions, or charged a lot more for that feature.

Lesson learned — if you can’t release without fear, you won’t release as often as you should to be competitive. And you need to enable anyone to securely deploy your app to avoid bottlenecks.

Enable Anyone to Securely Deploy

At one point in my career, I had invented a crazy-complex continuous delivery platform for a Fortune 50 telco’s eCommerce system that enabled us to update the shopping and checkout flows multiple times a day.

I say "invented" because this wasn’t just a Jenkins setup or shell scripts — this involved a huge enterprise-grade change management platform, dozens of new servers and network connections, whole-cloth invented code libraries, complex Apache configurations, and a major re-architecture of the massive Java app to enable this to happen. It was a whole application in and of itself, and would eventually power the entire web presence.

The entire thing was a workaround to avoid actually deploying the Java code, since that was only allowed to happen four times a year. And if you ever worked with mainframes, CORBA, and EJBs, you know why that would be. The opposite of loose coupling.

But the key feature was that it was entirely tracked and secure — you had to have a login to the change management platform, create a ticket, have it blessed by another person, and then "action" it to production. The CD system would deploy the attached changes to production and be instantly available to the app with no downtime.

Anyone with permissions could do this. It was not easy to work with the change management system, but developers worked with it every day, so it felt right at home for them. It meant that when we were doing emergency deployments at 3 AM, any employee developer could get those changes out. And that saved our bacon when people were out on vacation, sick, or left the company. And the version control system was the first permission that got taken away when you left, so it kept security crisp.

I did get into quite a fight with one of the security people about a potential vulnerability, but I eventually had him write up all the steps that someone would have to take, and how likely they were to happen, and what the alternatives were. It was a full page of things in a row that would have to be breached, and he had no alternative. With no alternative and the C-suite refusing to go back to four deployments a year, he eventually backed off.

Lesson learned — you can make continuous deployment secure, and doing so lets you actually take a vacation once in a while, and keeps the security people happy.

Make Tiny Changes the Norm

When I was only deploying a few times a year, there were tens of thousands of changes in each of the releases, and all of them had to not only be tested per-system, but tested while integrated with everyone else’s changes.

With so many changes, a lot of bugs appeared at the last minute, and while you might think that would lead to a lot of delays and rollbacks, that never happened. Because every system was dependent on every other system’s changes, no one could hold their release or roll it back. So typically the onshore team would work long hours for the last few weeks to get everything working in time for the release day.

Once I rolled out the continuous delivery system, not only could we release hundreds of tiny changes a month (as long as the Java code was not changed), but we could stub out imperfect UI features and deploy them after the release.

I can’t tell you how freeing it is to just fix something and release it. It makes testing infinitely faster and easier, as you are only validating one small thing at a time. You can also use feature flags to deploy more complex features and turn them on or off, or for certain groups of users.

The confidence of deploying a release that you know is working is a big lifestyle upgrade from releasing and praying. We’d often have situations where everyone would go on vacation right after the release, having worked 80 hours a week leading up to it, only to have serious bugs appear in the week after release.

With continuous delivery, we could turn things off in the UI, push that out, and wait for the senior engineers to return. Rarely did we have to deploy Java patches to match, but most things could be fixed via the parts of the app controlled by the CD system.

Lesson learned — if you can make tiny changes and deploy them without effort, you will get more value out to your stakeholders faster, reduce pressure on big releases, and be able to fix bugs (or turn off paths to bugs) immediately. Even for small apps, this is a big win.

Fail Fast, Fix Fast

The last benefit I wanted to cover is the ability to put something out quickly without worrying that it will break and be broken forever.

I had a project where the team liked to experiment with new widgets on pages. Building the full capability of each widget in such a way that they could edit it in all the ways they wanted to took some time, but a simple version of the widget was easy.

Using continuous delivery, I was able to get a simple version of the widget (well-instrumented with tracking telemetry) out to the site in a few hours. They were able to see if it had any response at all while I worked on a fancier version.

Many times, by the time I was ready to start on the fancier version, we knew whether it was worth investing more in or not. We also often discovered some core flaw in the assumptions and pulled it down immediately. If it took even a few hours to do a release, this would not have been practical.

A drawback to this approach was that some analysts realized we could get fixes out very quickly, and so didn’t do much testing before release, preferring to let us know about "urgent production issues." In the end, because of continuous delivery, this wasn’t a big deal, but it was a cultural thing we had to handle. Just because we can fix it quickly doesn’t mean that is the preferred approach to having it not be broken at all.

Lesson learned — if you can deploy cheaply, then testing new features is also cheap, and the cost of fixing bugs drops dramatically.

Final Thoughts

While building my first continuous delivery pipeline was a matter of necessity, since then I’ve insisted that we build out CD as a first step in any new project past the prototype stage. If we can’t get changes out rapidly and cleanly on a regular basis, with strong confidence, then it makes everything harder. It eventually changes the culture to one that bundles up changes and bug fixes to avoid doing releases.

Every deployed project should have continuous delivery these days, and it has never been easier.

So if you’ve been holding off on automating your deployments, this is the time to do it — you will be surprised how it changes your relationship to software.

Cheers!

Tim