DevOps in general and especially in the last few years has become a big issue and an important goal in every IT company. It is a fact that so many companies are moving much faster than planned from private data centers and dedicated servers to public clouds, and one of their main goals in this transition is to ensure a suitable environment in which modern CI/CD can run smoothly.

Before we begin our CI/CD story, you may ask yourself: What is CI/CD?

Continuous integration (CI)

In modern application development, several developers work on different features of the same application. Without a plan or clear procedures, this usually leads to a lot of problems when the time comes for developers to merge their code changes into a branch. Merging changes and resolving conflicts in code can be a very painful and hard task that usually requires a lot of effort. The worst part is that the process is extremely risky and error-prone, which in most cases leads to problems.

Continuous Integration (CI) allows developers to consolidate their code changes much more frequently, sometimes daily, into a common “branch” or “trunk” of the application. Once a developer’s changes are merged, they are validated in automated app builds and different levels of automation testing ( usually unit and integration testing). This ensures that functionality has not been impaired. If the automated test detects conflicts between current and new code, CI can resolve them faster and more often, and sometimes even without developer interaction.

Continuous delivery (CD)

The process of continuous delivery (CD) requires that CI must already be part of the development pipeline. When the application builds, unit and integration tests for the CI are ready, the main goal of continuous delivery is to automatically promote the most recently tested and stable application build in a new release. In simple terms, this means that the primary goal of continuous deployment is to have application code that can be pushed out to production workloads at any time and as quickly as possible.

Content Garden vs. CI/CD

In the early stages of Content Garden, we did not put so much effort into any CI/CD processes, in fact we didn’t have them at all!  We were a small team of enthusiasts who simply wanted to develop a good and simple tool that would help our internal colleagues to make their daily work of creating content and booking campaigns easier and more efficient.

Over time, Content Garden grew more and more, and as a result our application became a robust solution that was really hard to maintain and, trust me, even hard to deploy. Really quickly we got into a situation where testing code was a painful process, new features took weeks to deploy, and simple bug fixes in some cases took up to a few weeks on the road to production. If we can say something about deployment, our reality was that with each new deployment, not only the new features but also a new set of bugs was released, and as the icing on the cake, we used to have a “manual deployment” that usually failed and could not be completed in the first shoot.

The situation described above also brings us to the following issues:

• We had a branch popularly known as the “development” branch, which we used to merge on the “master” branch before deployment, which most of the time caused merge mistakes or failures
• We only had a single unified environment in which we manually tested our application functionalities before deploying them in production.
• We did not have an isolated environment in which our unit and integration tests could be executed in an automated way
• Each deployment required the active presence of all developers, as the manual process was hell!
• Without application monitoring, we keep hearing from our users about problems in production even before we had noticed it ourselves.

Now that we knew all these facts and problems, we realized that we had to change something as soon as possible. After we started, we quickly realized that the first thing we had to do was not to introduce some fancy tool or some new popular process, but to change our culture!

We had to change our working attitude from waterfall development to a more agile DevOps approach. CI/CD was a clear direction for our future process. In the CI/CD world, there was no more room for us to hold special sessions to discuss the new version, set a schedule for deployment, or facilitate the use of new features by the entire development team. We wanted to make the introduction process so simple that nobody even noticed that the introduction was taking place and that our developers, users and stakeholders could sleep better.

To achieve this, we realized that we had to approach and cover several issues:

• Application Environments
• Automated builds
• Automated testing
• Deploy
• Application Monitoring

Application Environments

As I mentioned before, we used to have only two app environments: “development” and “production”. As a result, most of the testing was done on development machines and then at some point was manually deployed with a number of other functions into this “development” environment, where it was finally tested and approved before the production deploy. Very quickly, we realized that we needed to give our developers the ability to deploy their code at any time and at any place.

For this reason we came to the conclusion that we needed the following environments:

• Test Environment

This environment will be mainly used by developers. Their latest code changes will be immediately deployed automatically after their push to the repository.

• Staging Environment

This environment was introduced with the idea of always having the most up-to-date stable set of functions. We introduced it because we needed a place where we could successfully run all our end-to-end tests before these latest changes were advertised for the final tests performed by our manual QA testers.

• QA Environment

This environment provides a stable or so-called “pre-production” environment used only by our manual QA testers, who run their manual test cases against the application features to double-check that everything works well and that it can be deployed for production.

• Production Environment

The name explains everything, it stands for the production environment in which our application runs for our end users.

By introducing all these new stages, we have achieved that there are no more conflicts between developers and QA regarding the stage. We have also ensured that each of these environments is part of our CI/CD pipeline and that they are deployed automatically and, in some cases, manually (a click of a button is all it takes) based on our current needs.

Automated Builds

Back then, in manual deployments, we ” deployed” by manually pulling the latest marked code from our Git repository, manually running the database migrations, and so on. At the beginning of our transition, we realized that this was not good practice, so we had to find a solution that would allow us to create the specific application builds. That was the moment when we knew that we needed to get our application running via dockers. And that was basically the first step to move on and achieve our goal of creating unique automated builds of our application.

Containerizing our application opened up a whole world of possibilities to us: First, we could have unified local development environments. No more “works on my machine” statements from developers while integrating the application building process within our CI/CD pipeline. Later, we could use those builds to promote them to the various environments which I had described above, not to forget the scaling possibilities using Kubernetes.

Automated Testing

Our next big and important goal was to expand automated testing and integrate it into our workflow. We realized that automated testing had to be a part of our culture and development process. To that purpose, we decided to implement our pipeline in such a way that every time a developer pushes his code into a repository, the pipeline is automatically triggered and performs the steps required to execute all our unit and integration testing.

This allows us to identify problems in time and prevent them from being used in any of our stages during an initial shoot. We have also taken the opportunity to integrate several code coverage and code quality measurements (SonarQube), which create and store really nice reports, so we can constantly monitor and improve our code quality and test coverage. We have also established a hard rule that if one of these automated test phases fails, the entire package cannot be moved further through the pipeline until the problem is resolved.

Deploy

The code that is prepared goes through all phases before it is ready and can be used in a real production environment.
During our transition, we made sure that we did not perform the production operations 100% manually, as we had done before. Instead, we have achieved that now the deploy to production can be performed at the touch of a button and only if the specific commit has passed all the necessary stages beforehand. This ensures that the content complies with our quality and testing measures.

Application Monitoring

Throughout the transition period, we also concluded that we always had a problem tracking and checking what was actually happening in our production environments. There was no centralized logging, monitoring, or alerting, which meant that after deployment it was often difficult to tell if things were working or not. The only thing we constantly checked manually was to track our application logs directly on the server. As our application evolved and became a more robust solution, we needed a “smarter way” to monitor it.

To achieve this, we started to integrate and use one of the most popular Cloud Monitoring SaaS. It helped us not only to see the specific metrics of our application, but also to track and monitor our entire infrastructure and identify or predict bottlenecks and performance problems in the shortest possible time. We also took the opportunity to set up alerts for specific events that are important to us and affect our application and infrastructure, so that when they occur, we are immediately notified via push notifications to our desired Slack channel.

Conclusion

To be honest with you, it was not an easy task for our team to go through all the transitions mentioned above. It also took us a lot of time to modify not only some tools and procedures, but also the way we think about the process of software development and deployment. During this time, we have been able to identify many mistakes we made in the past, unfortunately in a harder way, but the benefits we have gained from this transition are priceless for us today.

This does not mean that the process is over. There is still much that we want to achieve and improve in our current CI/CD process, and we are constantly working to make it better over time. I hope that this article can encourage everyone who wants to start or is planning a CI/CD transition. Me and my team are happy to share our experiences or answer any questions you may have about our experiences and this topic in general.