workshop,

Michael

DevOps automation

DevOps automation

related articles

Cloud

related documents

qrsolve-flyer qrsolve-whitepaper

tags

devops

Share

Michael
Written by Michael
Infrastructure & Automation master

For our team automation is a starting point, we always design automated development processes. Let’s find out how we do this.

When we take care of the automation in the first stages of the project, we can save a lot of time and money in the following stages. Our goal is to automate everything and don’t use human resources for repeatable and expensive activities like deploying, creating infrastructure, testing, and checking our software. Nowadays, we have many great tools and techniques to achieve this goal of providing and maintaining software with possibly the best quality.

Source code projects

We used to use GitLab as a source code repository based on Git technology. Starting a new project we prepare the right projects and group structure depending on project requirements.

Projects structure

projects structure
Figure 1. Projects

The structure of our example project looks like the below:

  • device - Sub group for IoT projects, deployed to electronic boards

  • internal - Sub group for UX, project informations, documentations

  • dashboard - Sub group for dashboard application, UI part, microservices, and infrastructures creation

  • public - Sub group for public resources like a web page and infrastructure creation for this area

Of course, the naming convention is very important and we need to remember that bad names, in the beginning, can have a big impact later on understanding by other team members and customers. A good practice is to prepare and maintain example projects to speed up new project creation by forking base projects.

projects
Figure 2. Base project structure

Project sources

Project structure needs to be compliant with the standard to be easily extended by another team or developer.

project structure
Figure 3. Project sources

Every project has to have its README documentation and CHANGELOG file to easily identify how to run and maintain source code. By keeping our projects always stable and well documented, we can easily avoid misunderstandings.

project readme
Figure 4. Project readme

Pipelines

This is definitely the heart of our project’s ecosystem. Well designed and used can be a solution to keep all projects and systems stable, secure, and maintenance-free. On the market there are various CI/CD systems, we used to use GitLab CI/CD in our projects.

Development flow

Every framework is different, we need to provide a suitable continuous integration process for every project and application.

Java microservice pipeline

For example, our Java microservice pipeline can look like the below:

service pipeline
Figure 5. Service pipeline
  • Build - compile Java source code, make JUnit tests and prepare JAR file

  • Test - module tests with TestContainers framework

  • Deploy - prepare Docker image, deploy docker to AWS EKS registry and deploy new ECS TaskDefinition to ECS container

We can easily extend our pipeline and add other stages like:

  • Anchor security scanner verification

  • Sonar Quality Gate verification

Infrastructure pipeline

Another pipeline example is Terraform project automation based on CI/CD scripts.

infrastructure pipeline
Figure 6. Infrastructure pipeline
  • Plan - create a plan for Terraform scripts. Comparing with current infrastructure and prepare ready to run Terraform plan

  • Apply - make changes for AWS infrastructure

  • Start and stop - optional step to start and stop our test environment in the cloud

Test results

Every pipeline can export test results and provide this for users and simply show them in our merge requests.

service tests
Figure 7. Test results

Group variables

To not repeat variables between projects is a good idea to gather all variables under group.

group variables
Figure 8. Group variables

Cloud Automation

Cloud automation is strictly related to the Infrastructure as a Code pattern where we can reflect our whole architecture on the source code. We used to use Terraform framework to onboard complete environments to the cloud. Terraform scripts can be integrated with CI/CD scripts and infrastructure creation can be also an automated process without any human activities.

GitFlow

GitFlow is a current standard for developing and maintaining projects by development teams. Everybody uses it to plan and release software without conflicts and problems due to concurrency.

gitflow
Figure 9. Git Flow

This methodology can be used to drive our pipeline flow. The most common use cases:

  • develop branch - flow dedicated to compile and deploy our changes merged from feature branches

  • feature branch - flow dedicated to compile developer changes ad hoc

  • release branch - flow dedicated to provide our build-to-stage environment

  • master branch - flow dedicated to production environment

  • merge requests - flow dedicated to compile, check and test developer changes from feature branches

We can easily design our own behavior for the above cases.

Summary

CI/CD is a very large topic, as always a good idea is to stay close to the standards, track Technology Radars, and practice ideas on real projects. Because of current architecture trends like microservices and cloud - DevOps is a key skill for a good working development team.