For our team automation is a starting point, we always design automated development process. Let’s find out how we do this.
When we take care about the automation on the first stages of the project we can simply save a lot of time and money on the nest stages of the project. Our goal is to automate everything and don’t use human resources for repeatable and expensive activities like deploying, creating infrastructure, testing and checking our software. Nowadays we have many of great tools and techniques to achieve this goal providing and maintaining software with possibly the best quality.
Source code projects
We used to use GitLab as a source code repository based on Git technology. Starting new project we are preparing the right projects and groups structure depending on project requirements.
Projects structure
Our example project has structure like below:
-
device - Sub group for IoT projects, deployed to electronic boards
-
internal - Sub group for UX, project informations, documentations
-
dashboard - Sub group for dashboard application, UI part, microservices and infrastructures creation
-
public - Sub group for public resources like web page and infrastructure creation for this area
Of course the naming convention is very important and we need to remember that bad names on the beginning can has big impact later to understanding by other team members and customers. Good practice is to prepare and maintain example projects to speed up new project creation by forking base projects.
Project sources
Project structure needs to be compliant with the standard to be easily extended by another team or developer.
Every project has to have own README documentation and CHANGELOG file to easily identify how to run and maintain source code. Keeping our projects always stable and well documented we can easily avoid misunderstanding and mess.
Pipelines
This is definitely a heart of our projects ecosystem. Well designed and used can be a solution to keep all projects and systems stable, secure and maintenance-free. On the market there are various CI/CD systems, we used to use GitLab CI/CD in our projects.
Development flow
Every framework is different, we need to provide suitable continuous integration process for every project an application.
Java microservice pipeline
For example our Java microservice pipeline can look like below:
-
Build - compile Java source code, make JUnit tests and prepare JAR file
-
Test - module tests with TestContainers framework
-
Deploy - prepare Docker image, deploy docker to AWS EKS registry and deploy new ECS TaskDefinition to ECS container
We can easily extend our pipeline and add another stages like
-
Anchor security scanner verification
-
Sonar Quality Gate verification
Infrastructure pipeline
Another pipeline example is Terraform project automation based on CI/CD scripts.
-
Plan - create a plan for Terraform scripts. Comparing with current infrastructure and prepare ready to run Terraform plan
-
Apply - make changes for AWS infrastructure
-
Start and stop - optional step to start ond stop our test environment in the cloud
Test results
Every pipeline can export test results and provide this for users and simply show in our merge requests.
Group variables
To not repeat variables between projects is a good idea to gather all variables under group.
Cloud Automation
Cloud automation is strictly related with Infrastructure as Code pattern where we can reflect our whole architecture on source code. Good used to use Terraform framework to onboard complete environments to the cloud. Terraform scripts can te integrated with CI/CD scripts and infrastructure creation can be also automated process without human activities.
GitFlow
GitFlow is a current standard for developing and maintaining projects by development teams. Everybody uses it to plan and release software without conflicts and problems due to concurrency.
This methodology can be used to drive our pipeline flow. The most common use cases:
-
develop branch - flow dedicated to compile and deploy our changes merged from feature branches
-
feature branch - flow dedicated to compile developer changes ad hoc
-
release branch - flow dedicated to provide our build to stage environment
-
master branch - flow dedicated to production environment
-
merge requests - flow dedicated to compile, check and test developer changes from feature branches
We can easily design our own behavior for the above cases.
Useful information
Summary
CI/CD is a very large topic, as always a good idea is to stay close the standards, track Technology Radars and practice ideas on real projects. Because of current architecture trends like microservices and cloud - DevOps is a key skill for good working development team.
