How to achieve high scalability of services infrastructure and CI/CD processes.

One of our clients, Liga Digital, an ERP software company, was struggling with achieving high scalability, reliability, and observability in their products. 

Although the company has a great engineering team, this was something they couldn’t solve on their own. In order to improve user experience and achieve high scalability in their software development products, they asked our engineering team to work on some system improvements. 

Their previous deployment and orchestration methods were based on software combined with customized solutions such as Docker Swarm, and Docker Compose. The build and deployment process was unreproducible,  causing troubleshooting problems. They used different infrastructures for testing and production environments, which made it difficult to reconcile changes between x and y. It also didn’t make much sense to assign containers to the machine, as it was best to have everything generated dynamically as far as the architecture allowed.

• very limited observability means, 
• no secret management process,
• limited scaling capabilities,
• containers were assigned to particular machines,
• lack of high availability or autoscaling possibilities in services

The point of the project was to transform “in-house” VMware architecture into a cloud solution, taking into account several crucial aspects of the design:

• high scalability
• reliability
• observability
• costs

The technologies and libraries we used: 

• AWS
• Kubernetes 
• Docker,
• Ansible, 
• Terraform,
• BitBucket, 
• Concourse, 
• Packer, 
• ArgoCD, 
• RabbitMQ, 
• Postgres, 
• MySQL

Most importantly we gave the  infrastructure a makeover, CI/CD, and deployment orchestration, but we also introduced a number of improvements to various areas of the project: 

• A Kubernetes cluster with certain additional features:
  • Nodes autoscaling
  • ArgoCD with GitOps approach
  • Secure setup with bastions
  • OIDC authentication for the Kubernetes cluster,
  • Implementation of automatic snapshots for storage and retention,
  • Addition of automatic downscaling for the night to test environments,
  • EC2 Spot infrastructure for test and staging environments
  • Installation of  Kyverno for policy management
  • Introduction of Calico for network traffic separation,
  • Security hardening using CIS benchmark,
  • Launch of a number of reliability addons for the cluster (node draining, removing stuck pods, automated problems detection and remediation, automatic pod restart on config change, etc.)
• A standardized deployment strategy for all applications across all environments using Bitbucket. Dockerhub and ArgoCD, 
• Containerized applications that still needed it,
• A Prometheus stack for monitoring with multiple exporters and dashboards
• All of the infrastructure work was done in the Infrastructure as a Code principle using Ansible, Terraform, and Packer
• Introduction of a secrets management process using Kubernetes integration with 1Password 
• Introduction of high availability capabilities to services, where possible

The focus was to standardize solutions to the various problems we had in the project. As the foundation of the whole solution, we chose AWS as the infrastructure provider and AWS managed Kubernetes as the orchestrator. Each workload had to be Docker managed. 

We developed one parametrized deployment manifest for all of the internal applications and used the GitOps approach with ArgoCD software, which allowed us to have just one Git repository which acts as a single source of truth.

However, the project was an evolution, not a revolution. We built on the foundations the customer already had. We still used AWS, BitBucket, Datadog, 1Password, and Dockerhub. As a result, we finalized the migration away from Cloudformation into Ansible and Terraform.

After two years of this remote project run by a single developer, we managed to deliver unique services that hardly anyone else was able to solve. During the timeline of the project, we also provided regular support for day-to-day operations, production problems, developer assistance, and consultations.  In addition, we maintained the legacy systems that hadn’t yet been replaced. 

Using proper project methodology, and with experts on our side, the project was delivered in iterations, where consultants worked together with stakeholders on their current priorities. 

The priority at the beginning of any project is to seize the opportunity to lead clients to lower maintenance costs. Choosing scalable solutions also protects technology investments. The client can continue to use the same software for longer because they are designed to grow alongside each other.

In addition, in this project, the developers had a great deal of freedom in pointing out current challenges and proposing ways to deal with them, which enhanced all of the processes and positively influenced the work of the software dev team. 

 Ultimately, the client gained:

• 2x faster data analysis, 

• 20% lower  maintenance costs, 

• 100% technology debt removal,

• high scalability

• reliability

• observability

• lower costs