DevOps architecture makover

Primotus has created a Business Process Management (BPM) web/mobile platform that allows software analysts to build entire workflow systems with little to no custom programming. The platform is flexible enough to be used in a myriad of industry verticals. Its goal is to significantly reduce the cost and increase the speed of system implementations.

And as you might expect, the infrastructure and processes gradually became more and more complex and time-consuming to handle over time.

To mention just a few of the needs:

▸ lower AWS bills,

▸ eliminate environment differences,

▸ enhance monitoring and logging,

▸ add enterprise-level security upgrades,

▸ add containerization,

▸ better self-healing mechanisms,

▸ improved configuration management.

The team simply didn’t have dedicated DevOps resources resulting in the needs outlined above. This is where the Scalac DevOps Team came in.

Cost optimization

We made an extensive review of the customer’s AWS accounts and found multiple areas for improvement:

▸ Better utilized resources resulting in reductions of 23% for all EC2, 60% for all RDS databases, 48% for all EBS volumes, and 30% for all Elastic IPs.

▸ Lowered AWS commercial support subscription

▸ Added certain Reserved Instances and utilized Spot instances where possible (mostly non-production).

▸ Added prepaid Savings Plan to all environments.

Containerization

We containerized all of the services and processes used across the organization. Using Docker gives us a fantastic possibility to create software and processes that are fully independent of the execution environment and are self-contained, easy migratable and reproducible. This not only applies to the final step in the workflow (running the application itself), but also to all of the steps along the way, such as building and packaging the application code or applying infrastructure changes.

GitOps

Thanks to our work, about 90% of all the actions in the system are automated or managed by a source control. Every manual modification of the managed resources will be reverted automatically. The desired state is described using manifests in Git, which acts as a single source of truth. All permanent changes should be put into code.

High Availability

Our solution is meant to be fully robust and requires no interventions to recover from failures. All of the services use health checks to assess the availability of the services and to act based on this data. We used state-of-the-art monitoring and reporting methods to get a deep-down look into the services.

Technical solution

To solve those problems, we implemented our Technical Solution, developed and dogfooded internally at Scalac. It’s a complete ecosystem of tools, code, and processes for most of your DevOps needs:

▸  Container orchestration,

▸  Configuration management,

▸  Monitoring,

▸  Logging,

▸  Infrastructure management,

▸  OS updates,

▸  Backups,

▸  Security.

Key achievements

The project ended with full success:

▸ 71% AWS cost reduction compared to the original bill,

▸ Implementation of a cutting edge, fully automated environment creation and management solution

▸ Advanced self-healing mechanism to keep environments available at all times

▸ Replacement of  an expensive NewRelic monitoring solution with an open-source Prometheus+Grafana stack,

▸ Migration to a managed Kafka service, reducing maintenance efforts,

▸ Reduction of the number of cloud environments by creating a ‘local environment’ that developers can run on their laptops.

Cooperation model

▸ 1 person,

▸  4 months,

▸ Our expert Piotr – an AWS and Kubernetes certified DevOps engineer with extensive automation and high-availability experience,

▸ Fully remote,

▸ Triweekly standup meetings,

▸ Shared Slack workspace for cooperation,

▸ Jira used for task management,

▸ Daily Jira ticket updates,

▸ 3 hour time zones overlap (CET, EST),

▸ Project management on the Scalac side,

▸ After the initial development phase, cooperation continues in the form of 10 hours a week of maintenance.