hire scala programmers

How to create an automation process using AWS, IaaS, IaaC, and CICD

hire scala programmers

Why do we need orchestration?

As Federico Garcia Lorca once said, besides black art, there is only automation and mechanization. And though for some of you, the title of this article might sound like black magic indeed, I will do my best to give you a sense of what these concepts mean.

Human nature is not as complicated as it may seem. In general, we are not keen on repeating things over and over again. Our laziness naturally bringths us closer to the automation of everything that might be automated. 

..hey, are you asleep already? No, no, no… You can’t! Because this is where the real story begins!

Infrastructure in Dark Times

When I joined the Scalac DevOps engineers crew my first assignment was to do some stocktaking and make a note of all the skeletons hidden in the closet. I was thunderstruck. We had two Jenkins instances – working simultaneously – some servers in OVH, some nodes in Microsoft Azure and the rest on a Desktop PC in HQ (also known as the Red Devil).  Almost every service was launched on screen. Apart from that, one of the biggest surprises was the mix of production and dev environments on the same servers without any segregation or divisions. One on all, all on one. 

Although we didn’t have many productive applications, every procedure still resembled open-heart surgery without an anesthetist. In addition, some of the source code was stored on GitHub and some on an internally hosted GitLab. We would have had to keep two sources up and running, and that was not an option.

After a lot of team meetings, we made a strong decision to reorganize it all – codename: kill’em all. We wanted to have everything consolidated so that management, modification, and maintenance were definitively easier than the configuration that had been there from the very beginning.

And the Phoenix rises from the ashes

For the code repositories, we chose GitHub because a lot of external tools can integrate with it like a charm, and thus the automation process becomes easier.

As we were struggling with different application setups, different configs and various versions all over the servers, Docker became the remedy we needed. With that, we were able to switch locations with the assurance that no matter what the host is, the applications will still work. 

Even though our infrastructure was not the biggest, it still required a lot of time to keep it safe and sound. It also called for a lot of manual actions and to be honest, it was a bit like the blind leading the blind across a pedestrian crossing. We had no monitoring solution and no alerting tool which would have allowed us to pinpoint where it hurts the most. Which it did badly, I can assure you, many times. Grafana and Prometheus – these services became our guides or – referring to the previous metaphor – our eyes. 


It’s all in the code

One of the main ingredients for well-working orchestrations is to have an entire infrastructure in the code (IaaC). We chose Terraform from the list of provisioning tools as it is the most developed with regard to our needs and widely supported by the open-source community. In that way, we were able to encapsulate the whole infrastructure in the code. This part of the whole process took a huge amount of time, but it was definitely worth it. I will be telling you about that in more detail a bit later.

Ansible – without this Configuration Management Tool, we would still be doing updates, changes and implementations on the servers manually. Similarly to Terraform, to choose the correct implement, we were guided by the following facts:

  • It is highly developed
  • There is a wide open-source community 
  •  It is agentless
  • There are a huge amount of modules we could use 

As with Terraform, I will elaborate on this later. 

Packer – this tool is a time-saver because it lets us create identical machine images with all of the required packages installed. Thanks to this, we could use them in Terraform right away, without installing everything from scratch.

Last but not least – Jenkins. Continuous Integration and Continuous Delivery, extremely valuable and important to the deployment orchestration. We based our CI/CD on Jenkins because we had experience with this solution before, it was suitable for our needs (plugins, modules, integrations) and we didn’t want to waste time looking for something new.    

Having a toolchain chosen, we had to make a big call – where to splice it all? 

..the cloud is all we need almost

In the last quarter of 2018, AWS dominated cloud solutions, which is why we decided to choose this provider. Having the largest number of services and resources and good dynamics when it comes to changes and updates gave us the confidence that we would not have to reconsider our decision next year if AWS were to suddenly stop developing.

Using AWS tools (Infrastructure as a Service) such as Spot Instances, Auto Scaling Groups, Launch Templates, RDS, S3, and many others, we started the migration process, so ensuring high availability and self-healing mechanisms if one or more of our basic elements were to unexpectedly be taken by evil spirits crawling from the shadows disappear.

The four members of the fellowship – Packer, Terraform, Ansible and Docker

Packer is an open-source solution for creating machine images with different providers, for example, AWS AMI. Packer is lightweight and can be configured easily. What is more, Packer allows the use of configuration management tools such as Ansible to install packages and any required software on a particular machine image during its creation. Because of this, when the image is prepared, we don’t have to worry about installing everything from scratch on the operating system. 

In our case, all the servers are based on Ubuntu 18.04 LTS and this exact Linux distribution was used in the Packer configuration as well. 

The Packer configuration includes two main blocks:

  • packer_template.json
  • Vars directory 

The template file looks like this:

<span class="token punctuation">{</span>
  <span class="token string">"variables"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span>
    <span class="token string">"home"</span><span class="token punctuation">:</span> <span class="token string">"{{env `HOME`}}"</span><span class="token punctuation">,</span>
    <span class="token string">"build_number"</span><span class="token punctuation">:</span> <span class="token string">"{{env `BUILD_NUMBER`}}"</span>
  <span class="token punctuation">}</span><span class="token punctuation">,</span>
  <span class="token string">"builders"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>
    <span class="token punctuation">{</span>
      <span class="token string">"type"</span><span class="token punctuation">:</span> <span class="token string">"amazon-ebs"</span><span class="token punctuation">,</span>
      <span class="token string">"name"</span><span class="token punctuation">:</span> <span class="token string">"{{user `hostname`}}_{{user `os_name`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"ami_name"</span><span class="token punctuation">:</span> <span class="token string">"{{user `hostname`}}_{{user `os_name`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"profile"</span><span class="token punctuation">:</span> <span class="token string">"{{user `aws_profile`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"region"</span><span class="token punctuation">:</span> <span class="token string">"{{user `region`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"source_ami_filter"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span>
        <span class="token string">"filters"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span>
          <span class="token string">"virtualization-type"</span><span class="token punctuation">:</span> <span class="token string">"hvm"</span><span class="token punctuation">,</span>
          <span class="token string">"name"</span><span class="token punctuation">:</span> <span class="token string">"{{user `image_query`}}"</span><span class="token punctuation">,</span>
          <span class="token string">"root-device-type"</span><span class="token punctuation">:</span> <span class="token string">"ebs"</span>
        <span class="token punctuation">}</span><span class="token punctuation">,</span>
        <span class="token string">"owners"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">"{{user `image_owner`}}"</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
        <span class="token string">"most_recent"</span><span class="token punctuation">:</span> <span class="token boolean">true</span>
      <span class="token punctuation">}</span><span class="token punctuation">,</span>
      <span class="token string">"instance_type"</span><span class="token punctuation">:</span> <span class="token string">"{{user `instance_type`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"iam_instance_profile"</span><span class="token punctuation">:</span> <span class="token string">"{{user `iam_instance_profile`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"ssh_username"</span><span class="token punctuation">:</span> <span class="token string">"{{user `ssh_username`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"ami_description"</span><span class="token punctuation">:</span> <span class="token string">"{{user `git_branch`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"launch_block_device_mappings"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token punctuation">{</span>
        <span class="token string">"device_name"</span><span class="token punctuation">:</span> <span class="token string">"/dev/sda1"</span><span class="token punctuation">,</span>
        <span class="token string">"volume_size"</span><span class="token punctuation">:</span> <span class="token string">"{{user `root_size`}}"</span><span class="token punctuation">,</span>
        <span class="token string">"volume_type"</span><span class="token punctuation">:</span> <span class="token string">"gp2"</span><span class="token punctuation">,</span>
        <span class="token string">"delete_on_termination"</span><span class="token punctuation">:</span> <span class="token boolean">true</span>
      <span class="token punctuation">}</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
      <span class="token string">"ami_block_device_mappings"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>
        <span class="token punctuation">{</span>
          <span class="token string">"device_name"</span><span class="token punctuation">:</span> <span class="token string">"/dev/sda1"</span><span class="token punctuation">,</span>
          <span class="token string">"volume_size"</span><span class="token punctuation">:</span> <span class="token string">"{{user `root_size`}}"</span><span class="token punctuation">,</span>
          <span class="token string">"volume_type"</span><span class="token punctuation">:</span> <span class="token string">"gp2"</span><span class="token punctuation">,</span>
          <span class="token string">"delete_on_termination"</span><span class="token punctuation">:</span> <span class="token boolean">true</span>
        <span class="token punctuation">}</span>
      <span class="token punctuation">]</span><span class="token punctuation">,</span>
      <span class="token string">"tags"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span>
        <span class="token string">"Name"</span><span class="token punctuation">:</span> <span class="token string">"packer_{{user `hostname`}}_{{user `os_name`}}"</span><span class="token punctuation">,</span>
        <span class="token string">"created_by"</span><span class="token punctuation">:</span> <span class="token string">"packer"</span><span class="token punctuation">,</span>
        <span class="token string">"environment"</span><span class="token punctuation">:</span> <span class="token string">"{{user `environment`}}"</span>
      <span class="token punctuation">}</span><span class="token punctuation">,</span>
      <span class="token string">"run_tags"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span>
        <span class="token string">"Name"</span><span class="token punctuation">:</span> <span class="token string">"packer_{{user `hostname`}}_{{user `os_name`}}"</span><span class="token punctuation">,</span>
        <span class="token string">"created_by"</span><span class="token punctuation">:</span> <span class="token string">"packer"</span><span class="token punctuation">,</span>
        <span class="token string">"environment"</span><span class="token punctuation">:</span> <span class="token string">"{{user `environment`}}"</span>
      <span class="token punctuation">}</span><span class="token punctuation">,</span>
      <span class="token string">"force_deregister"</span><span class="token punctuation">:</span> <span class="token string">"true"</span><span class="token punctuation">,</span>
      <span class="token string">"force_delete_snapshot"</span><span class="token punctuation">:</span> <span class="token string">"true"</span><span class="token punctuation">,</span>
      <span class="token string">"spot_price"</span><span class="token punctuation">:</span> <span class="token string">"auto"</span><span class="token punctuation">,</span>
      <span class="token string">"spot_price_auto_product"</span><span class="token punctuation">:</span> <span class="token string">"Linux/UNIX (Amazon VPC)"</span><span class="token punctuation">,</span>
      <span class="token string">"security_group_id"</span><span class="token punctuation">:</span> <span class="token string">"{{user `security_group_id`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"subnet_id"</span><span class="token punctuation">:</span> <span class="token string">"{{user `subnet_id`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"ssh_keypair_name"</span><span class="token punctuation">:</span> <span class="token string">"{{user `ssh_keypair_name`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"ssh_private_key_file"</span><span class="token punctuation">:</span> <span class="token string">"{{ user `home` }}/.ssh/{{user      `ssh_keypair_name`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"associate_public_ip_address"</span><span class="token punctuation">:</span> <span class="token boolean">true</span>
    <span class="token punctuation">}</span>
  <span class="token punctuation">]</span><span class="token punctuation">,</span>
  <span class="token string">"provisioners"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>
    <span class="token punctuation">{</span>
      <span class="token string">"type"</span><span class="token punctuation">:</span> <span class="token string">"shell"</span><span class="token punctuation">,</span>
      <span class="token string">"inline"</span><span class="token punctuation">:</span> <span class="token string">"{{user `install_ansible_script`}}"</span>
    <span class="token punctuation">}</span><span class="token punctuation">,</span>
    <span class="token punctuation">{</span>
      <span class="token string">"type"</span><span class="token punctuation">:</span> <span class="token string">"ansible-local"</span><span class="token punctuation">,</span>
      <span class="token string">"playbook_dir"</span><span class="token punctuation">:</span> <span class="token string">"."</span><span class="token punctuation">,</span>
      <span class="token string">"clean_staging_directory"</span><span class="token punctuation">:</span> <span class="token string">"true"</span><span class="token punctuation">,</span>
      <span class="token string">"playbook_file"</span><span class="token punctuation">:</span> <span class="token string">"{{user `ansible_playbook`}}"</span><span class="token punctuation">,</span>
      <span class="token string">"inventory_file"</span><span class="token punctuation">:</span> <span class="token string">"{{user `ansible_inventory`}}"</span><span class="token punctuation">,</span>
    <span class="token punctuation">}</span>
  <span class="token punctuation">]</span>
<span class="token punctuation">}</span>
JavaScript


The basic values are mentioned in common.json in the Packer vars directory. So as not to send you to sleep, this file contains all the necessary parameters for Packer to work – plus some extra. 

This file defines an AWS profile, the root size of the EBS volume, region, instance type, and a bit more:

<span class="token punctuation">{</span>
  <span class="token string">"aws_profile"</span><span class="token punctuation">:</span> <span class="token string">"packer-profile"</span><span class="token punctuation">,</span>
  <span class="token string">"root_size"</span><span class="token punctuation">:</span> <span class="token string">"10"</span><span class="token punctuation">,</span>
  <span class="token string">"region"</span><span class="token punctuation">:</span> <span class="token string">"eu-central-1"</span><span class="token punctuation">,</span>
  <span class="token string">"instance_type"</span><span class="token punctuation">:</span> <span class="token string">"t3.medium"</span><span class="token punctuation">,</span>
  <span class="token string">"iam_instance_profile"</span><span class="token punctuation">:</span> <span class="token string">"ci"</span><span class="token punctuation">,</span>
  <span class="token string">"ssh_keypair_name"</span><span class="token punctuation">:</span> <span class="token string">"ssh-key"</span><span class="token punctuation">,</span>
  <span class="token string">"security_group_id"</span><span class="token punctuation">:</span> <span class="token string">"sg-0459xxxx"</span><span class="token punctuation">,</span>
  <span class="token string">"subnet_id"</span><span class="token punctuation">:</span> <span class="token string">"subnet-0797xxxxx"</span><span class="token punctuation">,</span>
  <span class="token string">"ansible_inventory"</span><span class="token punctuation">:</span> <span class="token string">"packer_inventory.ini"</span>
<span class="token punctuation">}</span>
JavaScript

As we can see, besides the basic inputs (according to the Packer docs), there are also ones regarding Spot instances because each time Packer builds an image, it has to create an EC2 instance to configure the required components and execute the Ansible roles needed for any particular image. We used Spot instances because we wanted to cut the costs of using EC2 instances in general billings. When Packer finishes, the AWS AMI is created and the Spot Instance is terminated. 

Another extra part is at the end of this template file – Ansible. As I mentioned above, Packer allows us to use a Configuration Management Tool and this is where Ansible comes into play. 

For each host/server, we have defined particular Ansible roles to be applied to the image. The hosts that the roles are applied to are described in the inventory_file, which in this case is packer_inventory.ini, and for example look like this:

xwiki  ansible_host<span class="token operator">=</span><span class="token number">127.0</span><span class="token punctuation">.</span><span class="token number">0.1</span> ansible_python_interpreter<span class="token operator">=</span><span class="token operator">/</span>usr<span class="token operator">/</span>bin<span class="token operator">/</span>python3
JavaScript

In Packer vars, we have two modes: build and ci. For the purposes of building machine images , the build mode is used. This mode references the Ansible file with details about the specific roles to be applied, for example xwiki host. Ansible_host is set to localhost because of the “ansible-local” type (mentioned in the code snippet above). Of course, a remote address is possible as well. 

# ROLES FOR XWIKI
<span class="token operator">-</span> hosts<span class="token punctuation">:</span> xwiki
  any_errors_fatal<span class="token punctuation">:</span> <span class="token boolean">true</span>
  roles<span class="token punctuation">:</span>
    <span class="token operator">-</span> configure_os
    <span class="token operator">-</span> os<span class="token operator">-</span>update
    <span class="token operator">-</span> base<span class="token operator">-</span>packages
    <span class="token operator">-</span> docker
    <span class="token operator">-</span> ssh
    <span class="token operator">-</span> fail2ban
    <span class="token operator">-</span> aws<span class="token operator">-</span>cli
    <span class="token operator">-</span> xwiki
  become<span class="token punctuation">:</span> yes
JavaScript

The last part regarding Packer is a base image definition (ubuntu-18.04.json), and it goes as follows:So each time an image for Xwiki is built, Packer applies for the above roles on the AWS AMI. Thanks to this, when a new server with Xwiki is spawned (new EC2), it contains all of the necessary configurations, settings, and files. 

<span class="token punctuation">{</span>
  <span class="token string">"os_name"</span><span class="token punctuation">:</span> <span class="token string">"ubuntu18.04"</span><span class="token punctuation">,</span>
  <span class="token string">"image_query"</span><span class="token punctuation">:</span> <span class="token string">"ubuntu/images/*ubuntu-bionic-18.04-amd64-server-*"</span><span class="token punctuation">,</span>
  <span class="token string">"image_owner"</span><span class="token punctuation">:</span> <span class="token string">"099720109477"</span><span class="token punctuation">,</span>
  <span class="token string">"ssh_username"</span><span class="token punctuation">:</span> <span class="token string">"ubuntu"</span><span class="token punctuation">,</span>
  <span class="token string">"install_ansible_script"</span><span class="token punctuation">:</span> <span class="token string">"until [ -f /var/lib/cloud/instance/boot-finished ]; do sleep 1; done,sudo DEBIAN_FRONTEND=noninteractive apt-get update,sudo DEBIAN_FRONTEND=noninteractive apt-get -y install python3 python3-setuptools python3-dev python3-pip; sudo ln -s /usr/bin/python3 /usr/bin/python; sudo ln -s /usr/bin/pip3 /usr/bin/pip; sudo pip install ansible==2.8.3"</span>
<span class="token punctuation">}</span>
JavaScript

Install_ansible_script – this parameter defines instructions to be executed on this image before any others. We are using python3 as a default executor, which has to be installed and set as default by creating symlinks to the proper binary. Also, as we wanted to take advantage of Ansible, this had to be added because it was not a pre-installed software on OS. We base AWS AMIs on Ubuntu 18.04 and during the building process, Packer uses this particular image version mentioned in the image_query. This value can be found in the AWS EC2 section once a new instance has been spawned manually. 

Finally, to build an AWS AMI for Xwiki, we have to execute (with respect to the proper location of these files): 

packer build packer_template<span class="token punctuation">.</span>json common<span class="token punctuation">.</span>json os<span class="token operator">/</span>ubuntu<span class="token number">-18.04</span><span class="token punctuation">.</span>json mode<span class="token operator">/</span>build<span class="token punctuation">.</span>json host<span class="token operator">/</span>xwiki<span class="token punctuation">.</span>json
JavaScript
xwiki<span class="token punctuation">.</span>json<span class="token punctuation">:</span>
<span class="token punctuation">{</span>
  <span class="token string">"hostname"</span><span class="token punctuation">:</span> <span class="token string">"xwiki"</span><span class="token punctuation">,</span>
  <span class="token string">"environment"</span><span class="token punctuation">:</span> <span class="token string">"dev"</span>
<span class="token punctuation">}</span>
JavaScript
build<span class="token punctuation">.</span>json<span class="token punctuation">:</span>
<span class="token punctuation">{</span>
  <span class="token string">"ansible_playbook"</span><span class="token punctuation">:</span> <span class="token string">"internal_hosts.yml"</span><span class="token punctuation">,</span>
  <span class="token string">"ci_mode"</span><span class="token punctuation">:</span> <span class="token string">"True"</span>
<span class="token punctuation">}</span>
JavaScript

..please do not faint now, do not fly away to /dev/null with your brain.

..yet more amazing things to be discovered.

..overcome your doubts and be rewarded with great knowledge.

Terraform is a tool for creating, changing, and managing infrastructure defined in configuration files. It supports and works with well-known Cloud providers such as AWS, Azure or Google Cloud Platform. 

After every action in configs, Terraform can show what will change in the infrastructure and is then able to execute a plan to reach an appropriate state. 

When a small modification is made, Terraform will do only that, without rebuilding everything from scratch. This is why Terraform is based on state files. 

Terraform has well-described documentation with examples and code snippets, which can easily be adjusted to individual needs. 

As we decided to choose AWS as a major Cloud provider, we started to migrate the existing resources (created manually) such as IAM, S3 Buckets, VPC and Route53, etc. to the Terraform configuration files using the terraform import command. 

But this was just the tip of the iceberg.  With a lot of resources, like all the servers spread across hosting providers, Auto Scaling Groups, Launch Templates, EBS, ALB and more, we had to develop from 0 state to have our infrastructure as a code. 

All of this gave us the mobility to use the Terraform code from wherever we wanted, using laptops or Jenkins. Even if one or more components were to be taken by evil forces, we would be able to recreate them easily and flawlessly. 

This process took over six months and is actually still ongoing but now we can just add new bricks to this wall, without worrying that it is about to fall on our heads. 

..done talking, showtime!

This article is for an audience that already has some experience with the aforementioned tools, so I will not present basic stuff such as AWS CLI configuration or how to install Terraform.

When I previously described Packer, I based all of the examples on Xwiki. So let’s stick to that in this part as well. 

In this paragraph, my plan is to show you how we built an infrastructure using the IaaC concept and how it is a part of the entire orchestration process. 

During brainstorms within the DevOps team, one of the conclusions we came to was to ensure High Availability and Self-healing mechanisms for all Scalac’s applications. 

AWS comes with such solutions natively in the EC2 section and they are called:

  • Launch Template
  • Auto Scaling Group

The configuration for the first regarding Xwiki is as follows: 

data <span class="token string">"aws_ami"</span> <span class="token string">"ubuntu"</span> <span class="token punctuation">{</span>
  most_recent <span class="token operator">=</span> <span class="token boolean">true</span>

  filter <span class="token punctuation">{</span>
    name   <span class="token operator">=</span> <span class="token string">"name"</span>
    values <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">"xwiki_ubuntu18.04"</span><span class="token punctuation">]</span>
  <span class="token punctuation">}</span>

  filter <span class="token punctuation">{</span>
    name   <span class="token operator">=</span> <span class="token string">"virtualization-type"</span>
    values <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">"hvm"</span><span class="token punctuation">]</span>
  <span class="token punctuation">}</span>

  filter <span class="token punctuation">{</span>
    name   <span class="token operator">=</span> <span class="token string">"root-device-type"</span>
    values <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">"ebs"</span><span class="token punctuation">]</span>
  <span class="token punctuation">}</span>

  owners <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">"self"</span><span class="token punctuation">]</span>
<span class="token punctuation">}</span>

resource <span class="token string">"aws_launch_template"</span> <span class="token string">"xwiki-lt"</span> <span class="token punctuation">{</span>
  name_prefix <span class="token operator">=</span> <span class="token string">"lt-xwiki"</span>
  image_id    <span class="token operator">=</span> data<span class="token punctuation">.</span>aws_ami<span class="token punctuation">.</span>ubuntu<span class="token punctuation">.</span>id

  block_device_mappings <span class="token punctuation">{</span>
    device_name <span class="token operator">=</span> <span class="token string">"/dev/sda1"</span>

    ebs <span class="token punctuation">{</span>
      volume_size <span class="token operator">=</span> <span class="token number">50</span>
      volume_type <span class="token operator">=</span> <span class="token string">"gp2"</span>
    <span class="token punctuation">}</span>
  <span class="token punctuation">}</span>

  instance_type <span class="token operator">=</span> <span class="token string">"t3.small"</span>
  key_name      <span class="token operator">=</span> data<span class="token punctuation">.</span>terraform_remote_state<span class="token punctuation">.</span>global_variables<span class="token punctuation">.</span>outputs<span class="token punctuation">.</span>keypair<span class="token operator">-</span>scalac
  user_data     <span class="token operator">=</span> <span class="token function">base64encode</span><span class="token punctuation">(</span>data<span class="token punctuation">.</span>template_file<span class="token punctuation">.</span>user_data<span class="token punctuation">.</span>rendered<span class="token punctuation">)</span>

  network_interfaces <span class="token punctuation">{</span>
    device_index                <span class="token operator">=</span> <span class="token number">0</span>
    associate_public_ip_address <span class="token operator">=</span> <span class="token boolean">true</span>
    security_groups             <span class="token operator">=</span> <span class="token punctuation">[</span>aws_security_group<span class="token punctuation">.</span>sg<span class="token operator">-</span>xwiki<span class="token punctuation">.</span>id<span class="token punctuation">]</span>
  <span class="token punctuation">}</span>

  iam_instance_profile <span class="token punctuation">{</span>
    name <span class="token operator">=</span> aws_iam_instance_profile<span class="token punctuation">.</span>xwiki<span class="token operator">-</span>instance<span class="token operator">-</span>profile<span class="token punctuation">.</span>name
  <span class="token punctuation">}</span>

  tag_specifications <span class="token punctuation">{</span>
    resource_type <span class="token operator">=</span> <span class="token string">"volume"</span>

    tags <span class="token operator">=</span> <span class="token punctuation">{</span>
      Name       <span class="token operator">=</span> <span class="token string">"xwiki-root-volume"</span>
      environment <span class="token operator">=</span> <span class="token string">"prod"</span>
    <span class="token punctuation">}</span>
  <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
JavaScript

xwiki_ubuntu18.04 defines which AWS AMI will be used. This was created by Packer as pictured above in the Packer section. This config creates a Launch Template and all the resources related to it. 

Later on, the AWS EBS is configured, the Instance type on which Xwiki will be installed, and the IAM instance profile to be attached to the EC2 instance, security groups and tags. 

User_data describes pre-configured actions to be taken, such as: checking if the EBS is mounted, associating always the same Elastic IP, creating the proper FS on partitions. 

The Launch Template is strictly connected to the Auto Scaling Group, so the config goes as follows:

resource <span class="token string">"aws_autoscaling_group"</span> <span class="token string">"xwiki"</span> <span class="token punctuation">{</span>
  name             <span class="token operator">=</span> <span class="token string">"xwiki"</span>
  min_size         <span class="token operator">=</span> <span class="token string">"1"</span>
  max_size         <span class="token operator">=</span> <span class="token string">"1"</span>
  desired_capacity <span class="token operator">=</span> <span class="token string">"1"</span>
  force_delete     <span class="token operator">=</span> <span class="token boolean">false</span>

  launch_template <span class="token punctuation">{</span>
    id     <span class="token operator">=</span> aws_launch_template<span class="token punctuation">.</span>xwiki<span class="token operator">-</span>lt<span class="token punctuation">.</span>id
    version <span class="token operator">=</span> <span class="token string">"$Latest"</span>
  <span class="token punctuation">}</span>
  vpc_zone_identifier <span class="token operator">=</span> <span class="token punctuation">[</span>
    data<span class="token punctuation">.</span>terraform_remote_state<span class="token punctuation">.</span>internal<span class="token operator">-</span>prod<span class="token operator">-</span>vpc<span class="token punctuation">.</span>outputs<span class="token punctuation">.</span>network_public_id<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
  <span class="token punctuation">]</span>
tag <span class="token punctuation">{</span>
    key                 <span class="token operator">=</span> <span class="token string">"Name"</span>
    value               <span class="token operator">=</span> <span class="token string">"xwiki"</span>
    propagate_at_launch <span class="token operator">=</span> <span class="token boolean">true</span>
  <span class="token punctuation">}</span>
tag <span class="token punctuation">{</span>
    key                 <span class="token operator">=</span> <span class="token string">"environment"</span>
    value               <span class="token operator">=</span> <span class="token string">"prod"</span>
    propagate_at_launch <span class="token operator">=</span> <span class="token boolean">true</span>
  <span class="token punctuation">}</span>
tag <span class="token punctuation">{</span>
    key                 <span class="token operator">=</span> <span class="token string">"project"</span>
    value               <span class="token operator">=</span> <span class="token string">"xwiki"</span>
    propagate_at_launch <span class="token operator">=</span> <span class="token boolean">true</span>
  <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
JavaScript

ASG is also used here as a Self-healing mechanism, which is also a big keyword for the orchestration process. If it gets terminated somehow, it will be recreated one more time based on the above settings. ASG ensures how many instances will be spawned, which Launch Template will be used and in our example it also adds some tags describing resources. 

The previously mentioned resources regarding Xwiki are not the only ones we have in the Terraform config files. We also have Route53, Security Groups and EIP defined as IaaC, but it would be boring to show you all of these as they are the same as in the documentation, only just adjusted to our internal needs. 

Orchestration means possessing an Infrastructure which is safe and sound and also ready to be resurrected from the dead when required without any human interaction, or at least with less effort. 

Ansible is a configuration management and application-deployment solution designed by Red Hat. It works as anagentless model and requires only an SSH established connection and Python as an interpreter on the target machine. 

Similarly to Terraform, it also has a declarative language that is used to describe roles and tasks to be executed. 

Ansible can be run on one target or more simultaneously.

In our case, the structure is as follows:

  • host_vars –  variables used in roles and tasks for specific hosts.
  • group_vars – variables used for all hosts enclosed in the groups.
  • internal_hosts.yml – Ansible playbook.
  • inventory.ini – file containing all targets on which we execute Ansible.

So if we would like to launch this tool, we hit the terminals with the following command:

ansible<span class="token operator">-</span>playbook <span class="token operator">-</span>i inventory<span class="token punctuation">.</span>ini internal_hosts<span class="token punctuation">.</span>yml
JavaScript

If a certain object needs some extra variables included in the role or task, they are defined in the host_vars directory. On the particular target, it executes the roles and tasks described in internal_hosts.yml using the SSH connection. 

Based on Xwiki in my previous examples, I will do the same here.

For example, host_vars/xwiki looks like this:

hostname<span class="token punctuation">:</span> xwiki

xwiki_test_pass<span class="token punctuation">:</span> <span class="token operator">!</span>vault <span class="token operator">|</span>
          $ANSIBLE_VAULT<span class="token punctuation">;</span><span class="token number">1.1</span><span class="token punctuation">;</span>AES256        <span class="token number">633033383465346139386238643638366633613439356332333334623338643536616631363566653939353765343</span>         <span class="token number">61333533396438335343166313237613034653862316635336663623636313264386</span>

certbot_domains<span class="token punctuation">:</span>
  <span class="token operator">-</span> wiki<span class="token punctuation">.</span>scalac<span class="token punctuation">.</span>io

nginx_generic_domains<span class="token punctuation">:</span>
  <span class="token operator">-</span> <span class="token punctuation">{</span> domain_name<span class="token punctuation">:</span> <span class="token string">'wiki.scalac.io'</span><span class="token punctuation">,</span> dest<span class="token punctuation">:</span> <span class="token string">'https://localhost:8080'</span><span class="token punctuation">,</span> cert<span class="token punctuation">:</span> <span class="token string">'wildcard'</span><span class="token punctuation">,</span> action<span class="token punctuation">:</span> <span class="token string">'proxy_pass'</span> <span class="token punctuation">}</span>
JavaScript

This is a good time to explain the !vault existence in the above code as this is what we use very often in files with variables.  The hostname is matched with the SSH config; if the entry is here, Ansible starts the connection, if not, it will throw an error – hostname does not match – and will stop the execution. 

To protect important and crucial data such as passwords, tokens, access and secret keys, Ansible provides a tool called Ansible-vault. It is integrated within so there is no need to install it separately. 

All we have to do is just to put the important data in plaintext in some temporary file and then run:

ansible-vault encrypt this_temp_file and fill the Vault password on the terminal prompt. Black Magic happens and the data is encrypted. 

To decrypt:

ansible-vault decrypt this_temp_file and fill in the Vault password which was used to encrypt. 

We use Ansible roles for Nginx and Certbot, directives which are also included in the host_vars/xwiki file. 

It has already been mentioned in the Packer section what internal_hosts.yml for Xwiki looks like, so there is no need to repeat it once again. 

The information in the form of tasks to be executed on Xwiki hosts is defined in the Xwiki role. For this service actually there is only one: 

<span class="token operator">--</span><span class="token operator">-</span>
<span class="token operator">-</span> name<span class="token punctuation">:</span> Run xwiki container
  docker_container<span class="token punctuation">:</span>
    name<span class="token punctuation">:</span> xwiki
    image<span class="token punctuation">:</span> xwiki<span class="token punctuation">:</span><span class="token number">11.3</span><span class="token operator">-</span>postgres<span class="token operator">-</span>tomcat
    state<span class="token punctuation">:</span> started
    pull<span class="token punctuation">:</span> <span class="token boolean">true</span>
    restart_policy<span class="token punctuation">:</span> always
    published_ports<span class="token punctuation">:</span>
      <span class="token operator">-</span> <span class="token number">8080</span><span class="token punctuation">:</span><span class="token number">8080</span>
    volumes<span class="token punctuation">:</span>
      <span class="token operator">-</span> <span class="token operator">/</span>data<span class="token operator">/</span>xwiki<span class="token operator">/</span>xwiki<span class="token punctuation">:</span><span class="token operator">/</span>usr<span class="token operator">/</span>local<span class="token operator">/</span>xwiki
    env<span class="token punctuation">:</span>
      DB_USER<span class="token punctuation">:</span> xwiki<span class="token operator">-</span>test
      DB_PASSWORD<span class="token punctuation">:</span> <span class="token string">"{{ xwiki_test_pass }}"</span>
      DB_DATABASE<span class="token punctuation">:</span> xwiki<span class="token operator">-</span>test
      DB_HOST<span class="token punctuation">:</span> testenv<span class="token operator">-</span>postgres<span class="token punctuation">.</span>fnijfgtbdlyn<span class="token punctuation">.</span>eu<span class="token operator">-</span>central<span class="token number">-1</span><span class="token punctuation">.</span>rds<span class="token punctuation">.</span>amazonaws<span class="token punctuation">.</span>com
JavaScript

This runs a built-in docker container module with all the needed parameters, builds and finally launches Xwiki service aka wiki.scalac.io in a container with the mapping ports on the designated target. Database credentials (encrypted by ansible-vault) are injected as environment variables.  

In addition to the above role, we also have different ones:

Role name Short description
configure_os for configuring OS with setting up hostnames in /etc/hosts and disabling system services 
os-update for keeping OS always up to date
base-packages for installing commonly used packages like gzip, htop, git, etc.
docker for installing docker on every target
jenkins for installing and configuring Jenkins and its components
ssh for applying and conducting ssh keys and configs for every host
fail2ban for installing and configuring fail2ban – server security tool
chrony for installing and configuring one NTP server on every host
exporters for applying Prometheus exporters on every host
aws-cli for installing and configuring AWS CLI
backups for applying backup scripts and cron configs on every server
certbot for installing, setting up and generating certs using Certbot
nginx for installing and configuring Nginx
logs-filebeat for installing Filebeat on every host for ELK

In this entire AWS orchestration process, Ansible plays a big role. Having the  infrastructure in a code (thanks to Terraform) does not solve all the problems though. Without a configuration management tool we would still be doing OS-related stuff manually and this would consequently lead to repetitive mistakes. 

..eventually it had to happen, this journey is almost over, but yet two more stories remain to be unfolded. Brace yourself as we are sailing to undiscovered lands that anyone brave enough can conquer. Only you can decide this.

Docker was invented and developed to facilitate the entire deployment process. The software maker can easily package up an application with all its components such as libraries into one container and ship it all out to the desired target. 

Containers are separated from each other and thanks to this, you are able to run multiple applications on one server without having any problems with dependencies. 

During the discovery phase on old infrastructure, we decided to package every possible application into the Docker containers. Having done that, we were able to build it, ship it and deploy it onto AWS EC2 instances using Docker Hub as a harbor for container images.

To prepare an application package as a Docker image, we had to create Dockerfiles which are required in the building phase. 

This process is strictly related to Jenkins, as he is the captain of this orchestration ship, so it will be described in a bit more detail in the Jenkins section, where we are heading right now.

Jenkins – to gather them all

What do automation and orchestration processes look like without proper Continuous Integration and Continuous Delivery? There is no such thing. 

CI/CD is the first fiddle in the whole deployment methodology.  

In our story, we used Jenkins for this purpose. 

Jenkins is an open-source automation server. All non-human actions during the deployment activities can be managed by it. This CI/CD solution contains an enormous amount of packages and plugins to be used in job configurations. It can also very easily integrate with third-party applications to ensure a better and more efficient delivery flow. 

Jenkins reminds me of a little man with big and bare feet – Frodo Baggins when he was gathering the Fellowship of the Ring to make his  colossal commitment. 

Using it in our AWS orchestration, we were able to join all the pieces of the jigsaw together in one platform. 

This whole article is based on an Xwiki example. Actually, in this section, that could be a little inaccurate because building and deploying Xwiki has already been done by Ansible as mentioned a few paragraphs ago. But what goes around, comes around. 

Jenkins, in this case, is responsible for launching two jobs == tasks. 

  • devops-packer-xwiki
  • devops-refresh-servers

First of all, it downloads our repository from Github, runs the script for building a Packer image and later on executes the Terraform apply command on the AWS Launch Template to switch the ID’s of the xwiki_ubuntu18.04 image in AWS AMI. 

#<span class="token operator">!</span><span class="token operator">/</span>bin<span class="token operator">/</span>bash
<span class="token keyword">set</span> <span class="token operator">-</span>x
<span class="token keyword">set</span> <span class="token operator">-</span>euo pipefail

docker login <span class="token operator">-</span>u $build_user <span class="token operator">-</span>p $build_passwd

# build ami
<span class="token punctuation">.</span><span class="token operator">/</span>build_images<span class="token punctuation">.</span>sh xwiki

# update launch template
cd terraform
<span class="token punctuation">.</span><span class="token operator">/</span>apply scalac<span class="token operator">/</span>internal<span class="token operator">-</span>prod<span class="token operator">/</span>xwiki
JavaScript

This job runs every 2 hours every day, but if the service is down before another execution, Prometheus alerts are delivered to the Slack channel about downtimes and we have the  power to launch this job manually from the Jenkins UI. The second one is in play because it invokes the Ansible playbook with the roles defined in the internal_hosts.yml file matching the particular host from the inventory. In this example, it is the tasks for the Xwiki target (previously mentioned in the Ansible section of this article). 

Failure example:

If on the server where Xwiki is hosted,  the docker daemon or the docker container with the service stops working, devops-refresh-servers will relaunch the Xwiki Ansible role and restore the container to an up-and-running state. 

In addition, this particular job also protects us from arranging any manual interactions on the servers. Even if such a thing happens, it will restore the defaults written down in the host/group vars. 

The picture below  shows the flow for these two Jenkins jobs:

Automation AWS, IaaS, IaaC,CICD

AWS automation Jenkins Xwiki

The above examples were based on bash scripts we have created for our internal needs. Almost every project has Jenkins jobs because in every one there is a build and deployment phase. As we were trying to avoid manual steps as much as possible when starting the orchestration , we used our CICD to build Docker containers for applications as well. 

 

Build phase using Jenkins:

docker login <span class="token operator">-</span>u $scalac_user <span class="token operator">-</span>p $scalac_pass

<span class="token keyword">export</span> PROJECT_VERSION<span class="token operator">=</span><span class="token template-string"><span class="token string">`grep -E "^version" build.sbt | sed -e "s/.*= \"\(.*\)\"/\1/"`</span></span>

# clean up previous build artifacts
sudo rm <span class="token operator">-</span>rf hire<span class="token operator">-</span>help<span class="token operator">-</span>$PROJECT_VERSION target

# build the code
docker run <span class="token operator">--</span>rm <span class="token operator">-</span>i \
    <span class="token operator">--</span>name $JOB_NAME \
    <span class="token operator">-</span>v $WORKSPACE<span class="token punctuation">:</span><span class="token operator">/</span>src \
    <span class="token operator">-</span>v $WORKSPACE<span class="token operator">/</span><span class="token punctuation">.</span>ivy2<span class="token punctuation">:</span><span class="token regex">/root/</span><span class="token punctuation">.</span>ivy2 \
    <span class="token operator">-</span>e PROJECT_ENV<span class="token operator">=</span><span class="token string">"dev"</span> \
    <span class="token operator">-</span>e PROJECT_DB_PASSWORD \
    <span class="token operator">-</span>e PROJECT_DB_NAME \
    <span class="token operator">-</span>e PROJECT_DB_USERNAME \
    <span class="token operator">--</span>cpus<span class="token operator">=</span><span class="token string">"1"</span> \
    hseeberger<span class="token operator">/</span>scala<span class="token operator">-</span>sbt<span class="token punctuation">:</span>8u181_2<span class="token number">.12</span><span class="token punctuation">.</span>8_1<span class="token number">.2</span><span class="token punctuation">.</span><span class="token number">8</span> \
    <span class="token operator">/</span>bin<span class="token operator">/</span>bash <span class="token operator">-</span>c <span class="token string">'cd /src && ./build.sh'</span>

# unzip the <span class="token keyword">package</span>
docker run <span class="token operator">--</span>rm <span class="token operator">-</span>i \
    <span class="token operator">--</span>name $JOB_NAME \
    <span class="token operator">-</span>v $WORKSPACE<span class="token punctuation">:</span><span class="token operator">/</span>src \
    <span class="token operator">-</span>e PROJECT_VERSION \
    <span class="token operator">-</span>e PROJECT_DB_PASSWORD \
    <span class="token operator">-</span>e PROJECT_DB_NAME \
    <span class="token operator">-</span>e PROJECT_DB_USERNAME \
    <span class="token operator">-</span>w <span class="token operator">/</span>src \
    busybox \
    unzip <span class="token operator">/</span>src<span class="token operator">/</span>target<span class="token operator">/</span>universal<span class="token operator">/</span>project<span class="token operator">-</span>$project_VERSION

# build and push the container
docker build <span class="token operator">-</span>t our<span class="token operator">-</span>dockerhub<span class="token operator">/</span>scalac<span class="token operator">-</span>project<span class="token operator">-</span>backend<span class="token punctuation">:</span>dev <span class="token punctuation">.</span>
docker push our<span class="token operator">-</span>dockerhub<span class="token operator">/</span>scalac<span class="token operator">-</span>project<span class="token operator">-</span>backend<span class="token punctuation">:</span>dev
JavaScript

The deployment phase using Jenkins:Building the code and unzipping happens inside the Docker containers so not to pollute the server space where Jenkins’s jobs have been executed. 

APP_NAME<span class="token operator">=</span>project<span class="token operator">-</span>backend<span class="token operator">-</span>devel

<span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token template-string"><span class="token string">`docker ps -a | grep -E "\s$APP_NAME\s*$" | awk '{ print $1 }'`</span></span><span class="token punctuation">;</span> <span class="token keyword">do</span>
    docker kill $i <span class="token operator">||</span> <span class="token boolean">true</span>
    docker rm $i
done

docker pull scalac_docker_hub<span class="token operator">/</span>scalac<span class="token operator">-</span>project<span class="token operator">-</span>backend<span class="token punctuation">:</span>dev

docker run <span class="token operator">-</span>d \
    <span class="token operator">-</span>e PROJECT_S3_ACCESS_KEY \
    <span class="token operator">-</span>e PROJECT_S3_SECRET_KEY \
    <span class="token operator">-</span>e PROJECT_ENV<span class="token operator">=</span><span class="token string">"dev"</span> \
    <span class="token operator">-</span>e PROJECT_DB_PASSWORD\
    <span class="token operator">-</span>e PROJECT_DB_NAME \
    <span class="token operator">-</span>e PROJECT_DB_USERNAME \
    <span class="token operator">-</span>p <span class="token number">10000</span><span class="token punctuation">:</span><span class="token number">10000</span> \
    <span class="token operator">--</span>name $APP_NAME \
    <span class="token operator">--</span>restart always \
    scalac_docker_hub<span class="token operator">/</span>scalac<span class="token operator">-</span>project<span class="token operator">-</span>backend<span class="token punctuation">:</span>dev \
    <span class="token operator">/</span>src<span class="token operator">/</span>start<span class="token punctuation">.</span>sh
JavaScript

Without Jenkins, we would still have one foot in Dark Times, despite having IaaC, IaaS and other puzzles that fit in with modern orchestration.As I have already mentioned, we were trying to build and deploy every application of ours in the same way to standardize the entire process. I think we have managed to achieve 90% of that so far. But some work is still yet to be done. 

With GitFlow we all flow

Having CI / CD tools combined is one thing, but using them wisely is another – not only from the infrastructure point of view but also from the perspective of the software . 

In a situation where several developers work on an application code, there’s no room for using only one branch to direct all of the commits, without any order. 

Each stage of the project should have its own specific tree structure in the repository. GitFlow is a tool that in a well-defined way allows you to enter the life cycle of the entire project. Regarding this subject, we have a dedicated article where the entire flow we use is presented. 

To wrap up

At the beginning of this article, I declared I would try to explain all of the concepts in more detail. I hope I have managed to achieve that, despite the scope. Of course, our solution cannot ensure that all the problems will just disappear. This article gives an insight into the problems we were facing and the methods we picked to figure them out.  

Now you can see what orchestration and automation looks like, with regard to infrastructure and software. 

If you still have any doubts or questions, do not hesitate to leave them in the comments below.

See also

Download e-book:

Scalac Case Study Book

Download now

Authors

Błażej Obiała

I am a DevOps Engineer passionate about automating processes. In the course of my career, I have gained three AWS certificates, and thanks to them, I can support customers by answering their questions regarding AWS architecture in their projects. Recently I also became a Certified Kubernetes Administrator. Privately I am keen on reading historical and philosophical literature while listening to good vinyl records. I used to play bass guitar in a band called Mad Teacher during my studies, so I've been on both sides of the music world.

Latest Blogposts

28.03.2024 / By  Matylda Kamińska

Scalendar April 2024

scala conferences april 2024

Event-driven Newsletter Another month full of packed events, not only around Scala conferences in April 2024 but also Frontend Development, and Software Architecture—all set to give you a treasure trove of learning and networking opportunities. There’re online and real-world events that you can join in order to meet colleagues and experts from all over the […]

14.03.2024 / By  Dawid Jóźwiak

Implementing cloud VPN solution using AWS, Linux and WireGuard

Implementing cloud VPN solution using AWS, Linux and WireGuard

What is a VPN, and why is it important? A Virtual Private Network, or VPN in short, is a tunnel which handles all the internet data sent and received between Point A (typically an end-user) and Point B (application, server, or another end-user). This is done with security and privacy in mind, because it effectively […]

07.03.2024 / By  Bartosz Puszczyk

Building application with AI: from concept to prototype

Blogpost About Building an application with the power of AI.

Introduction – Artificial Intelligence in Application Development When a few years ago the technological world was taken over by the blockchain trend, I must admit that I didn’t hop on that train. I couldn’t see the real value that this technology could bring to someone designing application interfaces. However, when the general public got to […]

software product development

Need a successful project?

Estimate project