Exit e-book
Show all chapters
05
Jenkins – to gather them all.
05. 
Jenkins – to gather them all.

Sign up to our Newsletter

Signing up to our newsletter allows you to read all our ebooks.

    How to create an automation process using AWS, IAAS, IAAC.
    05

    Jenkins – to gather them all.

    What do automation and orchestration processes look like without proper Continuous Integration and Continuous Delivery? There is no such thing. 

    CI/CD is the first fiddle in the whole deployment methodology.  

    In our story, we used Jenkins for this purpose.

    Jenkins is an open-source automation server. All non-human actions during the deployment activities can be managed by that. This CI/CD solution contains an enormous amount of packages and plugins to be used in job configurations. It can also very easily integrate with third-party applications to ensure a better and more efficient delivery flow.

    Jenkins reminds me of a little man with big and bare feet – Frodo Baggins when he was gathering the Fellowship of the Ring to make a tremendous commitment. 

    Using that in our AWS orchestration, we were able to join all the pieces of the jigsaw together in one platform.

    This whole article is based on Xwiki example. Actually, in this section, it could be a little bit inaccurate because building and deploying Xwiki is already done by Ansible as mentioned a few blocks ago. But what goes around, comes around. 

    Jenkins, in this case, is responsible for launching two jobs == tasks. 

    • devops-packer-xwiki
    • devops-refresh-servers

    First of all, it downloads our repository from Github, runs the script for building a Packer image and later on executes the Terraform apply command on AWS Launch Template to switch ID’s of the xwiki_ubuntu18.04 image in AWS AMI.

    #!/bin/bash
    set -x 
    set -euo pipefail
    
    docker login -u $build_user -p $build_passwd
    
    # build ami
    ./build_images.sh xwiki
    
    # update launch template
    cd terraform
    ./apply scalac/internal-prod/xwiki
    
    

    The second one is in play because it invokes Ansible playbook with roles defined in internal_hosts.yml file matching the particular host from inventory. In this example, tasks for Xwiki target (previously mentioned in the Ansible section of this article). 

    This job runs every 2 hours a day, but if the service is down before another execution, Prometheus alerts are being delivered to the Slack channel about downtimes and we are in power to launch this job manually from the Jenkins UI.

    Failure example:

    If on the server where Xwiki is hosted, docker daemon or docker container with the service stop working, devops-refresh-servers will relaunch Xwiki Ansible role and restore container to the up and running state.

    Besides, this particular job also protects us from arranging any manual interactions on servers. Even if such happen, it will restore defaults written down in the host/group vars.

    Below picture presents the flow for these two Jenkins jobs:

    The above examples were based on bash scripts we have created for internal needs. Almost every project has Jenkins jobs because in every one there is a build and deployment phase. As we were trying to avoid manual steps starting orchestration as much as possible, we used our CICD to build Docker containers for applications as well.

    Build phase using Jenkins:

    docker login -u $scalac_user -p $scalac_pass
    
    export PROJECT_VERSION=`grep -E "^version" build.sbt | sed -e "s/.*= \"\(.*\)\"/\1/"`
    
    # clean up previous build artifacts
    sudo rm -rf hire-help-$PROJECT_VERSION target
    
    # build the code
    docker run --rm -i \
        --name $JOB_NAME \
        -v $WORKSPACE:/src \
        -v $WORKSPACE/.ivy2:/root/.ivy2 \
        -e PROJECT_ENV="dev" \
        -e PROJECT_DB_PASSWORD \
        -e PROJECT_DB_NAME \
        -e PROJECT_DB_USERNAME \
        --cpus="1" \
        hseeberger/scala-sbt:8u181_2.12.8_1.2.8 \
        /bin/bash -c 'cd /src && ./build.sh'
    
    # unzip the package
    docker run --rm -i \
        --name $JOB_NAME \
        -v $WORKSPACE:/src \
        -e PROJECT_VERSION \
        -e PROJECT_DB_PASSWORD \
        -e PROJECT_DB_NAME \
        -e PROJECT_DB_USERNAME \
        -w /src \
        busybox \
        unzip /src/target/universal/project-$project_VERSION
    
    # build and push the container
    docker build -t our-dockerhub/scalac-project-backend:dev .
    docker push our-dockerhub/scalac-project-backend:dev
    
    
    
    
    

    Building the code and unzipping happens inside Docker containers due to not polluting the server’s space where Jenkins’s job is executed.

    Deployment phase using Jenkins:

    APP_NAME=project-backend-devel
    
    for i in `docker ps -a | grep -E "\s$APP_NAME\s*$" | awk '{ print $1 }'`; do
        docker kill $i || true
        docker rm $i
    done
    
    docker pull scalac_docker_hub/scalac-project-backend:dev
    
    docker run -d \
        -e PROJECT_S3_ACCESS_KEY \
        -e PROJECT_S3_SECRET_KEY \
        -e PROJECT_ENV="dev" \
        -e PROJECT_DB_PASSWORD\
        -e PROJECT_DB_NAME \
        -e PROJECT_DB_USERNAME \
        -p 10000:10000 \
        --name $APP_NAME \
        --restart always \
        scalac_docker_hub/scalac-project-backend:dev \
        /src/start.sh
    
    
    
    
    

    As I mentioned, we were trying to build and deploy every application of ours in the same way to standardize the entire process and I think we managed to achieve 90% so far, but some work is yet to be done. 

    Without Jenkins, we would still have one foot in Dark Times, despite having IaaC, IaaS and other puzzles that fit to modern orchestration.

    PREVIOUS
    Chapter
    04
    NEXT
    Chapter
    06