Getting Started Guide¶

Welcome to the getting started guide of gcdt. In this guide we will cover what gcdt is and how to use it to create beautiful infrastructure as code (IaC):

using AWS Cloudformation with kumo
deploy and configure AWS Lambda with ramuda
deploy your application with AWS Codedeploy scripts and tenkai
deploy API Gateway and manage API keys with yugen All of these things you can do for different Environments (dev, stage, prod)

Infrastructure as code¶

Infrastructure as Code (IaC) is a type of IT infrastructure that teams can automatically provision through code, rather than using a manual GUI-centerd process. Through defining your infrastructure as code you can apply similar tools and workflows like when working with your application code like:

version control
test
review
share
reuse (like creating test envs)
audit

Installation¶

Install Python¶

First of all you need to have Python installed. Python should be 2.7 or higher

On MacOS try to use preinstalled Python

Install pip and virtualenv¶

virtualenv is a tool to create isolated Python environments. virtualenv creates a folder which contains all the necessary executables to use the packages that a Python project would need.

MacOS¶

$ sudo pip install virtualenv --upgrade

Install gcdt and gcdt plugins¶

Install gcdt¶

First of all you need to create virtualenv and activate it. We recommend create virtualenv in the same directory as a project, and add it to .gitignore. It’s pretty easy.

$ cd <project-folder>
$ virtualenv venv
$ source venv/bin/activate
$ pip install pip --upgrade # we always should have latest pip version in our virtualenv

gcdt needs some gcdt-glugins so you should install these together. gcdt-glugins are powerful tool to add features to gcdt without having to directly modify the gcdt core. The easiest way is to put the dependencies into a requirements_gcdt.txt file:

gcdt
gcdt-config-reader
gcdt-lookups
gcdt-bundler
gcdt-slack-integration

You can find more information about plugins in docs then

$ pip install -U -r requirements_gcdt.txt

To check that everything is good and gcdt installed just do:

$ gcdt version
gcdt version 0.1.418
gcdt plugins:
 * gcdt-config-reader version 0.0.11
 * gcdt-bundler version 0.0.27
 * glomex-config-reader version 0.0.18
 * gcdt-slack-integration version 0.0.11
 * gcdt-lookups version 0.0.12

Setting the ENV environment variable¶

For almost all gcdt commands you need to set the ENV environment variable. ENV is required to recognize which config file to use (gcdt_<env>.json). Usually you have a different config file for each environment (dev, stage, prod). You need to set ENV before running any gcdt command.

The following command will use the gcdt_dev.json config file

$ export PYTHONIOENCODING=UTF-8
$ ENV=dev kumo list 
...

Alternatively you can set the ENV variable for the duration of your terminal session. Set it like this:

$ export ENV=dev
$ kumo list
...

Kumo¶

Kumo is a tool which help you to manage and deploy your infrastructure. AWS Cloudformation uses helps you configure and manage your AWS infrastructure as code. In kumo we have our cloudformation templates generated by troposphere. With kumo you can easily create (and configure) infrastructure for different environments (and AWS accounts) like for example (dev, stage, prod).

Create your first stack with kumo¶

First of all you need to create two files:

cloudformation.py - here you will describe your infrastructure using troposphere
gcdt_(dev|stage|prod) - settings for your ENV in json format, needs to include all parameters for the cloudformation template + stack name. You should have separate config for each ENV.

Let’s create a simple gcdts_dev.json(please change all values according your AWS account):

{
  "kumo": {
    "stack": {
      "StackName": "gcdt-sample-stack"
    },
    "parameters": {
      "VPCId": "lookup:stack:<stack-name>:DefaultVPCId",
      "ScaleMinCapacity": "1",
      "ScaleMaxCapacity": "1",
      "InstanceType": "t2.micro",
      "DefaultInstancePolicyARN": "lookup:stack:<stack-name>:DefaultInstancePolicyARN",
      "AMI": "lookup:secret:ops.prod.base_ami"
    }
  }
}

The values for VPCId and DefaultInstancePolicyARN are filled by by the gcdt-lookups which then will be used in the template. The gcdt-lookups plugin will search the outputs in the CloudFormation stack (as mentioned in the config).

Instead of <stack-name> you should provide your stack name or use hardcoded value(not recommended). It’s time to create our first Infrastructure as Code. Let’s do this. Here is a simple cloudformation.py script. Use it as a template for creating your infrastructure.

Deploy stack to AWS¶

Before running a deployment we need to set some necessary ENV variables. Remember: You need this ENV variables exported each time before running any gcdt command.

$ export ENV=dev
$ export AWS_DEFAULT_PROFILE=glomex # Default profile. Generated by aws-mfa
$ export AWS_DEFAULT_REGION=eu-west-1
$ export PYTHONIOENCODING=UTF-8

Run your first infrastructure deployment. It’s really easy:

$ kumo deploy

Kumo_Deploy_Output Kumo deploy output More information about kumo can be found in docs

Ramuda¶

Ramuda will help you to deploy, manage and control AWS Lambda. Runtimes supported by ramuda are: nodejs4.3, nodejs6.10, python2.7, python3.6

Deploy simple AWS Lambda¶

Create a gcdt_(dev|stage|prod) file or update if you created it with kumo (please change all values according your AWS account):

"ramuda": {
  "bundling": {
    "folders": [
        {
            "source": "./node_modules",
            "target": "./node_modules"
        }
    ],
    "zip": "bundle.zip"
  },
  "lambda": {
    "name" = "jenkins-gcdt-lifecycle-for-ramuda",
    "description" = "lambda test for ramuda",
    "role" = "lookup:stack:<stack-name>:LambdaArnForDeploy",
    "handlerFunction" = "handler.handle",
    "handlerFile" = "handler.py",
    "timeout" = "300",
    "memorySize" = "256",
    "vpc": {
            "subnetIds": [
                "lookup:stack:<stack-name>:LambdaSubnetIda",
                "lookup:stack:<stack-name>:LambdaSubnetIdb",
                "lookup:stack:<stack-name>:LambdaSubnetIdc"
            ],
        }
  }
}

then do:

$ ramuda deploy

More information about ramuda can be found in docs

Tenkai¶

tenkai will help you to deploy your application using AWS Codedeploy. tenkai will create an application bundle file and upload it to s3 with all files that you have in your codedeploy folder. Create or update the gcdt_(dev|stage|prod) file (please change all values according your AWS account):

"tenkai": {
  "codedeploy": {
    "applicationName": "lookup:stack:gcdt-sample-stack:applicationName",
    "deploymentGroupName": "lookup:stack:gcdt-sample-stack:DeploymentGroup",
    "deploymentConfigName": "lookup:stack:gcdt-sample-stack:DeploymentConfig",
    "artifactsBucket": "lookup:stack:<stack-name>:s3DeploymentBucket"
  }
}

then do:

$ tenkai deploy

More information about tenkai can be found in docs

Yugen¶

yugen is a tool that will help you to deploy and manage your API with AWS API Gateway. All you need is to put your swagger.yml into the same folder as a gcdt_(dev|stage|prod) file. Also, add some new configs into it (please change all values according your AWS account):

"yugen": {
    "api": {
        "apiKey": "xxxxxxxxxxxxxx",
        "description": "Gcdt sample API based on dp api-mock",
        "name": "jenkins-gcdt-sample-api-dev",
        "targetStage": "mock"
    }
}

More information about yugen can be found in docs