kumo particles

At glomex we create our infrastructure via AWS cloudformation and in order to do that efficiently we wanted to use reusable building blocks. For that to achieve we where evaluating different solutions available to us via open source.

So this is basically conceptual paper-ware (structure and a few helpers). You still need to write the particles.

Please be aware not to let kumo particles stop you from anything. In case you do not have particles or you do not want to write any you can still build beautiful infrastructure from the raw services provided by AWS.

kumo particles are perfectly optional. There is no tight coupling! You can totally bring your own building-block-mechanism and still use kumo for deployment. You do not even have to use troposphere - as long as your mechanism can export a valid json cloudformation template we are fine. Actually we encourage you to do so. Please share with us what you come up with.

Goals

  • codify best practices for infrastructure
  • use cloudformation.py to assemble a stack from particles
  • complexity is handled in particle

Detailed requirements

  • particle has default parameters that can be overridden
  • particle provides default permission that can be overridden
  • we distribute particles as python packages (later move away from github subprojects)
  • we want to distribute generic std. particles company wide (e.g. glomex-particles)
  • we want to distribute squad specific particles (e.g. mes-particles)

Status on kumo particles implementation

  • kumo particle implementation is based on MES template_generator.py
  • answered “what is the minimum information we need to provide to use a particle?”
  • restore troposphere character (talked about context => template is the context)
  • added SERVICE_NAME and DEFAULT_TAGS to template
  • I liked the “template_generator” implementation but class structure gets in the way when creating stacks from multiple particle sources
  • move cloudformation parameters and outputs into particle
  • move permissions profile to particle

TODOs

  • create particle profiles using awacs to further shrink the particles
  • look into naming conventions / tooling for autogenerated resource names here it is important that in case we generate random names we can regenerate the same names during testing (gcdt placebo tools)
  • share particles via package (need Github repo, Jenkins build, …)

Usage

To build better infrastructure at glomex we want to assemble infrastructure from reusable particles.

The gcdt.kumo_particle_helper module contains the functionality (initialize, get_particle_permissions, and Particle) to integrate the kumo particles into troposphere

Sample particles

instance

will create or update a CloudFormation stack

Quickstart example using particles

With kumo particles you can import particles from multiple sources:

from gcdt_kumo import kumo_particle_helper as ph
import eventbus_particle as eb
import reusable_particles as rp

We use cloudformation.py to assemble a stack from particles:

def assemble_particles(template):
    ################# parameters #############################################
    param_sns_alerts_topic = template.add_parameter(troposphere.Parameter(
        'SNSAlertsTopic',
        Description='Name for topic that receive notifications for validation.',
        Type='String'
    ))

    ################# particles ##############################################
    particles = []  # list of Particle()
    sg_frontend_web = Ref('%sFrontendWeb' % template.SERVICE_NAME)

    ################# s3 bucket ##############################################
    particles.append(rp.create_s3_bucket(template))
    ...

Under the hood we use troposphere to code cloudformation templates. The troposphere template instance is used as a common context to exchange information between kumo particles. With kumo each cloudformation.py needs to implement a generate_template function.

def generate_template():
    template = troposphere.Template()
    ph.initialize(template, 'miaImportProcessor')
    assemble_particles(template)
    return template.to_json()

Developing your own particles

We just started with kumo particles and plan to provide more help on particle development in the future.