kumo particles¶
At glomex we create our infrastructure via AWS cloudformation and in order to do that efficiently we wanted to use reusable building blocks. For that to achieve we where evaluating different solutions available to us via open source.
So this is basically conceptual paper-ware (structure and a few helpers). You still need to write the particles.
Please be aware not to let kumo particles stop you from anything. In case you do not have particles or you do not want to write any you can still build beautiful infrastructure from the raw services provided by AWS.
kumo particles are perfectly optional. There is no tight coupling! You can totally bring your own building-block-mechanism and still use kumo for deployment. You do not even have to use troposphere - as long as your mechanism can export a valid json cloudformation template we are fine. Actually we encourage you to do so. Please share with us what you come up with.
Goals¶
- codify best practices for infrastructure
- use
cloudformation.py
to assemble a stack from particles - complexity is handled in particle
Detailed requirements¶
- particle has default parameters that can be overridden
- particle provides default permission that can be overridden
- we distribute particles as python packages (later move away from github subprojects)
- we want to distribute generic std. particles company wide (e.g. glomex-particles)
- we want to distribute squad specific particles (e.g. mes-particles)
Status on kumo particles implementation¶
- kumo particle implementation is based on MES
template_generator.py
- answered “what is the minimum information we need to provide to use a particle?”
- restore troposphere character (talked about context => template is the context)
- added SERVICE_NAME and DEFAULT_TAGS to template
- I liked the “template_generator” implementation but class structure gets in the way when creating stacks from multiple particle sources
- move cloudformation parameters and outputs into particle
- move permissions profile to particle
TODOs
- create particle profiles using awacs to further shrink the particles
- look into naming conventions / tooling for autogenerated resource names here it is important that in case we generate random names we can regenerate the same names during testing (gcdt placebo tools)
- share particles via package (need Github repo, Jenkins build, …)
Usage¶
To build better infrastructure at glomex we want to assemble infrastructure from reusable particles.
The gcdt.kumo_particle_helper
module contains the functionality (initialize, get_particle_permissions, and Particle) to integrate the kumo particles into troposphere
Quickstart example using particles¶
With kumo particles you can import particles from multiple sources:
from gcdt_kumo import kumo_particle_helper as ph
import eventbus_particle as eb
import reusable_particles as rp
We use cloudformation.py
to assemble a stack from particles:
def assemble_particles(template):
################# parameters #############################################
param_sns_alerts_topic = template.add_parameter(troposphere.Parameter(
'SNSAlertsTopic',
Description='Name for topic that receive notifications for validation.',
Type='String'
))
################# particles ##############################################
particles = [] # list of Particle()
sg_frontend_web = Ref('%sFrontendWeb' % template.SERVICE_NAME)
################# s3 bucket ##############################################
particles.append(rp.create_s3_bucket(template))
...
Under the hood we use troposphere
to code cloudformation templates. The troposphere template instance is used as a common context
to exchange information between kumo particles. With kumo each cloudformation.py
needs to implement a generate_template function.
def generate_template():
template = troposphere.Template()
ph.initialize(template, 'miaImportProcessor')
assemble_particles(template)
return template.to_json()
Developing your own particles¶
We just started with kumo particles and plan to provide more help on particle development in the future.