Replace your CI w/ some kind of lambdas (for jobs) + and terraform (for "centralized" infra).

Note: The work in this post will be open sourced soon, once I find a way to remove all the hacks and highly specific tooling i've put into  it :).... But you shouldnt need the source to  implement this ~ its super easy.

WHY PYTHON ARENT YOU A KUBERNETES GUY ?

Yeah but anyone can python.  So non-infra folks will actually help you maintain things for once.  That said, golang has a great cron library you can use https://godoc.org/github.com/robfig/cron that works the exact same as 'schedule' which is what i used for the python crons below.

Is traditional CI good for anything? 

Yeah... Your regular old jenkinsy CI is good for apps ~ code that runs consistently and deploys consistently at small scales.  Once you get to large scales and your testing performance, looking for holes in deployment platforms, or trying to push chaos boundaries ~ failures are actually an ok  (or maybe even the desired) result, and thus need to be visualized differently.  Rollups have very little value out of context in this world.

If you work in the infrastructure world, you'll find that canned CI always falls short:

- THE DATA AGGREGATION TOOLING IS MISMATCHED The user interface for things like jenkins is optimized for unit tests which are short running, and of which there are 1000s.  Infrastructure tests are often long running, and produce massive amounts of data.  Syslogs, docker logs, and so on are important to infra poeple.

- IDIOMS ARE FOR JAVA DEVELOPERS: The policies around how CI operates often make immutability a second class citizen: Puppetizing or separating out the configs for your jenkins instance is entirely possible: But how many people actually do it?  Not many.   Thus, its tricky to figure out the right way to make your CI stateless and recoverable, alot of the time.

- HACKING IS IMPOSSIBLE: SSH'ing into your boxes is paramount.  And providing developers with a bastion where they can "hack" infra related projects, possibly sharing infrastructure with your CI, helps to save cost, improve morale (cause tmux is fun - admit it), and completely distrupt productivity slums that come from developer-managed infra.  

- TRADITIONAL CI JOBS ASSUME FAILURES ARE DUE TO BAD CODE.  This is a huge problem.  When you know your infrastructure, you tend to get really good at summarizing the flakey sharp corners if you build your own logging and retry logic into the jobs themselves.  The output is more complex, but thats ok, because the outputs are designed to be read by people that dont think of software as a 'green' / 'red' state. 

So , whats the solution? 

BUILD YOUR OWN !

Its not that hard.  In this post I'll walk through how to build your own CI with nothing other then Terraform, python, and S3.  And, you'll get beautiful, scalable, searchable logs.

1) First, carve up your infrastructure needs into terraform recipes.

I cant stress this enough.  Once you do this, you have a basically stateless way to recapture your cloud infrastructure / CI needs.  You use that as a primitive to build the rest of the system on top of.

2) For every job you want to run, make a new directory.

3) Write a controller script thats hackable, and use pycron or any other programmatic cron library.  Write functions for all your jobs.

4) Make a function call that directly spams either your chat rooms or your email.  This is typically the big 'value adds' from build servers, and its amazing how easy it is to do in 2 lines of code... After all we're in the REST era here.

5) For each "job", enter a job-log.sh script

6) Write another function to export your data to s3, and have that function call  the script in 5).

SO you get something like this,

job directories: 

job1/

   job-create.sh  # does the terraform

   job-deploy.sh  # runs your code, containers, whatever

   job-destroy.sh # does the terraform tear down

   job-log.sh # runs docker logs or kubectl logs on all your pods, writes them to a file.

... 

And your CI?  

Looks something like this: 

Python libraries you'll want:

- s3cmd (for searchable logs w/o needing to worry about rotation)

- subprocess

- datetime (b/c strfrmttime is nice for logfiles on the fly).

-  time (because time.sleep() is nice).

- pip install schedule (import it as , schedule)