Shipping internal logs from your Kubernetes applications to a cetnralized AWS service.

I recently wanted to put a demonstration together of what log aggregation would look like for the ELK stack.

The AWS and Kuberentes specific parts of this are arbitrary.  If you have a write URL for elasticsearch and a working logstash, you can simply copy the output segment below, use it for your logstash startup (-f option), and you will see your data in elasticsearch.

If you are using Logstash, and looking for a visualization strategy for all your data, this should be super easy to do, especially if youre using EC2 already.

1) Setup an EC2 Elasticsearch service.  This comes out of the box.  You can set it to insecure if you want to, so that anyone can write logstash entries to your connection.  Chances are this isnt a huge deal unless your logs are sensitive.  After all, you're just prototyping.


2) Now, create a logstash configuration file:


    input {
        beats {
            port => 5044
        }
        log4j {
            mode => server
            host => "0.0.0.0"
            port => 4560
            type => "log4j"
        }
    }
    output {
        if [type] == "log4j" {
            if ![stack_trace] {
                elasticsearch {
                            hosts => "search-hackday-5zdf**************6rrcap7isy2bmqm.us-east-1.es.amazonaws.com:80"
                            manage_template => false
                            index => "%{[@timestamp][host]}-%{+YYYY.MM.dd}"
                            document_type => "%{[@metadata][type]}"
                }
            } else {
                elasticsearch {
                            hosts => "search-hackday*********6rrcap7isy2bmqm.us-east-1.es.amazonaws.com:80"
                            manage_template => false
                            index => "%{[@timestamp][host]}-%{+YYYY.MM.dd}"
                            document_type => "%{[@metadata][type]}"
                 }
            }
        } else {
            elasticsearch {
                        hosts => "search-hackday-5*****p246rrcap7isy2bmqm.us-east-1.es.amazonaws.com:80"
                        manage_template => false
                        index => "%{[@timestamp][host]}-%{+YYYY.MM.dd}"
                        document_type => "%{[@metadata][type]}"
             }
        }
    }

3) Save it somewhere (/tmp/logstash.conf).

4) Now, if running logstash in a container, you'll need to inject your configuration into it.  You can simply do this by ovderriding the elasticsearch command, because you cannot overwrite a file with a config-map, due to limitations in the way that files are overwritten in volume mounts in the linux kernel.  Since YAML spaces are hell, rather then copying and pasting which you will undoubtedly copy and screw up, I'll paste screenshots of what your volumeMount/mountPath combination should look like in your pod.

Specifically, you will have volumes declared in the top of your pod, and then, in your container definition of your pod, you will have a subsection that points directly to the named logstash configmap created above.

Ignore the logstash-grafana / graphite namings, they are a vestige of the past.

Notice that I use port 80 for the AWS instance.  Thats because, rather then 9200/9300 or whatever the ES defaults are, the AWS hosted elasticsearch instances use port 80.  If you dont get this right, you'll get unreachable errors and your logstash will die.

3)  Kubernetes:  Now, in your kubernetes cluster, create a configmap like so:

kubectl create configmap hub-logstash-grafana --from-file=/tmp/logstash.conf


4) Finally, fire up your services that write to logstash.  You should see something like this:

17:01:29.950 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
17:01:29.974 [LogStash::Runner] INFO  logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=>"c510c3fd-0efd-4ce5-a905-3059ede614c7", :path=>"/usr/share/logstash/data/uuid"}
17:01:30.937 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://search-hackday-5zdfpxm5p246rrcap7isy2bmqm.us-east-1.es.amazonaws.com/]}}
17:01:30.939 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://search-hackday-5zdfpxm5p246rrcap7isy2bmqm.us-east-1.es.amazonaws.com/, :path=>"/"}
log4j:WARN No appenders could be found for logger ...
w Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#]}
17:01:31.141 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://search-hackday-5z....
17:01:31.262 [[main]-pipeline-manager] INFO  logstash.inputs.beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
17:01:31.487 [[main]-pipeline-manager] INFO  logstash.inputs.log4j - Starting Log4j input listener {:address=>"0.0.0.0:4560"}
17:01:31.501 [[main]-pipeline-manager] INFO  logstash.inputs.log4j - Log4j input
17:01:31.507 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
17:01:31.618 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}