Running prometheus metrics

2020 update: I've moved this here...

https://github.com/jayunit100/k8sprototypes/tree/master/prometheus-quickly/README.md

Where there are real graphs etc, and specific kube instructions for using a simple prometheus deployment no operator required to quickly investigate ETCD performance.

I repeat.  NO OPERATOR REQUIRED.


This is so easy now, you don't really need to read this whole thing.  Instead, just do this:

docker run -p 9090:9090 prom/prometheus.

open your browser : localhost:9090.

execute a query / click on "Graph".

cutl localhost:9090/metrics to see the raw data exported from prometheus itself.


Original post 

Prometheus is a timeseries db + visualization tool which comes with a graph integrator, PromDash, which can aggregate multiple sources.

At the end of this post, we'll setup prom dash using two docker commands, and you can create graphs like this.

on the left, memory use , on the right cpu.  you're dashboard is 100% built from the easy to use promdash UI after running the node exporter and starting prometheus.

To get started, I just made shell scripts from the snippets here: http://www.boxever.com/monitoring-your-machines-with-prometheus.

These snippets

1) Put a metrics publisher up for you, which publishes metrics to localhost:9100.

2) Setup a prometheus server, which integrates data from metrics sources onto a user interface at localhost:9090.


How does the server know where to get metrics from?  It does this from the conf file that you create when you launch prometheus.  In my case, I just used the snippet in the above blog, which writes this file out for you.

job: {
      name: "node"
        target_group: {
               target: "http://localhost:9100/metrics"
        }
}

You can integrate multiple sources of data like so, by adding more target: fields.  thanks to tim st. claire at red hat for creating this dandy kubernetes scraper example !

job: {
      name: "node"
        target_group: {
                target: "http://localhost:9101/metrics"
                target: "http://localhost:9102/metrics"
                    ....
         }
}

Running PROMDASH


Running promdash is the next step.

Promdash will give you a visual dashboard where you can integrated the graphs created by prometheus metrics.  To run prometheus, you can use docker containers.

1) Create a prometheus database (to make this easy, we just use sql lite, but postgre/mysql is preferrable for a real setup) and write it to local disk...

docker run -v /tmp/prom:/tmp/prom -e DATABASE_URL=sqlite3:////tmp/prom/file.sqlite3 prom/promdash ./bin/rake db:migrate

2) Now, launch the prometheus server with the local disk mounted into your container.

docker run -p 3000:3000 -e DATABASE_URL=sqlite3:///tmp/file.sqlite3 prom/promdash

Thats it for now !