Adding more metrics to contrib/kubernetes prometheus recipes.

This is a very niche post, but just for my notes.

In kubernetes/contrib, we maintain a template for launching kube to monitor apiserver metrics through the kube RO service, which the APIServer itself binds to.


However, lately I've been wanting to analyze metrics coming from other places, for example, the controller manager.

Kubernetes binaries generally start a daemon attached to a particular port, and then they attach a /metrics handler to that port.

For example apiserver metrics are on :8080/metrics. 

Meanwhile controller-manager metrics are on :10252/metrics.

You can see where this happens in the code : just go to cmd/.. and you will see that when services launch up, they bind a metrics/ endpoint to an internal server that launches on a port (provided as an argument, or else, a default port).

Our contrib recipes unfortunately only scrape apiserver metrics.  If you need to add other prometheus metrics to your dashboards, you can do something like this...

                         "command": [
                             "./run_prometheus.sh",
                             "-t",
-                            "KUBERNETES_RO,PUSHGATEWAY",
+                            "KUBERNETES_RO,KCM,PUSHGATEWAY",
                             "-d",
                             "/var/prometheus/"
                         ],
@@ -69,7 +69,15 @@
                             {
                                 "name": "PUSHGATEWAY_SERVICE_PORT",
                                 "value": "9091"
-                            }
+                            },
+                            {
+                                "name": "KCM_SERVICE_HOST",
+                                "value": "123.111.222.333"
+                            },
+                            {
+                                "name": "KCM_SERVICE_PORT",
+                                "value": "10252"
+                            },
                         ],
                         "ports": [
Where of course the 123.111.222.333 above is the IP address of your controller manager.