Diving into the Kubernetes Prox, security, auth, and https

Today I tried to connect to a kubernetes internal service using an http endpoint.  I got the following error:

"E0904 10:11:56.307660 13442 event.go:195] Unable to write event: 'Post http://0.0.0.0:30291/api/v1/namespaces/default/events: malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"' (may retry after sleeping)".

After looking closer at the API server logs, I saw the following:

At the bottom you can see that the API Server logs are complaining that the incoming record isn't a TLS handshake.

whats going on?

It turns out that the HTTPS protocol can use TLS for securing bidirectional communication.  From stack overflow:

 HTTP is meant to run over a bidirectional tunnel for arbitrary binary data; when that tunnel is an SSL/TLS connection, then the whole is called "HTTPS".

So, in other words : If you get TLS errors or complaints that return from a server is malformed, you might be trying to connect to an https endpoint using http.


I still am having http handshake errors, even after fixing this.  But this is definetely progress. I'll leave notes as I learn here...

some more notes...

When i tested this, I got an error: Unable to register 192.168.1.190 with the apiserver: Post https://192.168.1.190:30291/api/v1/nodes x509: certificate signed by unknown authority, in the kubelet logs.

In the kube apiserver, I got a more course grained error "bad certificate".

Why?  Because the kubelet itself was saying "i can't authorize this api server", and the api-server was only receiving an undefiled failure message.

The moral of the story: When x509 authorization fails, you will see two errors - one on the place were the error originated (i.e. kubelet said "unknown authority") and another on the opposite end.  Obviously the error message on the origin of the failure is more useful.   :) 

nodeports vs ports

Understanding the semantics of ports, nodePorts, and TargetPorts is important.  This was recently asked on kubernetes-slack, I figured I'd diagram it for my notes and share it here.

Kubernetes "ports" are the internal ports associated with a load balanced endpoint.  If accessing a pod from outside the cabernets internal network (i.e the network which contains the VIPs), then you access a NodePort endpoint.  In any case, both of these ports get forwarded to a targetPort : This is the port that the pod itself listens on.