Use Istio to figure out what the hell you app is actually doing.

Make sure your kube supports mutation.

I've only used Istio's injector stuff as of today, and to get it working, I had to have kube 1.9:  You can figure out wether you'll be able to inject into your pods, try this:

kubectl api-versions | grep admissionregistration

Now install the sidecar injector so all your pods will be spyable.

https://istio.io/docs/setup/kubernetes/sidecar-injection.html#automatic-sidecar-injection.  You setup an ADM controller, it will mutate pods that come in with istio metadata sidecars....  You MUST have kube 1.9 for this, as its a new feature.  Heres what it looks like:

[11:59:11] training:istio-0.7.1 $ kubectl get pods -n istio-system
NAME                             READY     STATUS    RESTARTS   AGE
istio-ca-79556d4848-kjcj8        1/1       Running   0          18m
istio-ingress-5d6fc4ccdd-5xgp5   1/1       Running   0          18m
istio-mixer-7b7d4dc8db-8ztv9     3/3       Running   0          18m
istio-pilot-5cbb8fd59-gkqzj      2/2       Running   0          18m
[12:03:25] training:istio-0.7.1 $ kubectl get pod
[12:04:50] training:istio-0.7.1 $ kubectl apply -f install/kubernetes/istio.yaml

[12:05:15] training:istio-0.7.1 $ ./install/kubernetes/webhook-create-signed-cert.sh \
    --service istio-sidecar-injector \
    --namespace istio-system \
    --secret sidecar-injector-certs
creating certs in tmpdir /tmp/tmp.3lUlYyNE2X
Generating RSA private key, 2048 bit long modulus
......................+++
.............................................................................................................................+++
e is 65537 (0x010001)
certificatesigningrequest "istio-sidecar-injector.istio-system" created
NAME                                  AGE       REQUESTOR                     CONDITION
istio-sidecar-injector.istio-system   0s        jvyas@blackducksoftware.com   Pending
certificatesigningrequest "istio-sidecar-injector.istio-system" approved
secret "sidecar-injector-certs" created
[12:05:18] training:istio-0.7.1 $ kubectl apply -f install/kubernetes/istio-sidecar-injector-configmap-release.yaml

configmap "istio-inject" created
[12:05:36] training:istio-0.7.1 $ cat install/kubernetes/istio-sidecar-injector.yaml | \
     ./install/kubernetes/webhook-patch-ca-bundle.sh > \
     install/kubernetes/istio-sidecar-injector-with-ca-bundle.yaml
[12:05:38] training:istio-0.7.1 $ kubectl apply -f install/kubernetes/istio-sidecar-injector-with-ca-bundle.yaml

service "istio-sidecar-injector" created
serviceaccount "istio-sidecar-injector-service-account" created
deployment "istio-sidecar-injector" created
mutatingwebhookconfiguration "istio-sidecar-injector" created
[12:05:49] training:istio-0.7.1 $ kubectl -n istio-system get deployment -listio=sidecar-injector

NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
istio-sidecar-injector   1         1         1            1           15s

Ok... now before you try deploying an app, do this: make sure your namespace is labelled for istio to spy on...  my app namespace is hub-istio.

kubectl label namespace hub-istio istio-injection=enabled

Then create your app in the namespace.

Service graphs

Now, create the service graph service:

kubectl apply -f install/kubernetes/addons/servicegraph.yaml

kubectl expose svc servicegraph -n istio-system --type=LoadBalancer --name=ist-exp

And wait for the IP to come up....

Then you can go to:

http://:8088/force/forcegraph.html

Or use fancy kube forwarding proxy/nodeport stuff if your on an ancient cluster w/ no loadbalancer :).

And now, you can see all your application traffic !