Getting Audit policies to work when debugging external kube proxys or Worker node bootstrapping

Getting audit logs to actually work is hard bc you have to synchronize

- the fields in your policy WITH ACTUAL USERS and APIs that you KNOW are being called

- The PATH to the policy file on disk to the apiserver for k8s which MIGHT BE RUNNING in a container (i.e. openshift, TKG...) 

We need this for debugging complex flows in k8s where for example, a VM is coming up and needs to make a quick API Call to k8s, or else it dies.  You might never be able to trace this death without seeing APIServer auditng logs (example : CAPV nodes that need to do api calls to bootstrap themselves to get access to service account for reading the K8s Endpoints / Services for external kube proxys)  

So Heres how i got audit policies to work today:

First figure out WHO YOU ARE by installing kubectl krew

kubo@uOFLhGS9YBJ3y:/tmp$ kubectl whoami

kubernetes-admin

THEN after that , make a rbac policy 

kind: Policy rules: 

- level: Metadata 

  users: ["default", "kubernetes-admin" ] 

  namespaces: ["kube-system"] verbs: ["get"] resources: 

   - group: "" # core 

   resources:  ["configmaps","endpoints","pods"

This is now easy to test (once we do the rest of the instructions), bc you can just run kubectl get cm kubeproxy -n kube-system or whatever other conig-map you want to test audit logs for.... 

NOW WHERE DO I PUT THIS POLICY???

kube-apiserver - --audit-policy-file=/etc/kubernetes/manifests/policy.yaml --audit-log-path=/var/log/kubernetes/audit.log 

IF YOU RUN APISERVER IN A CONTAINER

Mount /etc/kubernetes/auditing or whereever you want your policy.yaml to live as a volume into the apiserver.

In TKG, we have  /etc/kubernetes/manifests/apiserver.yaml so we do  something like, 

1) Make a new /etc/kubernetes/auditing/ directory

2) put a Policy.yaml into it

3) Make the apiserver in a container MOUNT that directory 

  volumes:

  - hostPath:

      path: /var/log/kubernetes <-- where the audit logs go 

      type: DirectoryOrCreate

    name: audit-log

  - hostPath:

      path: /etc/kubernetes/admission-control-config.yaml

      type: File

    name: admin-control-conf

  - hostPath:

      path: /home/capv/audit-policy.yaml <-- what tells apiserver WHAT to audit

      type: File

    name: audit-policy

  - hostPath:

      path: /etc/ssl/certs

      type: DirectoryOrCreate

    name: ca-certs

AND NOW MOUNT THEM IN

    volumeMounts:

    - mountPath: /etc/audit-policy.yaml

      name: audit-policy

    - mountPath: /var/log/kubernetes

      name: audit-log