jayunit100: Can you delete kube-proxy from a TKG cluster? not really..... but some stuff does work... ?

Update!!!!!! After writing this that you need to enable proxyAll for Antrea to proxy all traffic. Turns out (thanks to Quan tian on the Antrea team) that

- AntreProxy handles service traffic for CNI pods normally.

- it CAN handle hostNetwork pods and external traffic , though

- kube proxy's iptables rules will intercept traffic before you get to OVS though, so, if kube proxy is enabled, antreaproxy loadbalancing is irrelevant, and redundant.

- to replace kube proxy you need to set kubeAPIServerOverride and proxyAll:true for Antrea so that it can (1) talk direct to the apiserver instead of using the internal svc endpoint and (2) proxy all traffic instead of only proxying internal ClusterIP traffic.

NOW THAT SAID

If you have OLD IPTABLES RULES in a cluster for services, then those services might accidentally load balance you to pods that don't exist anymore. Deleting the tube-proxy wont automatically delete it's iptables rules !!!!

-A KUBE-SVC-SAL3JMSY3XQSA64S -m comment --comment "tkg-system/tkr-resolver-cluster-webhook-service -> 100.96.1.15:9443" -j KUBE-SEP-APPCGWWBW4GQN2HK

-A KUBE-SVC-SZBZVVMNBX2D3VFK ! -s 100.96.0.0/11 -d 100.68.215.151/32 -p tcp -m comment --comment "capi-kubeadm-bootstrap-system/capi-kubeadm-bootstrap-webhook-service cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ -A KUBE-SVC-SZBZVVMNBX2D3VFK -m comment --comment "capi-kubeadm-bootstrap-system/capi-kubeadm-bootstrap-webhook-service -> 100.96.1.8:9443" -j KUBE-SEP-Z5LDPDXWJ7G5IJIQ -A KUBE-SVC-TCOU7JCQXEZGVUNU ! -s 100.96.0.0/11 -d 100.64.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ

-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns -> 100.96.0.3:53" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ONT5M3GKZCUX63CA -A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns -> 100.96.0.4:53" -j KUBE-SEP-X6Z4D2PLM3E4AFD7 -A KUBE-SVC-XI3EV6PAOLGPKNCY ! -s 100.96.0.0/11 -d 100.68.64.20/32 -p tcp -m comment --comment "tkg-system/tkr-conversion-webhook-service cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ

-A KUBE-SVC-XI3EV6PAOLGPKNCY -m comment --comment "tkg-system/tkr-conversion-webhook-service -> 100.96.1.14:9443" -j KUBE-SEP-EMM6VPN6RQ4BTZVT -A KUBE-SVC-Z4ANX4WAEWEBLCTM ! -s 100.96.0.0/11 -d 100.70.166.34/32 -p tcp -m comment --comment "kube-system/metrics-server:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ -A KUBE-SVC-Z4ANX4WAEWEBLCTM -m comment --comment "kube-system/metrics-server:https -> 100.96.1.4:4443" -j KUBE-SEP-FY6XAK6CPXJEDB4L

-A KUBE-SVC-ZUD4L6KQKCHD52W4 ! -s 100.96.0.0/11 -d 100.71.102.80/32 -p tcp -m comment --comment "cert-manager/cert-manager-webhook:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ -A KUBE-SVC-ZUD4L6KQKCHD52W4 -m comment --comment "cert-manager/cert-manager-webhook:https -> 100.96.1.5:10250" -j KUBE-SEP-WTGQJIGDMYOJ6KTB

These were all floating around in my cluster AFTER I deleted kube proxy

Here's my original debugging: you might find it useful if your trying to investigate failures in your kube proxy or king or antreaproxy or other changes to your k8s service proxying architecture.

I was seeing this in a cluster today

E0712 12:12:14.883784 1 cacher.go:450] cacher (vspheremachinetemplates.infrastructure.cluster.x-k8s.io): unexpected ListAndWatch error: failed to list infrastructure.cluster.x-k8s.io/v1alpha4, Kind=VSphereMachineTemplate: conversion webhook for infrastruc

ture.cluster.x-k8s.io/v1beta1, Kind=VSphereMachineTemplate failed: Post "https://capv-webhook-service.capv-system.svc:443/convert?timeout=30s": context deadline exceeded (Client.Timeout exceeded while awaiting headers); reinitializing... W0712 12:12:18.570987 1 dispatcher.go:187] Failed calling webhook, failing closed validation.vspheremachinetemplate.infrastructure.x-k8s.io: failed calling webhook "validation.vspheremachinetemplate.infrastructure.x-k8s.io": failed to call webhook: Post "

These seemed to only happen on

- Post "https://capv-webhook-service.capv-system.svc:443/convert?timeout=30s" ...

- Post "https://capi-webhook-service.capi-system.svc:443/convert?timeout=30s"...

So I wonder what is going on here? why are these web hooks failing? The services look existent

capi-system capi-webhook-service ClusterIP 100.70.159.142 <none> 443/TCP 3d9h cluster.x-k8s.io/provider=cluster-api

capv-system capv-webhook-service ClusterIP 100.69.250.28 <none> 443/TCP 3d9h cluster.x-k8s.io/provider=infrastructure-vsphere

Lets look at endpoints....

kubo@uOFLhGS9YBJ3y:~$ kubectl get endpoints -o wide -A E0712 12:16:36.748680 513955 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

E0712 12:16:36.752015 513955 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0712 12:16:36.755788 513955 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

E0712 12:16:36.759763 513955 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request NAMESPACE NAME ENDPOINTS AGE

...

capi-system capi-webhook-service 100.96.0.142:9443 3d10h

capv-system capv-webhook-service 100.96.0.134:9443 3d10h

...

Ok so we have services, and endpoints... why the hell is it hanging

100.70.159.142 -> 100.96.0.1.142

100.69.250.28 -> 100.96.0.1.134

Do pods work ?

First thing to do is check - do endpoints work ? Well... I can't exec into pods over my APIServer to see....

kubo@uOFLhGS9YBJ3y:~$ kubectl exec -t antrea-agent-gv76v -n kube-system /bin/bash kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

E0712 12:19:48.976536 514296 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request E0712 12:19:48.978008 514296 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

OK THERES A FIX TO THAT

kubectl delete apiservice v1beta1.metrics.k8s.io

Now, jumping into my Antrea pods ... bc Antrea has a shell thankfully

kubectl exec -t -i antrea-agent-q5ljs -n kube-system /bin/bash

root@tkg-mgmt-vsphere-20230709021218-md-0-5pzw8-c4bb7fc9fx6lq66dppdz:/# wget https://100.96.0.134:9443

--2023-07-12 12:24:29-- https://100.96.0.134:9443/

Connecting to 100.96.0.134:9443... connected.

### worked ^^^^^^ TCP connection was established to 9443

capv-system capv-controller-manager-855bd48957-hfcwp 1/1 Running 0 17h 100.96.0.134

So, the pod web hook for CAPV ... works ! the issue must be the service endpoint:

root@tkg-mgmt-vsphere-20230709021218-md-0-5pzw8-c4bb7fc9fx6lq66dppdz:/#

root@tkg-mgmt-vsphere-20230709021218-rgtp4-wpsk2:/# wget 100.69.250.28:9443 --2023-07-12 14:27:21-- http://100.69.250.28:9443/

Connecting to 100.69.250.28:9443... <--- HANGS :(

HUNG !!!!!! This is because well, I deleted my kube-proxy to see what happened.... AntreaProxy was running all the while so... I figured it would proxy things for me. But I guess not. Will have to see why?

IS DNS OK ?

I figured I should confirm wether coredns worked in this cluster, as that's another critical thing behind the service proxy.... yeah, it worked... tested this by adding log to the corefile./. Then after a while I just looked at logs for cordons. Found not much, so I ran kubernetes conformance via sonobuoy - - - and checked the logs

with this core file:

apiVersion: v1 data: Corefile: |

.:53 { log errors

health { lameduck 5s }

ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure

fallthrough in-addr.arpa ip6.arpa ttl 30 }

prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000

} cache 30 loop

reload loadbalance }

They looked ok

[INFO] 100.96.1.246:50278 - 43583 "AAAA IN affinity-nodeport-transition.services-8475.svc.cluster.local. udp 78 false 512" NOERROR qr,aa,rd 171 0.000278541s [INFO] 100.96.1.246:50278 - 43241 "A IN affinity-nodeport-transition.services-8475.svc.cluster.local. udp 78 false 512" NOERROR qr,aa,rd 154 0.000481075s

So WTF is going on....

Well... I messed w/ this cluster quite bit since I deleted kube proxy. Maybe it's related to Antrea not always properly proxying ClusterIPs over TCP but im not sure. Running sonobuoy it seemed like most of the tests passed actually....

kubo@uOFLhGS9YBJ3y:~$ sonobuoy status -n dns-testttttt PLUGIN STATUS RESULT COUNT PROGRESS

e2e running 1 Passed: 32, Failed: 1 systemd-logs complete 2

Well.... actually letting the E2E tests run longer I found a suspicious pattern, web hooks, and services , were all failing :

"failures": [

"[sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance]", "[sig-api-machinery] Namespaces [Serial] should ensure that all pods are removed when a namespace is deleted [Conformance]", "[sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin] should be able to convert from CR v1 to CR v2 [Conformance]",

"[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate configmap [Conformance]", "[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should be able to deny custom resource creation, update and deletion [Conformance]", "[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource with different stored version [Conformance]",

"[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing mutating webhooks should work [Conformance]", "[sig-network] Services should be able to create a functioning NodePort service [Conformance]", "[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate custom resource [Conformance]"

Moral of the story for me is that, I think for now TKG still needs a running kube proxy, but I have to investigate more, but I guess maybe there are other feature gates in antreaproxy https://github.com/antrea-io/antrea/blob/main/docs/feature-gates.md which I might need to enable.... ill dig more into that after asking my Antrea friends !

12.7.23

Can you delete kube-proxy from a TKG cluster? not really..... but some stuff does work... ?

If you have OLD IPTABLES RULES in a cluster for services, then those services might accidentally load balance you to pods that don't exist anymore. Deleting the tube-proxy wont automatically delete it's iptables rules !!!!

No comments:

Post a Comment