How Antrea/CSI/CPI is Configured in the future of "Classy" Kubernetes clusters for Cluster API

In new ClusterClass implementations of TKG, youll be able to make a specific object for all the customizations of things like

- cni

- cpi 

- csi

In general each of these object types is copied by TKG, from a starter template which (if pre-existing, is never copied)... Thus, the process of customizing TKG clusters is easily hackable - i.e. its about finiding a specific Config that you want, adding it to a namespace so that the default isnt used, and then creating a cluster.

You can see the default settings using kubectl to get the Config types to build your own, i.e. 

ubuntu-ci-1804-001 :: ~ » kubectl get VSphereCPIConfig  -A

NAMESPACE    NAME                                  NAME   MODE

tkg-system   mgmt-2                                       vsphereCPI

tkg-system   v1.23.8---vmware.2-tkg.1-zshippable          vsphereCPI

ubuntu-ci-1804-001 :: ~ » kubectl get antreaconfig -A

NAMESPACE    NAME                                  TRAFFICENCAPMODE   DEFAULTMTU   ANTREAPROXY   ANTREAPOLICY   SECRETREF

tkg-system   mgmt-2-antrea-package                 encap                           true          true           mgmt-2-antrea-data-values

tkg-system   v1.23.8---vmware.2-tkg.1-zshippable   encap                           true          true

and so on

Example antrea config

apiVersion: cni.tanzu.vmware.com/v1alpha1

kind: AntreaConfig

metadata:

  creationTimestamp: "2022-09-09T18:27:54Z"

  generation: 1

  labels:

    tkg.tanzu.vmware.com/cluster-name: mgmt-2

    tkg.tanzu.vmware.com/package-name: antrea.tanzu.vmware.com.1.5.3---tkg.1-advanced-zshippable

  name: mgmt-2-antrea-package

  namespace: tkg-system

  ownerReferences:

  - apiVersion: cluster.x-k8s.io/v1beta1

    kind: Cluster

    name: mgmt-2

    uid: 0ce8e78c-2cb4-4afb-9c3e-551bfdb73d9e

  - apiVersion: run.tanzu.vmware.com/v1alpha3

    blockOwnerDeletion: true

    controller: true

    kind: ClusterBootstrap

    name: mgmt-2

    uid: 11f198b6-20b6-4f50-9ea0-887cc38f3af1

  resourceVersion: "3174"

  uid: 1cbda6ab-89db-4f9d-83f6-f45f0f610a59

spec:

  antrea:

    config:

      defaultMTU: ""

      disableUdpTunnelOffload: false

      featureGates:

        AntreaIPAM: false

        AntreaPolicy: true

        AntreaProxy: true

        AntreaTraceflow: true

        Egress: false

        EndpointSlice: true

        FlowExporter: false

        Multicast: false

        NetworkPolicyStats: false

        NodePortLocal: false

        ServiceExternalIP: false

      kubeAPIServerOverride: encap

      noSNAT: false

      tlsCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384

      trafficEncapMode: encap