Its easy to get vsphere config wrong, i figured i should dump all of a working vsphere / api objects alongside corresponding tkg mc config for reference. YMMV
TLDR Working inputs are here https://gist.github.com/jayunit100/2c5bac8d257965df032a0ed1a472c326
This is a shoddy list of notes i took when trying to figure out why tanzu framework didnt think i had the right OVAs in my cluster, tldr, the vappproperties from image-builder are scanned by the client ... and even if an ova has a certain name, the client will error out unless you set the VAPP properties to map to the TKR you are installing..... folks who stumble upon this blog post from a google search will know what im talking about :)
Note also that there are vApp properties that are assigned by image builder to underlying VMs... but i dont think this is used..... Ultimately its the vappproperty set by image builder that is used by tanzu framework when it looks for a new VM....
Also you need a VSPHERE_FOLDER to be a certain type of folder, (i think vm and template folder, not sure, am verifying this....)
one other screenshot that might be useful.......to understand where this version stuff comes from
An OVA can be tar -xvf'd to an ovf, where you can see its VERSION,
tanzu will then look to match that VERSION to a "TKR ova version" when it scans your available vm templates...
I ran into this today:
: vSphere template kubernetes version validation failed: unable to get or validate VM Template for given Tanzu Kubernetes release: unable to find VM Template associated with TanzuKubernetesRelease v1.23.8+vmware.2-tkg.1-rc.3. Please upload at least one VM Template from versions [v1.23.8+vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7,v1.23.8+vmware.2-tkg.1-85a434f93857371fccb566a414462981,v1.23.8+vmware.2-tkg.1-3e6b7a84930e5368c38aa867f998ced0]
Looking in the tanzu .config/ directory:
ova:
- name: ova-photon-3
osinfo:
name: photon
version: "3"
arch: amd64
version: v1.23.8+vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7
- name: ova-ubuntu-2004
osinfo:
name: ubuntu
version: "20.04"
arch: amd64
version: v1.23.8+vmware.2-tkg.1-85a434f93857371fccb566a414462981
Ok, so it looks like
- tanzu cli declares versions
- those versions are somehow "looked up" inside of vpshere, as things that are associated with the OVA name...
So the question is how does tanzu determine of a VM is associated with TanzuKubernetesRelease v1.23.8+vmware.2-tkg.1-rc.3 ?
well
tanzu management-cluster create -f ....
in pkg/v1/tkg/client/validate.go ....
case VSphereProviderName:
err := c.ConfigureAndValidateVsphereConfig(tkrVersion, options.NodeSizeOptions, options.VsphereControlPlaneEndpoint, skipValidation, nil)
calls
if err := c.ConfigureAndValidateVSphereTemplate(vcClient, tkrVersion, dc); err != nil {
- vcClient.GetAndValidateVirtualMachineTemplate which then calls out to
vms, err := c.GetVirtualMachineImages(context.TODO(), dcMOID)
GetVirtualMachineImages then calls getVMMetadata ....
and THATS WHERE THE VAPP IS PARSED !!!!!!!!!!!
func (c *DefaultClient) getVMMetadata(vm *mo.VirtualMachine) (ovaVersion, distroName, distroVersion, distroArch string) {
if vm.Config == nil {
return
}
if vm.Config.VAppConfig == nil {
return
}
// The VappConfig information
vmConfigInfo := vm.Config.VAppConfig.GetVmConfigInfo()
if vmConfigInfo == nil {
return
}
for i := range vmConfigInfo.Property {
p := &vmConfigInfo.Property[i]
if p.Category == VMPropertyCategoryCAPI && p.Id == OVAVersionPropertyID {
ovaVersion = p.DefaultValue
}
ultimately my vsphere mc create yaml looks like this :
CLUSTER_PLAN: prod
CNI: antrea
INFRASTRUCTURE_PROVIDER: vsphere
KUBERNETES_VERSION: v1.23.8+vmware.2
OS_ARCH: amd64
OS_NAME: ubuntu
#OS_VERSION: '3' <-- dont use this w/ UBUNTU
_VSPHERE_CONTROL_PLANE_ENDPOINT: 10.23.53.141
MHC_FALSE_STATUS_TIMEOUT: 40m
VSPHERE_SERVER: "10.23.53.4"
VSPHERE_INSECURE: "true"
VSPHERE_DATACENTER: "Edge2"
VSPHERE_DATASTORE: "datastore2"
DEPLOY_TKG_ON_VSPHERE7: true
### These are checked after the initial stuff is set.
VSPHERE_RESOURCE_POOL: "RPJ"
VSPHERE_SSH_AUTHORIZED_KEY: "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6VfBKd6hqd5h7k5f+AtjJSV1hdW5u9/3uAolK3SD2/5GD9+rn+FMSdbtkeaKuuVJPi2HjnsVMO+r8WcuyN5ZSYHywiSoh4S7PamAxra1CLISsFHPYFlGrtdHC70wnoT7+/wAJk2D3CYkCNMWIxs5eR0cefDOytipBfDplhkJByyrcnXuhI8St3XJzpjlXu454diJOxfsk6axanWLOr/WZFmUi1U6V4gRE7XtKG9WFUm1bmNgkgd7lehKzi+isTjnI+b4tnD0yIzKFcsgIvLdGJTI6Lluj33CeBHIocwu0LbvowTyYSqhP6DzGhGuKfK9rMnJh/ll0Bnu1xf/ok0NSQ== Jpeerindex@doolittle-5.local"
VSPHERE_FOLDER: "Edge2-Cluster"
### VSPHERE_FOLDER:
CONTROL_PLANE_MACHINE_COUNT: 1
WORKER_MACHINE_COUNT: 1
WHAT ABOUT CUSTOM TKRs
So, this tells us some of the internals but concretely, one reason you might have been reading this is that your customizing Tanzu's OS for the kubelets...... For that you need to be thinking about customizing your TKR... Now, there is such a thing as a custom TKR which takes the VERSION field out of the XML files that are in an OVA, and adds them into a file which then gets picked up by the tkr controller - - so that you can tanzu create --tkr=... a new cluster with a custom OVA. Under the hood, you have to read the VERSION field from XML in the OVF file and make sure it matches the value in the TKR you create... detailed example on how to do that is here:
https://jayunit100.blogspot.com/2022/09/how-to-load-custom-tkr-in-tkg-16-ie-for.html (the official docs are also a good place to read about this https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.6/vmware-tanzu-kubernetes-grid-16/GUID-build-images-linux.html )
No comments:
Post a Comment