How to load a custom TKR in TKG 1.6 i.e. for pci passthrough

This is an old post, about TKG 1.6 !!! For TKG 2.1+ see 

- https://docs-staging.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/2.3/using-tkg/workload-clusters-advanced-vsphere.html#windows 

- Or my unofficial notes in https://tanzu-install-annotated.readthedocs.io/en/latest/z9-win/?#windows-osimage-debugging which has up to date instructions on the "new" way of loading TKRs

TLDR

In 2.1, we Add OSImages to EXISTING TKRs….

• Start w say 100s of OsImages, now,  user specifies a K8s version
• THEN the TKR looks up what OS Images exist for THAT K8s version , now you only have 4 Osimages to pick from
• THEN look at the OS that the user wants (ubuntu?photon?windows?) now you only have 1 TKR to pick from

THIS POST was about how,  In TKG 1.6, 

we ran custom OS Images, by making Custom TKRs and by making an OSIMage object that connects the <VERSION> of the OVA to the KubernetesVersion and OS that a user wants to create We would do a magic trick where we would make a ConfigMap the ConfigMap would define, essentially, a NEW TKR that had a new OVA in it

UPDATE at kpngcon we came up w/ this script, mikaels idea !!!

bomTag=v1.23.8---vmware.2-tkg.1
name=v1.23.8---vmware.2-tkg.1-jay2

cat <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
 namespace: tkr-system
 annotations:
   bomImageTag: ${bomTag}
 name: ${name}
 labels:
   tanzuKubernetesRelease: ${name}
binaryData:
 # multiline binary content below...
 bomContent: |
EOF

base64 <$1 | sed 's,^,  ,'

Which you can use to create a TKR ........

Original post

Follow up directions from the last post about how TKRs get their Version data..... just some personal notes, formal docs are here: https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.6/vmware-tanzu-kubernetes-grid-16/GUID-build-images-linux.html 

... 

- getting an OVA from somewhere

- finding its VERSION

- putting that VERSION into a new TKR yaml

- storing that TKR yaml as a base64 

- putting that Baset64 into  a configmap that tkr-controller is monitoring..

HERE GOES

If you look at your OVA , you can extract it ... and edit the OVF file.  OVF Files are XML ! 

1) tar -xvf ubuntu-2004-efi-kube-v1.22.5+vmware.1.ova then

2) vi ubuntu-2004-efi-kube-v1.22.5+vmware.1.ova 

3) Now youll see theres a VERSION...

4) ok.. so whats going on in this version ? in the above file, its v1.22.5+vmware.1-tkg.4-gpu

5) cp the file for your tkr... to a new file, i.e. run:

cp ~/.config/tanzu/tkg/bom/tkr-bom-v1.23.8+vmware.2-tkg.1.yaml gpu-tkr.yaml 

6) that string will be input to our TKR we create for tanzu... in vim above, add the highlighted text:

ova:

- name: ova-photon-3

  osinfo:

    name: photon

    version: "3"

    arch: amd64

  version: v1.23.8+vmware.2-tkg.2-81d1a7892ad39f017fbaf59f9907cbe7

- name: ova-ubuntu-2004

  osinfo:

    name: ubuntu

    version: "20.04"

    arch: amd64

  version: v1.23.8+vmware.2-tkg.1-85a434f93857371fccb566a414462981

- name: ova-tkg-gpu

  osinfo:

    name: ubuntu-efi

    version: "1"

    arch: "amd64"

  version: v1.23.8+vmware.2-myorg.0

#       <Property ovf:key="VERSION" ovf:type="string" ovf:userConfigurable="false" ovf:value="v1.23.8+vmware.2-myorg.0"/>

7) Ok, now time to tell tkg about your new tkr...   

1. cat tkr-bom-v1.23.8---vmware.2-tkg.1-mycustomtkr.yaml | base64 -w 0 > b64
   

2. make this configmap

apiVersion: v1

kind: ConfigMap

metadata:

 name: v1.23.8---vmware.2-tkg.1-mycustomtkr

 labels:

   tanzuKubernetesRelease: v1.23.8---vmware.2-tkg.1-mycustomtkr

binaryData:

 bomContent: CAgICAgIHRhZzogdjEuNS4zCiAgYXdzLWVicy1jc2ktZHJpdmVyOgogIC0gdmVyc2lvbjogdjEuOC4wK3Ztd2FyZS4xCiAgICBpbWFnZXM6CiAgICAgIGF3c0Vic0NzaURyaXZlckltYWdlOgogICAgICAgIGltYWdlUGF0aDogY3NpL2F3cy1lYnMtY3NpLWRyaXZlcgogICAgICAgIHRhZzogdjEuOC4wX3Ztd2FyZS4xCiAgYXp1cmVkaXNrLWNzaS1kcml2ZXI6CiAgLSB2ZXJzaW9uOiB2MS4xOS4wK3Ztd2FyZS4xCiAgICBpbWFnZXM6CiAgICAgIGF6dXJlZGlza0NzaURyaXZlckltYWdlOgogICAgICAgIGltYWdlUGF0aDogY2x1c3Rlci1hcGkvYXp1cmVkaXNrLWNzaS1kcml2ZXIKICAgICAgICB0YWc6IHYxLjE5LjBfdm13YXJlLjEKICBieW9oLWs4cy11YnVudHUtMjAwNDoKICAtIHZlcnNpb246IHYxLjIzLjgrdm13YXJlLjItdGtnLjEKICAgIGltYWdlczoKICAgICAgYnlvaEltZ3BrZ0J1bmRsZToKICAgICAgICBpbWFnZVBhdGg6IGNsdXN0ZXItYXBpL2J5b2gtYnVuZGxlLXVidW50dV8yMC4wNC4xX3g4Ni02NF9rOHMKICAgICAgICB0YWc6IHYxLjIzLjhfdm13YXJlLjItdGtnLjEKICBjYWxpY29fYWxsOgogIC0gdmVyc2lvbjogdjMuMjIuMSt2bXdhcmUuMQogICAgaW1hZ2VzOgogICAgICBjYWxpY29DbmlJbWFnZToKICAgICAgICBpbWFnZVBhdGg6IGNhbGljby1hbGwv

3. Now, create the configmap.  itll get read in by the tkr-controller, bc of the label

kubectl create -f gpu-tkr.yaml.secret.cm.yamll -n tkr-system

kubo@tX34cWJGn61FF:~$ kubectl get tkr | grep True

v1.21.14---vmware.2-tkg.1              v1.21.14+vmware.2-tkg.1              True         14h

v1.22.11---vmware.2-tkg.1              v1.22.11+vmware.2-tkg.1              True         14h

v1.23.8---vmware.2-tkg.1               v1.23.8+vmware.2-tkg.1               True         14h

v1.23.8---vmware.2-tkg.1-mycustomtkr   v1.23.8+vmware.2-tkg.1               True         8s

Tada ~ ok now your new tkr controller has
- loaded the VERSION from the OVA into the bom of the tkr . 
- you can REFERENCE that TKR when doing tanzu cluster create --tkr combined with setting your OS.NAME and OS.VERSION and OS.ARCH fields 

- then , tanzu will grab the TKR, find the template in your vsphere instance, check that the template exists .... and inject that template name to capv provider