EKS, windows, and k8s e2es

 Weve recently found that amazon EKS is highly dependent on 

kubernetes.io/os : windows

to be labelled.  The reason is that

- in EKS linux, the CNI is capable of doing its own ipam

- in EKS windows, the CNI doesnt manage its own IP Addresses! 

Thus, EKS contrlplane does IPAM for the windows nodes, and then uses the pod spec as a communication channel to send along an IP address that the CNI gives back to containerd !!! 

Of course... this isnt news --- This is well documented in the EKS install docs for windows...and in general its advised to do this for all pods, anyways, but some of the k8s e2e tests dont do this - and other vendors are happy to accidentally schedule pods to windows - which is fine if you have a windows only worker nodes cluster (i..e. common in VMWare tanzu bc we didnt support hybrid clusters early on, for this very reason :)) .  

Anyways, hope to see EKS CNI become more flexible w/ scheduling pods to windows only worker clusters soon.  The EKS team has been awesome working w/ us upstream in sig-windows to understand this.  Thanks to Tatenda and Kulwant !!!