VMware Tanzu Basic: Customizing

I have done some research on the customization possibilities for the TKG service (integrated into vSphere) because I hit some limits (see previous blog post). Luckily I have an installation on my home lab where I can go some steps further than I would do on a productive system. The following write-up introduces some actions that won’t be supported by VMware and should only be followed if you know exactly what you’re doing and if you don’t rely on the official support. My PoC, however, can be installed on the management or a workload cluster, it’s up to you, how far you want to go.

Implemented Functionality

After having issues with custom CAs and insecure docker registries, the first goal of my PoC was to introduce a solution to this problem and if possible further reconfiguration possibilities that are not available out of the box, yet. I had a discussion with one of the PMs for Tanzu and he promised to come with a solution integrated into the vCenter UI.

The second part is reuse of some code I have written for a Swisscom project, it provides automated DNS record management for Active Directory DNS servers over WinRM. I’m using the watch API functionality of K8s and the possibility to generate API stubs in Java over Swagger.

Access to the Management Cluster (Danger Zone)

If you do not want to install something on the Management Cluster (not recommended!), you can skip this paragraph. The most important things can also be achieved by switching the Kube context to the namespace you have defined in vCenter.

I was quite surprised that it is possible to access all the Tanzu clusters, even the vCenter Management Cluster. You have to ssh into vCenter and execute


you’ll get the ssh PW and the Cluster IP. You can also look up the IPs in vCenter and login with root.

Copy the file


to your working environment and adapt the server address within this file to the cluster IP. Now you can access the management cluster as a cluster-admin by setting an environment variable:

export KUBECONFIG=<path-to-your-file>

Now you are able to inspect what is under the hood of these managed components!

Access to the Workload Clusters

I have configured a namespace called “niceneasy” and a workload cluster called “workload-01”. First I switch the Kube context and get all secrets in this namespace – btw: it is a namespace within the management cluster.

> kubens niceneasy
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "niceneasy".
> kubectl get secrets
NAME                              TYPE                                  DATA   AGE
default-token-9bnkn               kubernetes.io/service-account-token   3      3d23h
workload-01-antrea                kubernetes.io/tls                     3      3d23h
workload-01-ca                    Opaque                                2      3d23h
workload-01-ccm-token-j59vx       kubernetes.io/service-account-token   3      3d23h
workload-01-control-plane-sb9rq   cluster.x-k8s.io/secret               1      3d23h
workload-01-encryption            Opaque                                1      3d23h
workload-01-etcd                  Opaque                                2      3d23h
workload-01-kubeconfig            Opaque                                1      3d23h
workload-01-proxy                 Opaque                                2      3d23h
workload-01-pvcsi-token-72tvv     kubernetes.io/service-account-token   3      3d23h
workload-01-sa                    Opaque                                2      3d23h
workload-01-ssh                   kubernetes.io/ssh-auth                1      3d23h
workload-01-ssh-password          Opaque                                1      3d23h
workload-01-workers-lk46m-65nq6   cluster.x-k8s.io/secret               1      3d23h
workload-01-workers-lk46m-fstz4   cluster.x-k8s.io/secret               1      3d23h
workload-01-workers-lk46m-tgsc5   cluster.x-k8s.io/secret               1      3d23h

The following secrets are particularly interesting:

  • workload-01-kubeconfig contains the admin.conf for this cluster
  • workload-01-ssh contains a private key for SSH access
  • workload-01-ssh-password contains the password for SSH, user: vmware-system-user

These are the credentials which are needed by the approach of Eric Shanks and William Lam. I did similar tests but I hit the following two problems:

  • My first implementation was trying to restart containerd on the same node that the container was running on. This led to errors, systemctl came back with a message that some files have not been accessible.
  • I was looking for a very generic solution without messing with passwords in configurations.

That’s why I planned to install the component directly in the management cluster. Here you have full access to all necessary information and credentials and can solve the problem without exposing any secrets. In addition, the management cluster should be managing the nodes…

Now let’s extract the private key to the VMs for the workload cluster:

> kubectl get secret workload-01-ssh -o jsonpath='{.data.ssh-privatekey}' | base64 -d > ~/.ssh/workload-01-ssh
> chmod 600 ~/.ssh/workload-01-ssh
> kubectx workload-01
> kubectl get nodes -o wide
NAME                                         STATUS   ROLES    AGE    VERSION             INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                 KERNEL-VERSION       CONTAINER-RUNTIME
workload-01-control-plane-4cpqg              Ready    master   4d1h   v1.18.10+vmware.1     <none>        VMware Photon OS/Linux   4.19.154-3.ph3-esx   containerd://1.3.4
workload-01-workers-fs9dc-7c57f6d49b-hb2lz   Ready    <none>   4d1h   v1.18.10+vmware.1    <none>        VMware Photon OS/Linux   4.19.154-3.ph3-esx   containerd://1.3.4
workload-01-workers-fs9dc-7c57f6d49b-lxhqx   Ready    <none>   4d1h   v1.18.10+vmware.1     <none>        VMware Photon OS/Linux   4.19.154-3.ph3-esx   containerd://1.3.4
workload-01-workers-fs9dc-7c57f6d49b-ztjqh   Ready    <none>   4d1h   v1.18.10+vmware.1     <none>        VMware Photon OS/Linux   4.19.154-3.ph3-esx   containerd://1.3.4
> ssh vmware-system-user@ -i .ssh/workload-01-ssh

The first line extracts and decodes the key into a file that needs restricting permissions for use as an SSH identity key. The third line switches to the workload context and lists the nodes of the cluster. “-o wide” offers the IP of the VMs. And the last line does the ssh into the control-plane VM.

This extraction and decoding with jsonpath is quite handy. If you are not sure, how the structure of a secret looks like, just do a “-o yaml” to check the valid field names.

Management Component

Check out the code from GitHub and adapt the config files to your environment (see documentation). This time, I used the built-in kustomize utility, so if you have added the necessary kubeconf and account tokens, deploy it simply by issueing

kubectl apply -k k8s/base

from the root of the project. Kustomize will generate configmaps and secrets as needed.

You can deploy it as-is on the management cluster or you can roll it out on a workload cluster, but in this case, I would remove the host networking (lines 22 and 31 in deployment.yaml). I have installed Tanzu (very) Basic without NSX-T and it looks to me as if  the management cluster is running without an (accessible?) CNI. When I tried to deploy the component without host networking, an error of a missing NSX-T CNI was popping up. I assume this would be solved if the management cluster would run on NSX-T. For a workload cluster, you have to state a CNI implementation, hence you should not need host networking at all.

Limitations: currently only one  DNS server and only one workload cluster can be configured.

Node Customizer Component

In the folder k8s/daemonset you’ll find the component that will add CAs to each node and demonstrates some other tricks. You need to add a service account with a clusterrolebinding to privileged access in order to permit file modifications on the underlying VM. Adjust the referenced endpoint for the management component. It will create a load balancer and you can get the exposed IP by looking up the tanzu-customizer service (external IP).

1 Trackback / Pingback

  1. VMware Tanzu Basic: Customizing II - vDan's Blog Posts

Leave a Reply

Your email address will not be published.