EnablevSphere Supervisor on a Cluster withNSX as the Networking Stack
vSphere Supervisor
on a Cluster withNSX
as the Networking StackThrough the
vSphere
Automation
APIs, you can enable a vSphere cluster for managing Kubernetes workloads. A cluster configured withNSX
supports runningvSphere Pod
andTKG
clusters.- Verify that your environment meets the system requirements for enablingvSphere Supervisoron the cluster. For more information about the requirements, see thevSphere IaaS Control Plane Concepts and Planningdocumentation.
- Verify that theNSXis installed and configured. See Configuring NSX for vSphere Supervisor.
- Create storage policies for the placement of pod ephemeral disks, container images, andSupervisorcontrol plane cache.
- Verify that DRS is enabled in fully automated mode and HA is also enabled on the cluster.
- Configure shared storage for the cluster. Shared storage is required for vSphere DRS, HA, and storing persistent volumes of containers.
- Verify that the user who you use to access thevSphere Automationservices has theModify cluster-wide configurationprivilege on the cluster.
- Create a subscribed content library on thevCenter Serversystem to accommodate the VM image that is used for creating the nodes of theTKGclusters.
To enable a vSphere cluster for Kubernetes workload management, you use the services under the
namespace_management
package.- Retrieve the IDs of the tag-based storage policies that you configured forvSphere Supervisor.Use theGET https://request to retrieve a list of all storage policies and then filter the policies to get the IDs of the policies that you configured for the<vcenter_ip_address_or_fqdn>/api/vcenter/storage/policiesSupervisor.
- Retrieve the IDs of the vSphere Distributed Switch and theNSX Edgecluster that you created when configuring theNSXforvSphere Supervisor.Use thePOST https://request to list all vSphere Distributed Switches associated with the specific vSphere cluster and then retrieve the ID of the Distributed Switch that you configured to handle overlay networking for the<vcenter_ip_address_or_fqdn>/api/vcenter/namespace-management/networks/nsx/distributed-switches?action=check_compatibilitySupervisor.Use thePOST https://request to retrieve a list of the created<vcenter_ip_address_or_fqdn>/api/vcenter/namespace-management/networks/nsx/edges?action=check_compatibilityNSX Edgeclusters for the specific vSphere cluster and associated with the specific vSphere Distributed Switch. Retrieve the ID of theNSX Edgecluster that has the tier-0 gateway that you want to use for the namespaces networking.
- Retrieve the ID of the port group for the management network that you configured for the management traffic.To list the visible networks available on thevCenter Serverinstance that match some criteria, use theGET https://request and then retrieve the ID of the management network you previously configured.<vcenter_ip_address_or_fqdn>/api/vcenter/namespace-management/clusters/<cluster_id>/networks
- Create aClusters.EnableSpecJSON object anddefinethe parameters of theSupervisorthat you want to create.You must specify the following required parameters of the enable specification:
- Storage policies settings and file volume support. The storage policy you set for each of the following parameters ensures that the respective object is placed on the datastore referenced in the storage policy. You can use the same or different storage policy for the different inventory objects.PropertyDescriptionephemeral_storage_policySpecify the ID of the storage policy that you created to control the storage placement of thevSphere Pods.image_storageSet the specification of the storage policy that you created to control the placement of the cache of container images.master_storage_policySpecify the ID of the storage policy that you created to control the placement of theSupervisorcontrol plane cache.Optionally, you can activate the file volume support by usingcns_file_config. See Enabling ReadWriteMany Support.
- Management network settings. Configure the management traffic settings for theSupervisorcontrol plane.PropertyDescriptionnetwork_providerSpecify the networking stack that must be used when theSupervisoris created. To use theNSXas the network solution for the cluster, setNSXT_CONTAINER_PLUGIN.master_management_networkEnter the cluster network specification for theSupervisorcontrol plane. You must enter values for the following required properties:
- network- Use the management network ID retrieved in Step 3.
- mode- SetSTATICRANGEorDHCPfor the IPv4 address assignment mode. TheDHCPmode allows an IPv4 address to be automatically assigned to theSupervisorcontrol plane by a DHCP server. You must also set the floating IP address used by the HA primary cluster by usingfloating_IP. Use the DHCP mode only for test purposes. TheSTATICRANGEmode, allows theSupervisorcontrol plane to have a stable IPv4 address. You can use it in a production environment.
- address_range- Optionally, you can configure the IPv4 addresses range for one or more interfaces of the management network. Specify the following settings:
- The starting IP address that must be used for reserving consecutive IP addresses for theSupervisorcontrol plane. Use up to 5 consecutive IP addresses.
- The number of IP addresses in the range.
- The IP address of the gateway associated with the specified range.
- The subnet mask to be used for the management network.
master_DNSEnter a list of the DNS server addresses that must be used from theSupervisorcontrol plane. If yourvCenter Serverinstance is registered with an FQDN, you must enter the IP addresses of the DNS servers that you use with the vSphere environment so that the FQDN is resolvable in theSupervisor. The list of DNS addresses must be specified in the order of preference.master_DNS_search_domainsSet a list of domain names that DNS searches when looking up for a host name in the Kubernetes API server. Order the domains in the list by preference.master_NTP_serversSpecify a list of IP addresses or DNS names of the NTP server that you use in your environment, if any. Make sure that you configure the same NTP servers for thevCenter Serverinstance, all hosts in the cluster, theNSX, andvSphere Supervisor. If you do not set an NTP server, VMware Tools time synchronization is enabled. - Workload network settings. Configure the settings for the networks for the namespaces. The namespace network settings provide connectivity tovSphere Pods and namespaces created in theSupervisor.PropertyDescriptionncp_cluster_network_specSet the specification for theSupervisorconfigured with theNSXnetworking stack. Specify the following cluster networking configuration parameters forNCPClusterNetworkEnableSpec:
- cluster_distributed_switch- The vSphere Distributed Switch that handles overlay networking for theSupervisor.
- nsx_edge_cluster- TheNSX Edgecluster that has tier-0 gateway that you want to use for namespace networking.
- nsx_tier0_gateway- The tier-0 gateway that is associated with the cluster tier-1gateway. You can retrieve a list ofNSXTier0Gatewayobjects associated with a particular vSphere Distributed Switch and determine the ID of the tier-0 gateway you want to set.
- namespace_subnet_prefix- The subnet prefix that defines the size of the subnet reserved for namespaces segments. Default is 28.
- routed_mode- The NAT mode of the workload network. If set tofalse:
- The IP addresses of the workloads are directly accessible from outside the tier-o gateway and you do not need to configure the egress CIDRs.
- File Volume storage is not supported.
true. - egress_cidrs- The external CIDR blocks from which theNSXManager assigns IP addresses used for performing source NAT (SNAT) from internalvSphere Pods IP addresses to external IP addresses. Only one egress IP address is assigned for each namespace in theSupervisor. These IP ranges must not overlap with the IP ranges of thevSphere Pods, ingress, Kubernetes services, or other services running in the data center.
- ingress_cidrs- The external CIDR blocks from which the ingress IP range for the Kubernetes services is determined. These IP ranges are used for load balancer services and Kubernetes ingress. All Kubernetes ingress services in the same namespace share a common IP address. Each load balancer service is assigned a unique IP address. The ingress IP ranges must not overlap with the IP ranges of thevSphere Pods, egress, Kubernetes services, or other services running in the data center.
- pod_cidrs- The internal CIDR blocks from which the IP ranges forvSphere Pods are determined. The IP ranges must not overlap with the IP ranges of the ingress, egress, Kubernetes services, or other services running in the data center. AllvSphere Pods CIDR blocks must be of at least /23 subnet size.
worker_DNSSet a list of the IP addresses of the DNS servers that must be used on the worker nodes. Use different DNS servers than the ones you set for theSupervisorcontrol plane.service_cidrSpecify the CIDR block from which the IP addresses for Kubernetes services are allocated. The IP range must not overlap with the ranges of thevSphere Pods, ingress, egress, or other services running in the data center.For the Kubernetes services and thevSphere Pods, you can use the default values which are based on the cluster size that you specify. - Supervisorsize. You must set a size to theSupervisorwhich affects the resources allocated to the Kubernetes infrastructure. The cluster size also determines default maximum values for the IP addresses ranges for thevSphere Pods and Kubernetes services running in the cluster. You can use theGET https://calls to retrieve information about the default values associated with each cluster size.<vcenter_ip_address_or_fqdn>/api/vcenter/namespace-management/cluster-size-info
- Optional. Associate theSupervisorwith the subscribed content library that you created for provisioningTKGclusters. See Creating, Securing, and Synchronizing Content Libraries for TKG Releases.To set the library, usedefault_kubernetes_service_content_libraryand pass the subscribed content library ID.
- EnablevSphere Supervisoron a specific cluster by passing the cluster enable specification to theClustersservice.
A task runs on
vCenter Server
for turning the cluster into aSupervisor
. Once the task completes, Kubernetes control plane nodes are created on the hosts that are part of the cluster enabled withvSphere Supervisor
. Now you can createvSphere Namespace
s.Create and configure namespaces on the
Supervisor
. See Create a vSphere Namespace.