Frequently Asked Questions

This document answers some frequently asked questions for

AKO

How do I clean up all my configs?

The key deleteConfig in the data section of

AKO

configmap can be used to clean up the setup. Edit

AKO

configmap and set

deleteConfig: true

to delete ako created objects in

VMware NSX Advanced Load Balancer

. After the flag is set in configmap, annotation AviObjectDeletionStatus is added in the

AKO

statefulset with the value Started.

annotations: AviObjectDeletionStatus: Started

After all relevant objects gets deleted from

VMware NSX Advanced Load Balancer

, the value of the annotation is changed to Done.

annotations: AviObjectDeletionStatus: Done

To re-create the objects in

VMware NSX Advanced Load Balancer

, the configmap has to be edited to set

deleteConfig: false

How is the shared virtual service lifecycle controlled?

AKO

follows hostname-based sharding to sync multiple ingresses with same hostname to a single virtual service. When an Ingress object is created with multiple hostnames,

AKO

generates an md5 hash using the hostname and the Shard VS number. This uniquely maps an FQDN to a given Shared virtual service and avoids DNS conflicts. During initial clean bootup, if the Shared virtual service does not exist in

VMware NSX Advanced Load Balancer

AKO

creates the same and then patches the ingress FQDN to it either in the form of a pool (for insecure routes) or in the form of an SNI child virtual service (for secure routes).

The Shared virtual services are not deleted if all FQDNs mapped to it are removed from Kubernetes. However, if the user wants

AKO

to delete unused shared VSes - a pod restart is required that would evaluate the VS and delete it appropriately.

Even though the unused shared VSes are deleted, the shared virtual service VIPs are retained to regain the same FQDN to virtual service VIP mapping. To delete the retained shared virtual service VIPs, the user has to manually delete them from the Controller UI or shell.

How are virtual services sharded?

If you create ingress with an insecure host/path combination, the

AKO

creates a corresponding

VMware NSX Advanced Load Balancer

pool object and patches the pool on one of the existing shard virtual services. The shard virtual service has a DataScript associated with it that reads the host/path of the incoming requests and appropriately selects a pool by matching it with the priority label specified for each pool member (corresponding to a host/path combination).

For secure ingresses, an SNI virtual service is created which although is a dedicated virtual service, does not have any IP addresses associated with it. The SNI virtual service is a child to a parent virtual service and is created based on the secret object specified in the ingress file against the host/path that is meant to be accessed securely.

How do I decide the size of the shard virtual service?

In the current

AKO

model, the Shard VS size is an enum. It allows 3 pre-fixed sets of values:

LARGE (8 virtual services)

MEDIUM (4 virtual services)

SMALL (1 virtual service)

DEDICATED (Shard Mode is deactivated, dedicated virtual services would be created per hostname)

The decision of selecting one of these sizes for Shard virtual service depends on the size of the Kubernetes cluster’s ingress requirements. It is recommended to always go with the highest possible Shard virtual service number that is(LARGE) to account for expansion in the future.

Can I change the Shard VS number?

To Shard to virtual services,

AKO

uses a sharding mechanism that is driven by the hostname of each rule within an ingress object. This ensures that a unique hostname is always sharded consistently to the same virtual service. Since the sharding logic is determined by the number of Shard virtual services, changing the Shard VS number has the potential hazard of disrupting an existing cluster’s already synced objects. Hence it’s recommended that the Shard VS numbers are not changed once fixed.

How do I alter the Shard VS number?

Altering the shard VS number is considered as disruptive. This is because dynamic re-adjustment of shard numbers might re-balance the ingress to virtual service mapping. Hence, if you want to alter the shard VS number, first delete the older configmap and trigger a complete cleanup of the virtual services in the Controller. Then, edit the configmap and restart

AKO

What happens if the number of DNS records exceeds a Shard VS?

Currently, the number of A records allowed per virtual service is

1000

. If a shard size of

SMALL

is selected and the number of A records through the Ingress objects exceed

1000

, then a greater shardSize has to be configured through the shardSize knob. Alternatively, one can create a separate IngressClass for a set of Ingress objects and specify a shardSize in the AviInfraSettings CRD which would allow

AKO

to place the A records scoped to the VS that is mapped to the IngressClass.

What is the use of static routes?

Static routes are created with cluster name as the label. While deploying

AKO

the admin or the operator decides a Service Engine Group for a given Kubernetes cluster. The same labels are tagged on the routes of this

AKO

cluster. These routes are pushed to the Service Engines created on the Service Engine Group. The static routes map each pod CIDR with the Kubernetes node’s IP address. However, for static routes to work, the Service Engines must be L2 adjacent to your Kubernetes nodes.

What happens if I have the same SNI host across multiple namespaces?

The ingress API does not prohibit the user from creating the same SNI hostname across multiple namespaces.

AKO

will create 1 SNI virtual service and gather all paths associated with it across namespaces to create corresponding switching rules. However, the user needs to denote each ingress with the TLS secret for a given hostname to qualify the host for the SNI virtual service.

Consider the below example:

Ingress 1 (default namespace) --> SNI hostname --> foo.com path: /foo, Secret: foo 
Ingress 1 (foo namespace) --> SNI hostname --> foo.com path: /bar, Secret: foo

In this case, only one SNI virtual service will be created with sslkeyandcertificate as

foo

However if the following happens, the behavior of the SNI virtual service would be indeterministic, since the secrets for the same SNI are different:

Ingress 1 (default namespace) --> SNI hostname --> foo.com path: /foo, Secret: foo 
Ingress 1 (foo namespace) --> SNI hostname --> foo.com path: /bar, Secret: bar

This is not supported.

What out-of-band operations can I do on the objects created by
AKO
?

AKO

runs a refresh cycle that currently just refreshes the cloud object parameters. However, if some out-of-band operations are performed on objects created by

AKO

by directly interacting with the

VMware NSX Advanced Load Balancer

APIs,

AKO

might not always be able to mitigate an error caused due to this.

AKO

has the best effort, retry layer implementation that would try to detect a problem (For example, an SNI VS deleted from the

VMware NSX Advanced Load Balancer

UI), but it is not guaranteed to work for all such manual operations.

Upon reboot of

AKO

, a full reconciliation loop is run and most of the out-of-band changes are overwritten with

AKO

’s view of the intended model. This does not happen in every full sync cycle.

What is the expected behavior for the same host/path combination across different secure/insecure ingresses?

The ingress API allows users to add duplicate hostpaths bound to separate backend services. For example,

Ingress1 (default namespace) --> foo.com path: /foo, Service: svc1

Ingress2 (default namespace) --> foo.com path: /foo, Service: svc2

Also, ingress allows you to have a mix of secure and insecure hostpath bound to the same backend services like:

Ingress1 (default namespace) --> SNI hostname --> foo.com path: /foo, Secret: secret1

Ingress2 (default namespace) --> foo.com path: /foo, Service: svc2

AKO

does not explicitly handle these conditions and would continue syncing these objects on the

NSX Advanced Load Balancer Controller

, but this might lead to traffic issues.

AKO

does the best effort of detecting some of these conditions by printing them in logs. A sample log statement looks like this:

key: Ingress/default/ingress2, msg: Duplicate entries found for hostpath
default/ingress2: foo.com/foo in ingresses: ["default/ingress1"]

What happens to static routes if the Kubernetes nodes are rebooted/shutdown?

AKO

programs a static route for every node IP and the pod CIDR associated with it. Even though node state changes to NotReady, in Kubernetes this configuration is stored in the node object and does not change when the node rebooted/shutdown. Hence,

AKO

will not remove the static routes until the Kubernetes node is completely removed from the cluster.

Can I point my ingress objects to a service of type `Loadbalancer`?

No. The ingress objects must point to the service of type clusterIP. Loadbalancer services either point to an ingress Controller pod if one is using an in-cluster ingress controller or they can directly point to application PODs that need layer 4 load-balancing.

In a configuration where the ingress objects are pointing to services of the type load balancer,

AKO

’s behavior would be indeterministic.

What happens when
AKO
fails to connect to the
NSX Advanced Load Balancer Controller
while booting up?

AKO

would stop processing the Kubernetes objects and no update would be made to the

NSX Advanced Load Balancer Controller

. After the connection to the Controller is restored,

AKO

pod has to be rebooted. This can be done by deleting the exiting POD and ako deployment would bring up a new pod, which would start processing Kubernetes objects after verifying connectivity to the Controller.

What happens if we create ingress objects in an OpenShift environment?

AKO

does not process ingress objects in an OpenShift environment. If any route corresponding to the ingress object is found,

AKO

would process that route.

What are the virtual services for passthrough routes or ingresses?

A set of shared virtual services are created for passthrough hosts to listen on port 443 to handle secure traffic using L4 DataScript. These virtual services have names of the format

‘cluster-name’-Shared-Passthrough-‘shard-number’

. The number of shards can be configured using the flag passthroughShardSize while installation using Helm. These virtual services would listen on port 443 and have one L4 SSL DataScript each. Name of the virtual service would be of the format

clustername–‘Shared-Passthrough’-shardnumber

. The number of shards can be configured using the flag passthroughShardSize while installing using Helm.

What happens if `insecureEdgeTerminationPolicy` is set to
redirect
for a `passthrough` route?

A set of shared virtual services are created for pass through hosts to listen on port 443 to handle secure traffic using L4 datascript. These virtual services have names of the format

‘cluster-name’-Shared-Passthrough-‘shard-number’

. Number of shards can be configured using the flag passthroughShardSize while installing using Helm. For passthrough routes, the supported values for insecureEdgeTerminationPolicy are

None

and

Redirect

. To handle insecure traffic for passthrough routes, a set of shared virtual services are created with names of the format

'cluster-name'-Shared-Passthrough-'shard-number'-insecure

. These virtual services listen on port 80. If for any passthrough route, the insecureEdgeTerminationPolicy is found to be

Redirect

, then an HTTP Policy is configured in the insecure passthrough shared virtual service to send appropriate response to an incoming insecure traffic.

How to debug
Invalid input detected
errors?

AKO

reboots and retries some of the invalid input errors. Look out for the following cases in the logs:

If an invalid cloud name is given in

values.yaml

or if ipam_provider_ref is not set in the vCenter and No Access clouds.

If the same Service Engine group is used for multiple clusters for vCenter and No Access clouds in Cluster IP mode. This happens as

AKO

expects unique SE group per cluster if routes are configured by

AKO

for POD reachability. Look for the

labels does not match with cluster name

message in the logs which points to two clusters using the same Service Engine Group.

Some of the pool servers in the NodePort mode of
AKO
are down. How is this fixed?

The default behavior for

AKO

is to populate all the Node IP as pool server. If master node is not schedulable then, it will be marked down. nodePortSelector can be used to specify the labels for the node. In that case, all the node with that label will be picked for the pool server. If the master node is not schedulable then, remove the nodePortSelector label for the master node.

Can we create a secure route with edge/reencrypt termination without key or certificate?

For secure routes having termination type

edge/reencrypt

, key and certificate must be specified in the spec of the route.

AKO

would not handle routes of these types without key and certificate.

What happens if a route is created with multiple backends having the same service name?

AKO

rejects those routes, as each backend must be unique with its own weight. Multiple backends having same service would make the weight calculation indeterministic.

Is the NodePortLocal feature available for all CNIs?

No. The feature NodePortLocal can be used only with Antrea CNI and the feature must be enabled in Antrea feature gates.

Can we use kubernetes Service of type
NodePort
as backend of an Ingress in
NodePortLocal
mode?

No. Users can only use service of type

ClusterIP

as backend of Ingresses in this mode.

Can we use serviceType
NodePort
or
ClusterIP
in
AKO
, if the CNI type is
Antrea
and `NodePortLocal` feature is enabled in Antrea?

Yes.

AKO

would create the

NSX Advanced Load Balancer

objects based on the relevant service Type set in

AKO

What are the steps for ServiceType change?

The serviceType in

AKO

can be changed from ClusterIP to NodePortLocal or NodePort. The serviceType change is considered disruptive. Hence before the serviceType change, all the existing

AKO

configuration must be deleted. To do this, follow the steps given below:

Set the deleteConfig flag to

True

Wait for

AKO

to delete all the relevant

NSX Advanced Load Balancer

configuration and update the deletion status in

AKO

’s statefulset status.

Change the serviceType.

Set the deleteConfig flag to

False

Reboot

AKO

For example, during the change of serviceType from ClusterIP to NodePortLocal, the deleteConfig flag will:

Delete the static routes.

Delete the SE group labels.

Can the service Type in
AKO
be changed dynamically?

No. After changing the service type,

AKO

has to be rebooted and all objects which are not required, would be deleted as part of the reboot process.

If service Type is changed from NodePortLocal, will
AKO
remove NPL annotation from the Services automatically?

No. After changing the service type, the users have to remove NPL annotation from the Services themselves.

How can I see the marker labels associated with an object on the
NSX Advanced Load Balancer Controller
?

Markers, associated with an

NSX Advanced Load Balancer

object, are visible on the

NSX Advanced Load Balancer

UI and the shell. To view the markers through the UI,

Edit an object like a virtual service or pool or pool group.

Navigate to the

Advanced

tab.

The Role-Based Access Control (RBAC) shows the markers associated with the object.

To view markers for an object through the Shell, use the following command.

[avi-shell] show <object type> <object name>

For example,

[avi-shell] show virtualservice kubernetes--Shared-L7-EVH-0
+------------------------------------+----------------------------------------------------------------------------------+
| Field                              | Value                                                                            |
+------------------------------------+----------------------------------------------------------------------------------+
| uuid                               | virtualservice-bc4e964a-af79-4b8f-91b2-de0e7ee9388d                              |
| name                               | kubernetes--Shared-L7-EVH-0                                                      |
| enabled                            | True                                                                             |
| vsvip_ref                          | kubernetes--Shared-L7-EVH-0                                                      |
| use_vip_as_snat                    | False                                                                            |
|        .                           | .                                                                                |
|        .                           | .                                                                                |
|        .                           | .                                                                                |
|        .                           | .                                                                                |
|        .                           | .                                                                                |
| markers[1]                         |                                                                                  |
|   key                              | clustername                                                                      |
|   values[1]                        | kubernetes                                                                       |
| allow_invalid_client_cert          | False                                                                            |
| vh_type                            | VS_TYPE_VH_ENHANCED                                                              |
+------------------------------------+----------------------------------------------------------------------------------+

What is the difference between `AKOSetttings.namespaceSelector` and `AKOSettings.blockedNamespaceList`? What are the scenarios to use one of them?

AKOSetttings.namespaceSelector allows

AKO

to process Ingress/Routes, L4 services from the namespaces that have given labels.

AKO

blocks processing Ingress/Routes and L4 services from the namespaces that does not have given labels, hence prevents

NSX Advanced Load Balancer

object creation. However,

AKO

also processes other objects (such as secrets, endpoints, and, so on) from incorrect or unlabelled namespaces.

AKOSettings.blockedNamespaceList is a list of namespaces specified upfront from which

AKO

will not process any objects, such as ingress/route, L4 services, secrets, endpoints, and so on. So, AKOSettings.blockedNamespaceList is a broader way of blocking unwanted objects compared to AKOSetttings.namespaceSelector. AKOSettings.blockedNamespaceList currently does not support regex, so the user must specify each namespace.

Following are the few scenarios to consider AKOSettings.blockedNamespaceList:

If the list of namespaces to be blocked is known to the user upfront (For example, system-namespaces), users typically do not deploy any application. Any new addition of namespace to this list will require an

AKO

reboot.

There are many objects (such as secret, pods, endpoint, and, so on) in unwanted namespaces that are consuming

AKO

processing time.

Following are few scenarios to consider AKOSetttings.namespaceSelector:

In a cluster, namespace churn is more frequent. Therefore, using a namespace selector is a better way of processing only wanted namespaces.

One of the ways to use AKOSetttings.namespaceSelector and AKOSettings.blockedNamespaceList effectively are:

Use AKOSettings.blockedNamespaceList for system-generated namespaces where users typically do not deploy any application.

Use AKOSetttings.namespaceSelector for user-defined namespaces.

What is the minimum stable Kubernetes version which supports SCTP protocol?

The minimum stable Kubernetes version which supports SCTP protocol is 1.20.

What is the minimum
NSX Advanced Load Balancer Controller
version which supports the SCTP protocol?

The minimum

NSX Advanced Load Balancer Controller

version which supports the SCTP protocol is 22.1.3.

What is the minimum
AKO
version which supports the SCTP protocol?

The minimum

AKO

version which supports the SCTP protocol is 1.9.1.

How to manually override the Active
AKO
when HA is enabled?

Trigger a deletion of the Active

AKO

pod, and the passive

AKO

pod automatically comes up as the new active

AKO

Can the user scale beyond two instances of
AKO
in HA?

Currently,

AKO

beyond two instances has not been tested. Hence, it is not supported.

What happens during a Split brain scenario in HA?

AKO

detects this during the periodic refresh of the lease lock object and makes itself a passive

AKO

With
AKO
version 1.10.3, the field networkUUID has been added as part of vipNetworkList and nodeNetworkList. Does it impact existing AviInfra objects?

On upgrading from an earlier version of

AKO

1.10.3 or later, always upgrade the CRD schemas as part of upgrade process. There is no impact on existing AviInfraSetting objects and the user does not require to do changes to existing objects.

AKO

accepts either networkName or networkUUID as part of these fields. If there are network duplicates on the

NSX Advanced Load Balancer

side,

AKO

recommends the use of networkUUID while creating new AviInfrasetting objects.

How can I identify networkUUID for the networkName?

To identify networkUUID for given networkName,

NSX Advanced Load Balancer

through a browser.

On the browser, enter the fqdn as

https://<controller-ip-or-fqdn>/api/network?name=<NameOfNetwork>

Choose the uuid field of one of the retrieved records.

In case if multiple records are retrieved,use switch_name and/or configured_subnet fields of the record, to choose the correct network

Which of the
AKO
CRDs have been migrated to the `v1beta1` version?

With

AKO

1.11.1, the schemas of the following AKO CRDs have been migrated from version v1alpha1 to v1beta1.

AviInfrasetting

Hostrule

HttpRule

In my current cluster, there are hostrule, aviinfrasetting, and httprule CRD objects in the `v1alpha1` version. What changes or impacts should I expect if I upgrade
AKO
to version 1.11.1 and above?

AKO version 1.11.1 supports CRD objects in v1alpha1 and v1beta1 version. So existing objects will be processed by

AKO

It is recommended to create new CRD objects inv1beta1 version. Also, transition the existing objects to v1beta1 version since AKO will deprecate v1alpha1 in future releases.

How to resolve the error: "...failed to list v1beta1.HostRule: the server could not find the requested resource (get hostrules.ako.vmware.com)" that is displayed in
AKO
logs after upgrading to
AKO
version 1.11.1 or higher?

During

AKO

upgrade, the first step is to apply the latest CRD.

The following error message is displayed when this step is skipped:

2023-09-21T05:08:47.280Z INFO logr/logr.go:261 github.com/vmware/load-balancer-and-ingress-services-for-kubernetes/internal/k8s/controller.go:1389: failed to list *v1beta1.HostRule: the server could not find the requested resource (get hostrules.ako.vmware.com)

To explicitly update CRD, see Upgrade AKO.

Content feedback and comments

Avi Kubernetes Operator Guide 1.11

Frequently Asked Questions

How do I clean up all my configs?

How is the shared virtual service lifecycle controlled?

How are virtual services sharded?

How do I decide the size of the shard virtual service?

Can I change the Shard VS number?

How do I alter the Shard VS number?

What happens if the number of DNS records exceeds a Shard VS?

What is the use of static routes?

What happens if I have the same SNI host across multiple namespaces?

What out-of-band operations can I do on the objects created by AKO?

What is the expected behavior for the same host/path combination across different secure/insecure ingresses?

What happens to static routes if the Kubernetes nodes are rebooted/shutdown?

Can I point my ingress objects to a service of type Loadbalancer?

What happens when AKO fails to connect to the NSX Advanced Load Balancer Controller while booting up?

What happens if we create ingress objects in an OpenShift environment?

What are the virtual services for passthrough routes or ingresses?

What happens if insecureEdgeTerminationPolicy is set to redirect for a passthrough route?

How to debug Invalid input detected errors?

Some of the pool servers in the NodePort mode of AKO are down. How is this fixed?

Can we create a secure route with edge/reencrypt termination without key or certificate?

What happens if a route is created with multiple backends having the same service name?

Is the NodePortLocal feature available for all CNIs?

Can we use kubernetes Service of type NodePort as backend of an Ingress in NodePortLocal mode?

Can we use serviceType NodePort or ClusterIP in AKO, if the CNI type is Antrea and NodePortLocal feature is enabled in Antrea?

What are the steps for ServiceType change?

Can the service Type in AKO be changed dynamically?

If service Type is changed from NodePortLocal, will AKO remove NPL annotation from the Services automatically?

How can I see the marker labels associated with an object on the NSX Advanced Load Balancer Controller?

What is the difference between AKOSetttings.namespaceSelector and AKOSettings.blockedNamespaceList? What are the scenarios to use one of them?

What is the minimum stable Kubernetes version which supports SCTP protocol?

What is the minimum NSX Advanced Load Balancer Controller version which supports the SCTP protocol?

What is the minimum AKO version which supports the SCTP protocol?

How to manually override the Active AKO when HA is enabled?

Can the user scale beyond two instances of AKO in HA?

What happens during a Split brain scenario in HA?

With AKO version 1.10.3, the field networkUUID has been added as part of vipNetworkList and nodeNetworkList. Does it impact existing AviInfra objects?

How can I identify networkUUID for the networkName?

Which of the AKO CRDs have been migrated to the v1beta1 version?

In my current cluster, there are hostrule, aviinfrasetting, and httprule CRD objects in the v1alpha1 version. What changes or impacts should I expect if I upgrade AKO to version 1.11.1 and above?

How to resolve the error: "...failed to list v1beta1.HostRule: the server could not find the requested resource (get hostrules.ako.vmware.com)" that is displayed in AKO logs after upgrading to AKO version 1.11.1 or higher?

What out-of-band operations can I do on the objects created by
AKO
?

Can I point my ingress objects to a service of type `Loadbalancer`?

What happens when
AKO
fails to connect to the
NSX Advanced Load Balancer Controller
while booting up?

What happens if `insecureEdgeTerminationPolicy` is set to
redirect
for a `passthrough` route?

How to debug
Invalid input detected
errors?

Some of the pool servers in the NodePort mode of
AKO
are down. How is this fixed?

Can we use kubernetes Service of type
NodePort
as backend of an Ingress in
NodePortLocal
mode?

Can we use serviceType
NodePort
or
ClusterIP
in
AKO
, if the CNI type is
Antrea
and `NodePortLocal` feature is enabled in Antrea?

Can the service Type in
AKO
be changed dynamically?

If service Type is changed from NodePortLocal, will
AKO
remove NPL annotation from the Services automatically?

How can I see the marker labels associated with an object on the
NSX Advanced Load Balancer Controller
?

What is the difference between `AKOSetttings.namespaceSelector` and `AKOSettings.blockedNamespaceList`? What are the scenarios to use one of them?

What is the minimum
NSX Advanced Load Balancer Controller
version which supports the SCTP protocol?

What is the minimum
AKO
version which supports the SCTP protocol?

How to manually override the Active
AKO
when HA is enabled?

Can the user scale beyond two instances of
AKO
in HA?

With
AKO
version 1.10.3, the field networkUUID has been added as part of vipNetworkList and nodeNetworkList. Does it impact existing AviInfra objects?

Which of the
AKO
CRDs have been migrated to the `v1beta1` version?

In my current cluster, there are hostrule, aviinfrasetting, and httprule CRD objects in the `v1alpha1` version. What changes or impacts should I expect if I upgrade
AKO
to version 1.11.1 and above?

How to resolve the error: "...failed to list v1beta1.HostRule: the server could not find the requested resource (get hostrules.ako.vmware.com)" that is displayed in
AKO
logs after upgrading to
AKO
version 1.11.1 or higher?