Tunables for
AKO

The values.yaml in

AKO

affects a configmap that

AKO

’s deployment reads to make adjustments as per user needs. This article lists all the fields specified in values.yaml and helps you make the right choices when deploying

AKO

with the configurable settings.

If a field is marked as editable, it implies that the field can be edited without an

AKO

POD restart.

AKOSettings.fullSyncFrequency

This field is used to set a frequency of consistency checks in

AKO

. Typically inconsistent states can arise if users make changes out of band with respect to

AKO

. For example, a pool is deleted by the user from the UI of the

Avi Load Balancer Controller

. The full sync frequency is used to ensure that the models are reconciled and the corresponding

Avi Load Balancer

objects are restored to the original state.

AKOSettings.enableEvents (editable)

This flag provides the ability to enable/disable Event broadcasting from

AKO

. The value specified here gets populated in the

ConfigMap

and can be edited at any time while

AKO

is running.

AKO

picks up the change in the param value and enables/disables Event broadcasting in the cluster at runtime, so

AKO

pod restart is not required.

AKOSettings.logLevel (editable)

This flag defines the logLevel for logging and can be set to one of DEBUG, INFO, WARN, ERROR (case sensitive). The logLevel value specified here gets populated in the

ConfigMap

and can be edited at any time while

AKO

is running.

AKO

picks up the change in the param value and sets the logLevel at runtime, so

AKO

pod restart is not required.

AKOSettings.deleteConfig (editable)

This flag is intended to be used for deletion of objects in the

Avi Load Balancer Controller

. The default value is

false

. If the value is set to true while booting up,

AKO

will not process any Kubernetes object and stop regular operations.

While

AKO

is running, this value can be edited to true in the

AKO

configmap to delete all objects created by

AKO

Avi Load Balancer

. After that, if the value is set to false,

AKO

resumes processing Kubernetes objects and recreates all the objects in

Avi Load Balancer

AKOSettings.disableStaticRouteSync

This flag can be used in two scenarios:

If your POD CIDRs are routable either through an internal implementation or by default.

If you are working with multiple NICs on your kubernetes worker nodes and the default gateway is not from the same subnet as your VRF’s PG network.

AKOSettings.clusterName

The clusterName field primarily identifies your running

AKO

instance.

AKO

internally uses this field to tag all the objects it creates on

Avi Load Balancer Controller

. All objects created by a particular

AKO

instance have a prefix of

<clusterName&gt

in their names and also populates the

created_by

so ako-<clusterName&gt

Each

AKO

instance mapped to a given

Avi Load Balancer

cloud must have a unique

clusterName

parameter. This maintains uniqueness of the object naming across Kubernetes clusters.

AKOSettings.apiServerPort

The apiServerPort field is used to run the API server within the

AKO

pod. The Kubernetes API server uses the

/api/status

API to verify the health of the

AKO

pod on the

pod:port

where the port is defined by this field. This is configurable, because some environments can block usage of the default

8080

port. This field is purely used for

AKO

’s internal API server and must not be confused with a Kubernetes pod port.

AKOSettings.cniPlugin

Use this flag only if you are using

calico/openshift/ovn-kubernetes/cilium

as a CNI and you are looking to sync your static route configurations automatically.

Cilium CNI

For cilium CNI, setting this flag is only required when using Cluster Scope mode for IPAM. With Cilium CNI, there are two ways to configure the per-node PodCIDRs. In the default cluster scope mode, the podCIDRs range are made available through the CiliumNode (cilium.io/v2.CiliumNode) CRD and

AKO

reads this CRD to determine the Pod CIDR to Node IP mappings when the flag is set as

cilium

In Kubernetes host scope mode, podCIDRs are allocated out of the PodCIDR range associated to each node by Kubernetes. Since

AKO

determines the Pod CIDR to Node IP mappings from Node Spec by default, the

cniPlugin

flag is not required to be set exclusively.

Calico

Once enabled, for

calico

, this flag is used to read the

blockaffinity

CRD to determine the Pod CIDR to Node IP mappings. If you are on an older version of calico where

blockaffinity

is not present, then leave this field as blank.

OpenShift

For openshift hostsubnet CRD is used to determine the Pod CIDR to Node IP mapping.

Ovn-kubernetes

For ovn-kubernetes the

k8s.ovn.org/node-subnets

annotation in the Node metadata is used to determine the Pod CIDR to Node IP mapping.

AKO

determines the static routes based on the Kubernetes Nodes object as done with other CNIs.

In case of NCP CNI,

AKO

automatically disables the configuration of static routes.

There are certain scenarios where

AKO

cannot determine the Pod CIDRs being used in the Kubernetes Nodes, for instance, when deploying calico using etcd as the datastore. In such cases,

AKO

provides its own interface to feed in Pod CIDR to Node mappings, using an annotation in the Node object. While keeping the cniPlugin value to be empty, add the following annotation in the Node object to provide Pod CIDRs being used in the Node. Note that for multiple Pod CIDRs that are being used in the Node, simply provide the entries as a comma separated string.

annotations: ako.vmware.com/pod-cidrs: 192.168.1.0/24,192.169.1.0/24

AKOSettings.layer7Only

Use this flag for

AKO

to act as a pure layer 7 Ingress Controller.

AKO

has to be rebooted for this flag change to take effect. If the configmap is edited while

AKO

is running, then the change will not take effect. If

AKO

was working for both L4-L7 prior to this change and then this flag is set to true, then

AKO

will delete the layer 4 LB virtual services from the

Avi Load Balancer Controller

and keep only the Layer 7 virtual services. If the flag is set to false the service of type Loadbalancers would be synced and Layer 4 virtual services would be created.

NetworkSettings.nsxtT1LR

This knob is used to specify the T1 logical router’s name in the format of

/infra/tier-1s/<name-of-t1>

. Preconfigure this T1 router with a logical segment in the NSX-T cloud as a data network segment.

AKO

uses this information to populate the virtual service’s and pool’s T1Lr attribute.

AKOSettings.enableEVH

Use this flag if you want to create an Enhanced Virtual Hosting model for virtual service objects in

Avi Load Balancer

. It is disabled by default. Set the flag to true to enable the flag.

Before enabling the flag in the existing deployment delete the config and then enable the flag. This will ensure SNI based virtual services are deleted before creating EVH virtual services.

AKOSetttings.namespaceSelector.labelKey, AKOSetttings.namespaceSelector.labelValue

AKO

allows ingresses/routes from specific namespace/s to be synced to

Avi Load Balancer Controller

. This key-value pair represent a label that is used by

AKO

to filter out namespace/s. If one of key/values specified are empty, then ingresses/routes from all namespaces will be synched to the

Avi Load Balancer Controller

AKOSettings.servicesAPI

Use this flag to enable

AKO

to watch over Gateway API CRDs that is. GatewayClasses and Gateways.

AKO

only supports Gateway APIs with Layer 4 Services. Setting this to true enables users to configure GatewayClass and Gateway CRDs to aggregate multiple Layer 4 Services and create one virtual service per Gateway Object.

AKOSettings.blockedNamespaceList

The blockedNamespaceList lists the Kubernetes/OpenShift namespaces blocked by

AKO

will not process any Kubernetes/OpenShift object updates from these namespaces. The default value is empty list.

blockedNamespaceList:
- kube-system
- kube-public

AKOSettings.istioEnabled (Tech Preview)

AKO

can be deployed in an Istio environment. Setting AKOSettings.istioEnabled to true indicates to

AKO

that the environment is Istio. By default this is set to false.

AKOSettings.ipFamily (Tech Preview)

IPv6 is currently supported only for vCenter cloud with calico CNI.

AKO

can be deployed with ipFamily as V4 or V6. When the ipFamily is set to V6,

AKO

looks for V6 IP for nodes from the Calico annotation and creates routes on the Controller. Only servers with V6 IP will get added to Pools. This setting is for the backend pools to use IPv6 or IPv4. For frontend virtual services, use v6cidr.

AKOSettings.useDefaultSecretsOnly

restricts the secret handling to default secrets present in the namespace where

AKO

is installed in the OpenShift clusters if set to true. By default, this flag is set to False.

NetworkSettings.nodeNetworkList

The nodeNetworkList lists the Networks (specified using either networkName or networkUUID) and Node CIDR’s where the Kubernetes nodes are created. This is only used in thevCenter cloud and only when disableStaticRouteSync is set to

false

If two Kubernetes clusters have overlapping Pod CIDRs, the Service Engine needs to identify the right gateway for each of the overlapping CIDR groups. This is achieved by specifying the right placement network for the pools that helps the Service Engine place the pools appropriately.

NetworkSettings.subnetIP and NetworkSettings.subnetPrefix

AKO

supports dual arm deployment where the Virtual IP network can be on a different subnet than the actual Port Groups on which the kubernetes nodes are deployed.

These fields are used to specify the Virtual IP network details on which the user wants to place the

Avi Load Balancer

virtual services on.

NetworkSettings.vipNetworkList

List of VIP Networks can be specified through vipNetworkList with key as networkName or networkUUID. Except AWS cloud, for all other cloud types, only one networkName is supported. For example in vipNetworkList:


vipNetworkList:
- networkName: net1

vipNetworkList:      
- networkUUID: dvportgroup-4167-cloud-d4b24fc7-a435-408d-af9f-150229a6fea6f

In addition to the networkName or networkUUID, we can also provide CIDR information that allows us to specify the Virtual IP network details on which the user wants to place the

Avi Load Balancer

virtual services on.

vipNetworkLists:
- networkName: net1
cidr: 10.1.1.0/24
v6cidr: 2002::1234:abcd:ffff:c0a8:101/64

vipNetworkLists:
- networkUUID: dvportgroup-4167-cloud-d4b24fc7-a435-408d-af9f-150229a6fea6f
cidr: 10.1.1.0/24
v6cidr: 2002::1234:abcd:ffff:c0a8:101/64

v6cidr will only work for Enterprise license with

Avi Load Balancer Controller

. You can provide either cidr, v6cidr, or both. v6cidr will enable the virtual services networks to use IPv6.

For all Public clouds, vipNetworkList must have at least one networkName. For other cloud types too, it is suggested that networkName must be specified in vipNetworkList. With

Avi Load Balancer

IPAM, if networkName is not specified in vipNetworkList, an IP can be allocated from the IPAM of the cloud.

In AWS cloud, multiple networkNames are supported in vipNetworkList.

NetworkSettings.enableRHI

This feature allows the

Avi Load Balancer

Service Engines to publish the VIP to SE interface IP mapping to the upstream BGP peers. Using BGP, a virtual service enabled for RHI can be placed on up to 64 SEs within the SE group. Each SE uses RHI to advertise a /32 host route to the virtual service’s VIP address, and is able to accept the traffic. The upstream router uses ECMP to select a path to one of the SEs. Based on this update, the BGP peer connected to the

Avi Load Balancer

SE updates its route table to use the

Avi Load Balancer

SE as the next hop for reaching the VIP. The peer BGP router also advertises itself to its upstream BGP peers as a next hop for reaching the VIP.The BGP peer IP addresses, and the local Autonomous System (AS) number and a few other settings, are specified in a BGP profile on the

Avi Load Balancer Controller

This feature is available as a global setting in

AKO

which means if it’s set to true then it would apply for all virtual services created by

AKO

. Since RHI is a Layer 4 construct, the settings applies to all the host FQDNs patched as pools/SNI virtualservices to the parent shared virtualservice.

NetworkSettings.defaultDomain

Here are the two use cases for the

defaultDomain

flag:

For L4 virtual services, if multiple sub-domains are configured in the cloud, this flag allows you to specify the default sub-domain for the virtual service. It is used to generate the FQDN for the service of type LoadBalancer. If the default sub-domain is not specified, the system will choose the first sub-domain based on sorting logic. Therefore, it is recommended to use this parameter to have control over DNS resolution for services of type LoadBalancer.

The

L4Settings.defaultDomain

flag will be deprecated in a future release, so use

NetworkSettings.defaultDomain

instead. If both

NetworkSettings.defaultDomain

and

L4Settings.defaultDomain

are configured,

NetworkSettings.defaultDomain

will be used.

For L7 virtual services (created from OpenShift Routes), if the

spec.subdomain

field is specified instead of the

spec.host

field in an OpenShift route, the specified default domain is appended to the

spec.subdomain

to generate the FQDN for the virtual service. The

defaultDomain

must be configured as a sub-domain in your

Avi Load Balancer

cloud.

defaultDomain: "avi.internal"

For instance, if the

spec.subdomain

for an OpenShift route is

my_route-my_namespace

and

defaultDomain

is specified as

avi.internal

, the FQDN for the L7 virtual services will be

my_route-my_namespace.avi.internal

image.pullsecrets

If you are setting the

image repository

field to use a secure private container image registry for the

AKO

image, then you must specify the pull secrets in this field. The pull secrets are a list of Kubernetes Secret objects that are created from the login credentials of a secure private image registry. The container runtime uses the pull secrets to authenticate with the registry in order to pull the

AKO

image. The image pull secrets must be created in the avi-system namespace before deploying

AKO

pullSecrets:
- name: regcred

L7Settings.shardVSSize

AKO

uses a sharding logic for Layer 7 ingress objects. A sharded virtual service involves hosting multiple insecure or secure ingresses hosted by one virtual IP or VIP. Having a shared virtual IP allows lesser IP usage since reserving IP addresses particularly in public clouds incur greater cost.

L7Settings.noPGForSNI

Currently http caching is not available on pool groups from the

Avi Load Balancer Controller

AKO

uses pool groups for canary style deployments. If a user does not require canary deployments and they have an immediate requirement for HTTP caching then this flag can be helpful. Use of this flag is highly discouraged unless required, as it will be deprecated in future once

Avi Load Balancer

Pool Groups implement HTTP caching in the

Avi Load Balancer Controller

If this flag is set to true then

AKO

would program http policy set rules to switch between pools instead of pool groups. This feature only applies to secure FQDNs.

L7Settings.passthroughShardSize

This is applicable only in OpenShift environment.

AKO

uses a sharding logic for pass through routes, these are distinct from the shared virtual services used for Layer 7 Ingress or Route objects. For all passthrough routes, a set of shared virtual services are created. The number of such virtual services is controlled by this flag.

L7Settings.defaultIngController

This field is related to the ingress class support in

AKO

specified through kubernetes.io/ingress.class annotation specified on an ingress object. If

AKO

is set as the default Ingress Controller, then it will sync everything other than the ones on which the Ingress class is specified and is not equal to avi. If

Avi Load Balancer

(Avi) is not set as the default Ingress controller then

AKO

will sync only those Ingresses which have the ingress class set to avi. If you do not use ingress classes, then do not modify this knob and

AKO

will sync all your Ingress objects to

Avi Load Balancer

L7Settings.fqdnReusePolicy

This field is used to restrict or allow FQDN to span across multiple namespaces.

InterNamespaceAllowed

: When set to this value,

AKO

allows the hostname/FQDN to be associated with Ingresses/Routes to span across multiple namespaces.

Strict

: When set to this value,

AKO

restricts the hostname/FQDN to be associated only with Ingresses/Routes within the same namespace.

L4Settings.defaultDomain

If you have multiple sub-domains configured in your

Avi Load Balancer

cloud, use this knob to specify the default sub-domain. This is used to generate the FQDN for the Service of type loadbalancer. If unspecified, the behavior works on a sorting logic. The first sorted sub-domain in chosen, so we recommend using this parameter if you want to be in control of your DNS resolution for service of type LoadBalancer.

The

L4Settings.defaultDomain

flag will be deprecated in a future release, so use

NetworkSettings.defaultDomain

instead. If both

NetworkSettings.defaultDomain

and

L4Settings.defaultDomain

are configured,

NetworkSettings.defaultDomain

will be used.

L4Settings.autoFQDN

This knob is used to control how the layer 4 service of type Loadbalancer’s FQDN is generated.

AKO

supports 3 options:

Default: In this case, the FQDN format is: where the namespace refers to the Service’s namespace. The sub-domain is picked up from the IPAM/DNS profile.

Flat: In this case, the FQDN format is: disabled. In this case, the FQDNs are not generated for service of type Loadbalancers.

ControllerSettings.controllerVersion

This field is used to specify the

Avi Load Balancer Controller

version. While

AKO

is backward compatible with most of the 18.2.x

Avi Load Balancer Controller

, the tested and preferred Controller version is 18.2.10

ControllerSettings.controllerHost

This field is usually not present in the values.yaml by default but can be provided with the helm install command to specify the IP address or Hostname of the

Avi Load Balancer Controller

. If you are using a containerized deployment of the controller, use a fully qualified controller IP address/FQDN. For example, if the Controller is hosted on 8443, then the Controller host must be of the format x.x.x.x:8443.

ControllerSettings.cloudName

This field is used to specify the name of the IaaS cloud in

Avi Load Balancer Controller

. For example, if you have the vCenter cloud named as Demo then specify the name of the cloud name with this field. This helps

AKO

determine the IaaS cloud to create the service engines on.

ControllerSettings.tenantsPerCluster

If this field is set to true,

AKO

will map each Kubernetes / OpenShift cluster uniquely to a tenant in

Avi Load Balancer

. If enabled, then the tenant must be created in

Avi Load Balancer

to map to a cluster and needs to be specified in

ControllerSettings.tenantName

field.

ControllerSettings.tenantName

The tenantName field is used to specify the name of the tenant where all the

AKO

objects will be created in

Avi Load Balancer

. This field is only required if tenantsPerCluster is set to true. The tenant in

Avi Load Balancer

needs to be created by the

Avi Load Balancer Controller

admin before the

AKO

bootup.

ControllerSettings.cloudName

This field is used to specify the name of the IaaS cloud in

Avi Load Balancer Controller

. For example, if you have the vCenter cloud named as Demo then specify the name of the cloud name with this field. This helps

AKO

determine the IaaS cloud to create the service engines on.

ControllerSettings.vrfName

The

vrfName

field is used to specify the name of the VRFContext where all the

AKO

objects are created. The VRFContext in

Avi Load Balancer

needs to be created by the

Avi Load Balancer Controller

admin before the

AKO

boots up. This is applicable in the vCenter cloud only.

AWS and Azure Cloud in NodePort mode of
AKO

If the IaaS cloud is Azure then subnet name is specified in networkName within vipNetworkList. Azure IaaS cloud is supported only in NodePort mode of

AKO

. If the IaaS cloud is AWS then, subnet uuid is specified in networkName within the vipNetworkList. AWS IaaS cloud is supported only in the NodePort mode of

AKO

. The subnetIP and subnetPrefix are not required for AWS and Azure Cloud.

avicredentials.username and avicredentials.password

The username/password of the

Avi Load Balancer Controller

is specified with this flag. The username/password are base64 encoded by helm and a corresponding secret object is used to maintain the same. Editing this field requires a restart (delete/re-create) of the

AKO

pod.

avicredentials.authtoken

The generated authtoken from the

Avi Load Balancer Controller

can be specified with this flag as an alternative to password. The authtoken is also base64 encoded and updated regularly by secret object. The token refresh is managed by

AKO

. In case of token refresh failure, a new token needs to be generated and updated into the secret object.

Following are a few ways to encode a token generated from the Controller to directly patch avi-secret:

Shell

echo -n '<authtoken>' | base64

Python

import base64
authtoken = "<authtoken>"
print(base64.b64encode(authtoken.encode("ascii")))

Starting with version 1.12.1,

AKO

supports reading

Avi Load Balancer Controller

credentials including

certificateAuthorityData

from existing

avi-secret

from the namespace in which

AKO

is installed. If the

username

and either

password

authtoken

are not specified,

avi-secret

will not be created as part of Helm installation.

AKO

will assume that

avi-secret

already exists in the namespace in which the

AKO

Helm release is installed and will reference it.

avicredentials.certificateAuthorityData

This field allows setting the rootCA of the

Avi Load Balancer Controller

, that

AKO

uses to verify the server certificate provided by the Controller during the TLS handshake. This also enables

AKO

to connect securely over SSL with the

Avi Load Balancer Controller

, which is not possible in case the field is not provided. The field can be set as follows:

certificateAuthorityData: |-
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

replicaCount

This option specifies the number of replicas of the

AKO

pod.

Starting with version 1.9.1, two instances of

AKO

are supported.

AKO

s runs in active and passive modes, respectively. Once the active

AKO

goes down, the passive

AKO

will be ready to take over.

image.repository

If you are using a private container registry and you would like to override the default Docker Hub settings, then this field can be edited with the private registry name.

L7Settings.serviceType

This option specifies whether

AKO

functions in

ClusterIP

mode or

NodePort

mode. By default, it is set to

ClusterIP

. Allowed values are ClusterIP, NodePort. If CNI type for the cluster is antrea, then another

serviceType

named

NodePortLocal

is allowed.

nodeSelectorLabels.key and nodeSelectorLabels.value

It might not be desirable to have all the nodes of a Kubernetes cluster to participate in becoming server pool members, hence key/value is used as a label-based selection on the nodes in Kubernetes to participate in NodePort. If the key/value is not specified then all nodes are selected.

persistentVolumeClaim

By default,

AKO

prints all the logs to standard output. Instead,

persistentVolumeClaim

(PVC) can be used for publishing logs of

AKO

pod to a file in PVC. To use this, the create a PVC (and a persistent volume, if required) and specify the name of the PVC as the value of

persistentVolumeClaim

securityContext

SecurityContext holds a security configuration that is applied to the

AKO

pod. Some fields are present in both SecurityContext and PodSecurityContext. When both are set, the values in SecurityContext take precedence. Refer to the SecurityContext v1 core.

podSecurityContext

This can be used to set the securityContext of

AKO

pod, if necessary. For example, in an OpenShift environment, if a persistent storage with hostpath is used for logging, then the securityContext must have the field privileged set to true. For more information, see Persistent storage using hostPath.

featureGates.GatewayAPI

Use this flag to enable Gateway API feature for

AKO

. It is disabled by default. To enable, set the flag to true.

This feature is currently under Tech Preview.

GatewayAPI

Enable Gateway API in under featureGate to use this field.

GatewayAPI.image.repository

If you are using a private container registry, to override the default dockerhub settings, configure this field with the private registry name.

featureGates.EnableEndpointSlice

Enable this flag to use EndpointSlices instead of Endpoints in

AKO

. It also supports the graceful shutdown of servers. Starting from

AKO

version 1.13.1, this feature is enabled by default.

Overview

Content feedback and comments

Avi Kubernetes Operator 1.13

Tunables for AKO

AKOSettings.fullSyncFrequency

AKOSettings.enableEvents (editable)

AKOSettings.logLevel (editable)

AKOSettings.deleteConfig (editable)

AKOSettings.disableStaticRouteSync

AKOSettings.clusterName

AKOSettings.apiServerPort

AKOSettings.cniPlugin

Cilium CNI

Calico

OpenShift

Ovn-kubernetes

AKOSettings.layer7Only

NetworkSettings.nsxtT1LR

AKOSettings.enableEVH

AKOSetttings.namespaceSelector.labelKey, AKOSetttings.namespaceSelector.labelValue

AKOSettings.servicesAPI

AKOSettings.blockedNamespaceList

AKOSettings.istioEnabled (Tech Preview)

AKOSettings.ipFamily (Tech Preview)

AKOSettings.useDefaultSecretsOnly

NetworkSettings.nodeNetworkList

NetworkSettings.subnetIP and NetworkSettings.subnetPrefix

NetworkSettings.vipNetworkList

NetworkSettings.enableRHI

NetworkSettings.defaultDomain

image.pullsecrets

L7Settings.shardVSSize

L7Settings.noPGForSNI

L7Settings.passthroughShardSize

L7Settings.defaultIngController

L7Settings.fqdnReusePolicy

L4Settings.defaultDomain

L4Settings.autoFQDN

ControllerSettings.controllerVersion

ControllerSettings.controllerHost

ControllerSettings.cloudName

ControllerSettings.tenantsPerCluster

ControllerSettings.tenantName

ControllerSettings.cloudName

ControllerSettings.vrfName

AWS and Azure Cloud in NodePort mode ofAKO

avicredentials.username and avicredentials.password

avicredentials.authtoken

avicredentials.certificateAuthorityData

replicaCount

image.repository

L7Settings.serviceType

nodeSelectorLabels.key and nodeSelectorLabels.value

persistentVolumeClaim

securityContext

podSecurityContext

featureGates.GatewayAPI

GatewayAPI

GatewayAPI.image.repository

featureGates.EnableEndpointSlice

Tunables for
AKO

AWS and Azure Cloud in NodePort mode of
AKO