L4 CRD Rule

L4Rule CRD is created in a given namespace where the operator desires to have more control. The topic discusses the details and associated rules of using each field of the L4Rule CRD.

L4 Rule CRD can be used to modify the default properties of the L4 virtual service and the pools created from a service of Type LoadBalancer. Service of type LoadBalancer has to be annotated with the name of the CRD to attach the CRD to the service. A sample L4 Rule CRD is as shown below:

apiVersion: ako.vmware.com/v1alpha2
kind: L4Rule
metadata:
  name: my-l4-rule
  namespace: green
spec:
  analyticsProfileRef: Custom-Analytics-Profile
  analyticsPolicy:
    fullClientLogs:
      enabled: true
      duration: 0
      throttle: 30
  applicationProfileRef: Custom-L4-Application-Profile
  loadBalancerIP: 49.20.193.207
  performanceLimits:
    maxConcurrentConnections: 105
    maxThroughput: 100
  networkProfileRef: Custom-Network-Profile
  networkSecurityPolicyRef: Custom-Network-Security-Policy
  securityPolicyRef: Custom-Security-Policy
  vsDatascriptRefs:
    - Custom-DS-01
    - Custom-DS-02
  backendProperties:
    - port: 80
      protocol: TCP
      enabled: true
      applicationPersistenceProfileRef: Custom-Application-Persistence-Profile
      healthMonitorRefs:
        - Custom-HM-01
        - Custom-HM-02
      lbAlgorithm: LB_ALGORITHM_CONSISTENT_HASH
      lbAlgorithmHash: LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
      lbAlgorithmConsistentHashHdr: custom-string
      sslProfileRef: Custom-SSL-Profile
      sslKeyAndCertificateRef: Custom-Key-And-Certificate
      pkiProfileRef: Custom-PKI-Profile
      analyticsPolicy:
        enableRealtimeMetrics: true
      minServersUp: 1
  listenerProperties:
    - port: 80
      protocol: TCP
      enableSsl: true
  sslKeyAndCertificateRefs:
    - Custom-L4-SSL-Key-Cert
  sslProfileRef: Custom-L4-SSL-Profile

The L4 Rule CRD must be configured in the same namespace as the service of type LoadBalancer.

Attaching L4Rule to LoadBalancer type of Services

An L4 Rule is applied to a virtual service and pool created from the LoadBalancer type of Services when the l4 rule is attached to the service. An L4Rule can be attached by annotating the service with the name of the L4Rule CRD with

ako.vmware.com/l4rule

as the key and name of the l4 rule crd as the value.

metadata:
  annotations:
    ako.vmware.com/l4rule: <name-of-the-l4-rule-crd>

Consider the following example showing a service my-service of type LoadBalancer annotated with an L4Rule my-l4-rule.

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotation:
    ako.vmware.com/l4rule: my-l4-rule
spec:
  selector:
    app.kubernetes.io/name: MyApp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376
  clusterIP: 10.0.171.239
  type: LoadBalancer

Express Custom Analytics Profiles

L4Rule CRD can be used to express analytics profile references. The analytics profile reference must have been created in the

NSX Advanced Load Balancer Controller

before the CRD creation.

analyticsProfile: Custom-Analytics-Profile

The analytics profiles can be used for various Network/Health Score analytics settings, log processing, and so on.

Configure Analytics Policy

The L4Rule CRD can be used to configure analytics policies such as enable/disable non-significant logs, throttle the number of non-significant logs per second on each SE, and the duration for which the system must capture the logs.

analyticsPolicy:
  fullClientLogs:
    enabled: true
    duration: 0
    throttle: 30

Here, the field throttle is configured as shown below:

For throttle to take effect, the value of enabled has to be set to

True

Range: 0 to 65535

Default value: 10

To deactivate throttling, set the value of throttle to 0

By default, the

AKO

sets the duration of logging the non-significant logs to 30 minutes. Configure the duration as Zero (0) to capture the logs indefinitely.

Express Custom Application Profiles

L4 Rule CRD can be used to express application profile references. The application profile can be used to enable PROXY Protocol, rate limit the connections from a client, and so on. The application profile must be created in the

NSX Advanced Load Balancer Controller

before referring to it. The application profile must be of TYPE of APPLICATION_PROFILE_TYPE_L4.

applicationProfile: Custom-L4-Application-Profile

The application profile should be of type

L4 SSL/TLS

If SSL is enabled for any port in

listenerProperties

, then application profile should be of type

L4 SSL/TLS

Starting with

AKO

version 1.11.1,

L4 SSL/TLS

is supported.

Express Custom Load Balancer IP

The loadBalancerIP field can be used to provide a valid preferred IPv4 address for L4 virtual services. The preferred IP must be part of the IPAM configured for the cloud, and must not overlap with any other IP addresses already in use. In case of any misconfigurations whatsoever,

AKO

would fail to configure the virtual service appropriately throwing an ERROR log for the same.

loadBalancerIP: "49.20.193.207"

The L4Rule CRD is not aware of any misconfigurations during its creation process, and as a result, the L4Rule will still be marked as

Accepted

Configure Performance Limits

The L4 Rule CRD can be used to configure the performance limit settings such as maximum concurrent client connections allowed, and maximum throughput per second for all clients allowed through the client side.

performanceLimits:
  maxConcurrentConnections: 105
  maxThroughput: 100

The maxConcurrentConnections and maxThroughput supports values from 0 to 65535.

Express Custom Network Profile

The L4 Rule CRD can be used to express a custom network profile. The network profile can be used to configure either TCP/UDP proxy settings or TCP/UDP fast path settings. The network profile must be created in the

NSX Advanced Load Balancer Controller

before referring to it.

networkProfileRef: Custom-Network-Profile

The

AKO

defaults the network profile to System-TCP-Proxy.

The network profile settings are dependent on the license configured in the

NSX Advanced Load Balancer Controller

. Before configuring the profile in the CRD, see Licensing topic in the

VMware NSX Advanced Load Balancer

Administration Guide

Starting with

AKO

version 1.11.1, SSL for L4 virtual service is supported. If SSL is enabled for any port in

listenerProperties

, then network profile should be of type

TCP proxy

, since only a single

TCP

port definition is allowed in the LoadBalancer service for L4 SSL.

Express Custom Network Security Policy

The L4 Rule CRD can be used to express a custom network security policy. The Network security policy can be configured with rules to allow/deny/rate limit connections from a single or group of IP addresses, and so on.

networkSecurityPolicyRef: Custom-Network-Security-Policy

The Network Security Policy must be created in the

NSX Advanced Load Balancer Controller

before referring to it.

Express Custom Security Policy

The L4Rule CRD can be used to express a custom Security Policy. Security Policy is applied to the traffic of the virtual service, and it is used to specify various configuration information used to perform Distributed Denial of Service (DDoS) attacks detection and mitigation.

securityPolicyRef: Custom-Security-Policy

The Security Policy must be created in the

NSX Advanced Load Balancer Controller

before referring to it.

Express Custom DataScripts

The L4Rule CRD can be used to express DataScripts references. The DataScripts references must have been created in the

NSX Advanced Load Balancer Controller

before the CRD creation.

vsDatascriptRefs:
  - Custom-DS-01
  - Custom-DS-02

The DataScripts can be used to apply custom scripts to data traffic. The order of evaluation of the DataScripts is in the same order they appear in the CRD definition.

Configure Backend Properties

The backendProperties section in the L4Rule can be used to configure pool settings such as custom health monitors, application persistence profiles, LB algorithms, and so on. The L4Rule CRD identifies the pools based on the port and protocol, and

AKO

applies the configuration to it.

AKO

logs a warning if the port and protocol do not match the service’s port and protocol configurations.

A sample backendProperties looks like this:

backendProperties:
  - port: 80
    protocol: TCP
    enabled: true
    applicationPersistenceProfileRef: Custom-Application-Persistence-Profile
    healthMonitorRefs:
      - Custom-HM-01
      - Custom-HM-02
    lbAlgorithm: LB_ALGORITHM_CONSISTENT_HASH
    lbAlgorithmHash: LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
    lbAlgorithmConsistentHashHdr: custom-string
    sslProfileRef: Custom-SSL-Profile
    sslKeyAndCertificateRef: Custom-Key-And-Certificate
    pkiProfileRef: Custom-PKI-Profile
    analyticsPolicy:
      enableRealtimeMetrics: true
    minServersUp: 1

The fields port and protocol are mandatory and

AKO

uses these fields to identify the pool. The port and protocol must equal the service’s port and protocol.

AKO

defaults the fields enabled to

true

and lbAlgorithm to

LB_ALGORITHM_LEAST_CONNECTIONS

Enable/ Disable Pool

The field enabled in the L4Rule can be used to activate/deactivate pools attached to an L4 virtual service. By default, the value of the field is true and the user has to set the value to false to disable the pool.

enabled: true # or false

Express Custom Application Persistence Profile

The L4Rule CRD can be used to express a custom Application Persistence Profile reference. The Application Persistence Profile reference must have been created in the

NSX Advanced Load Balancer Controller

before the CRD creation.

applicationPersistenceProfileRef: Custom-Application-Persistence-Profile

Express Custom Health Monitors

L4Rule CRD can be used to express custom health monitor references. The health monitor reference must have been created in the

NSX Advanced Load Balancer Controller

before the CRD creation.

healthMonitorRefs:
  - Custom-HM-01
  - Custom-HM-02

The health monitors can be used to verify server health. A server (Kubernetes pods in this case) will be marked UP only when all the health monitors return successful responses. Health monitors provided here overwrite the default health monitor configuration set by

AKO

, that is, System-TCP for TCP traffic and System-UDP for UDP traffic based on the service configuration.

Configure LB Algorithm

The L4Rule CRD can be used to select suitable LB algorithms to effectively distribute traffic across healthy servers. The LB algorithm can be used for distributing TCP and UDP connections across servers.

A sample LB algorithm configuration is shown below:

lbAlgorithm: LB_ALGORITHM_CONSISTENT_HASH
lbAlgorithmHash: LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
lbAlgorithmConsistentHashHdr: "custom-string"

The lbAlgorithm allows a predefined set of values, and the user can choose the desired one. Presently the following values are supported for the field lbAlgorithm:

LB_ALGORITHM_LEAST_CONNECTIONS

LB_ALGORITHM_ROUND_ROBIN

LB_ALGORITHM_FASTEST_RESPONSE

LB_ALGORITHM_CONSISTENT_HASH

LB_ALGORITHM_LEAST_LOAD

LB_ALGORITHM_FEWEST_SERVERS

LB_ALGORITHM_RANDOM

LB_ALGORITHM_FEWEST_TASKS

LB_ALGORITHM_NEAREST_SERVER

LB_ALGORITHM_CORE_AFFINITY

LB_ALGORITHM_TOPOLOGY

The lbAlgorithmHash field is used only when the algorithm is chosen as LB_ALGORITHM_CONSISTENT_HASH. The following values are supported for lbAlgorithmHash:

LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS_AND_PORT LB_ALGORITHM_CONSISTENT_HASH_URI LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_STRING LB_ALGORITHM_CONSISTENT_HASH_CALLID

The lbAlgorithmConsistentHashHdr field is used only when the lbAlgorithmHash is chosen as LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER.

Express Custom SSL Profile

The custom SSL profile can be used to configure the desired set of SSL versions and ciphers to accept SSL/TLS terminated connections.

sslProfileRef: Custom-SSL-Profile

The custom SSL profile must have been created in the

NSX Advanced Load Balancer Controller

before the CRD creation.

Express Custom SSL Key And Certificate

The L4Rule CRD can be used to express a custom SSL key and certificate reference. The service engines present this certificate to the backend servers. The custom SSL key and certificate must have been created in the NSX Advanced Controller before the CRD creation.

sslKeyAndCertificateRef: Custom-Key-And-Certificate

Express Custom PKI Profile

The L4Rule CRD can be used to express a custom PKI profile reference. Once configured, the

NSX Advanced Load Balancer Controller

validates the SSL certificate present by a server against the custom PKI Profile configured in the CRD. The custom PKI Profile must be created in the

NSX Advanced Load Balancer Controller

before referring to it.

pkiProfileRef: Custom-PKI-Profile

Configure Analytics Policy

The L4Rule CRD can be used to configure the analytics settings for the pool. Set the enableRealtimeMetrics to

true

false

to enable/disable real-time metrics for server and pool metrics.

Configure Minimum Servers UP

The L4Rule CRD can be used to configure the minimum number of servers in the UP state for marking the pool UP.

minServersUp: 1

The value given must be equal to or less than the number of health monitors attached to the pool.

Configure Listener Properties

The

listenerProperties

section in the L4Rule can be used to enable or disable SSL support for L4 virtual services. Each item in the

listenerProperties

array corresponds to a port definition in the LoadBalancer service with the option to enable SSL termination in the service/listener settings created for that port as part of the

NSX Advanced Load Balancer

virtual service.

When an L4Rule object is created with listener properties,

AKO

identifies the service/listener setting on the virtual service based on the port and protocol, and applies the SSL configuration to it.

AKO

logs a WARNING if the port and protocol does not match the service's port and protocol configurations. There are also limitations and conditions for using listener properties, see Caveats section for more details.

A sample

listenerProperties

looks as shown below:

listenerProperties:
  - port: 80
    protocol: TCP
    enableSsl: true

The fields

port

and

protocol

are mandatory, and

AKO

uses these fields to identify the corresponding service/listener setting in the virtual service. The port and protocol must equal the service's port and protocol. Currently, only a single

TCP

port is allowed in the LoadBalancer service definition if SSL is required to be enabled. Hence, the same limitation also applies to

listenerProperties

, which can also have only one matching

TCP

based port definition.

Enable or Disable SSL

The field

enableSsl

in the L4Rule can be used to enable SSL termination and offload for traffic from clients for an L4 virtual service. The

enableSsl

field is specified for a port, and AKO configures the associated service/listener setting in the virtual service with the value. By default, the value of this field is

false

, and the user has to set the value to

true

to enable SSL.

enableSsl: true # or false

Express custom SSL Profile for Virtual Service

The custom SSL profile can be used to configure the desired set of SSL versions and ciphers to accept SSL/TLS terminated connections for the virtual service.

sslProfileRef: Custom-SSL-Profile

The custom SSL profile must be created in the

NSX Advanced Load Balancer Controller

before the CRD creation.

The

sslProfileRef

must only be specified when SSL is enabled for a virtual service. The L4Rule will otherwise be rejected.

Express custom SSL Keys And Certificates for Virtual Service

The L4Rule CRD can be used to express custom SSL key and certificate references for a virtual service. These certificates will be presented to SSL/TLS terminated connections. The custom SSL keys and certificates must be created in the

NSX Advanced Load Balancer Controller

before the CRD creation.

      sslKeyAndCertificateRefs:
      - "Custom-SSL-Key-Cert"

The

sslKeyAndCertificateRefs

must only be specified when SSL is enabled for a virtual service. The L4Rule will otherwise be rejected.

Status Messages

The status messages are used to give instantaneous feedback about the reference objects specified in the L4Rule CRD.

Following are some sample status messages:

Accepted L4Rule Object

$ kubectl get l4rule
NAME         STATUS     AGE
my-l4-rule   Accepted   3d5s

An L4Rule is accepted only when all the reference objects specified inside it exist in the

NSX Advanced Load Balancer Controller

Rejected L4Rule Object

$ kubectl get l4rule
NAME            STATUS     AGE
my-l4-rule-alt  Rejected   2d23h

The detailed reason for rejection can be obtained from the status:

status: error: applicationprofile "My-L4-Application" not found on controller status: Rejected

Caveats

Sharing L4Rule with Load Balancer IP

The L4Rule CRD with load balancer IP can be shared among services only when the services contain the

ako.vmware.com/enable-shared-vip

annotation. However, L4Rule cannot be shared if SSL termination is required to be enabled for the services. So, if

enableSsl

is set to

true

for any port in

listenerProperties

section, then that L4Rule must only be applied to a single LoadBalancer service.

L4Rule Deletion

If an L4Rule is deleted, the L4 VSes and Pools in the

NSX Advanced Load Balancer Controller

will be configured with the default values.

L4Rule Admission

An L4Rule CRD is only admitted if all the objects referenced in it, exist in the

NSX Advanced Load Balancer Controller

. If after admission the object references are deleted out-of-band, then

AKO

does not re-validate the associated HostRule CRD objects. The user needs to manually edit or delete the object for new changes to take effect.

Enabling SSL with L4Rule

Following are the limitations when trying to enable SSL termintaion for an L4 virtual service with L4Rule:

Currently, only a single

TCP

port is allowed in the LoadBalancer service definition if SSL is required to be enabled. Hence, the same limitation also applies to

listenerProperties

, which can also have only one matching

TCP

based port definition with

enableSsl

field. This is because

NSX Advanced Load Balancer

only supports SSL termination with TCP protocol, and also a virtual service of type L4 SSL can have only one back end pool configured.

enableSsl

is set to

true

for any port in

listenerProperties

section, then

applicationProfile

must be of type

L4 SSL/TLS

. If application profile is not of type

L4 SSL/TLS

, then L4Rule will be rejected. If

applicationProfile

is not set, then it defaults to

System-L4-Application

in the CRD, but

AKO

internally sets the application profile as

System-SSL-Application

, which is the default value when SSL is enabled.

enableSsl

is set to

true

for any port in

listenerProperties

section, then

networkProfileRef

must be of type

TCP proxy

. Currently, only a single

TCP

port definition is allowed in the LoadBalancer service and listener properties.

The

sslProfileRef

and

sslKeyAndCertificateRefs

must be set for the virtual service only if SSL termination is enabled for any port in

listenerProperties

and application profile is of type

L4 SSL/TLS

, otherwise the L4Rule will be rejected. If

enableSsl

is set to

true

, but

sslProfileRef

and

sslKeyAndCertificateRefs

are not set specified in the L4Rule, then these fields will be set with their default values in

NSX Advanced Load Balancer

Setting up Routing Rules using CRDs

Content feedback and comments

Avi Kubernetes Operator Guide 1.11

L4 CRD Rule

Attaching L4Rule to LoadBalancer type of Services

Express Custom Analytics Profiles

Configure Analytics Policy

Express Custom Application Profiles

Express Custom Load Balancer IP

Configure Performance Limits

Express Custom Network Profile

Express Custom Network Security Policy

Express Custom Security Policy

Express Custom DataScripts

Configure Backend Properties

Enable/ Disable Pool

Express Custom Application Persistence Profile

Express Custom Health Monitors

Configure LB Algorithm

Express Custom SSL Profile

Express Custom SSL Key And Certificate

Express Custom PKI Profile

Configure Analytics Policy

Configure Minimum Servers UP

Configure Listener Properties

Enable or Disable SSL

Express custom SSL Profile for Virtual Service

Express custom SSL Keys And Certificates for Virtual Service

Status Messages

Caveats