Ingress

NCP will create one layer-7 load balancer for Ingresses with TLS specification, and one layer-7 load balancer for Ingresses without TLS specification. You can also create CRDs (CustomResourceDefinitions) to handle Ingress scaling.
Note the following:
  • All Ingresses will get a single IP address.
  • The Ingress resource is allocated an IP address from the external IP pool specified by the
    external_ip_pools
    option in the
    [nsx_v3]
    section in
    ncp.ini
    . The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).
  • The Ingress resource is allocated an IP address from the external IP pool specified by the
    external_ip_pools_lb
    option in the
    [nsx_v3]
    section in
    ncp.ini
    . If the
    external_ip_pools_lb
    option does not exist, the pool specified by
    external_ip_pools
    is used. The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).
    Note that after you set the
    external_ip_pools_lb
    option, you can add more IP pools. However, updating the IP pools is not supported. Removing IP pools is also not supported.
  • You can change to a different IP pool by changing the configuration and restarting NCP.
  • You can specify a default certificate for TLS. See below for information about generating a certificate and mounting a certificate into the NCP pod.
  • Ingresses without TLS specification will be hosted on HTTP virtual server (port 80).
  • Ingresses with TLS specification will be hosted on HTTPS virtual server (port 443). The load balancer will act as an SSL server and terminate the client SSL connection.
  • Modification of Ingress by adding or removing the TLS section is supported. When the
    tls
    key is removed from the Ingress specification, the Ingress rules will be transferred from the HTTPS virtual server (port 443) to the HTTP virtual server (port 80). Similarly, when the
    tls
    key is added to Ingress specification, the Ingress rules are transferred from the HTTP virtual server (port 80) to the HTTPS virtual server (port 443).
  • If there are duplicate rules in Ingress definitions for a single cluster, only the first rule will be applied. The other Ingresses with the duplicate rules will be annotated with error. For example, if you create two Ingresses with the same
    host
    and
    path
    , and one Ingress is TLS while and the other is non-TLS, and
    kubernetes.io/ingress.allow-http
    is
    false
    , the two rules will be created on different virtual servers and will not conflict with each other. However, if
    kubernetes.io/ingress.allow-http
    is
    true
    , the Ingress that is applied later will be annotated with an error. See the "Errors" section below for more information.
  • Only a single Ingress with a default backend is supported per cluster. Traffic not matching any Ingress rule will be forwarded to the default backend.
  • If there are multiple Ingresses with a default backend, only the first one will be configured. The others will be annotated with an error. See the "Errors" section below for more information.
  • The rules are applied in the following order:
    1. Rules with both
      host
      and
      path
      specified.
      1. Rules with the
        Exact
        pathType
        .
      2. Rules with the
        Prefix
        pathType
        .
      3. Rules with the
        Regex
        pathType
        .
    2. Rules with
      host
      or
      path
      specified.
      1. Rules with the
        Exact
        pathType
        .
      2. Rules with the
        Prefix
        pathType
        .
      3. Rules with the
        Regex
        pathType
        .
    Note: If multiple paths match a request, precedence will be given to the longest matching path.
    About
    pathType
    :
    • Starting with Kubernetes 1.19
      pathType
      is mandatory.
    • If
      use-regex
      is set, it will only take effect if
      pathType
      is
      ImplementationSpecific
      .
    • If
      use-regex
      is not set,
      ImplementationSpecific
      is treated the same as
      Exact
      .
      pathType
      takes precedence over other NCP annotations.
    • Two matching paths with different
      pathType
      s can co-exist.
    • For the
      Prefix
      type,
      /foo
      will match
      /foo/
      ,
      /foo/bar
      but not
      /foo
      or /
      foobar
      . To match
      /foo
      , add the
      Exact
      path
      /foo
      to the Ingress rule.
  • Hostname wildcards
    When specifying hostname, you can specify the exact name, such as abc.example.com, or use a wildcard, such as *.example.com. Note that the wildcard will match one or more DNS labels. The following table shows the match results if you specify *.example.com.
    Host Header
    Match Result
    abc.example.com
    Match
    abc.def.example.com
    Match
    example.com
    No match
  • (This is applicable if you use Policy mode.) If a TLS Ingress is created with a default backend, it is recommended that you set up the default backend to respond to both HTTP and HTTPS requests because:
    • The load balancer will terminate TLS and send HTTP requests to the default backend server if there is a TLS Ingress (in the cluster for the Kubernetes/TKGI use case, or in the same namespace for the Project Pacific use case) with host which matches the host in the request.
    • The load balancer will re-encrypt and send HTTPS request to the backend server if there is no TLS Ingress (in the cluster for the Kubernetes/TKGI use case, or in the same namespace for the Project Pacific use case) with host which matches the host in the request.
  • The
    IngressClass
    resource is supported.
  • The
    pathType
    attribute is supported.
  • JWT (JSON Web Token) client authentication is supported. This feature requires
    NSX
    3.0.0 or later.
  • The HTTP and HTTPS request body size is unlimited. NCP does not support setting a limit for the request body size.
Starting with Kubernetes 1.18, the
kubernetes.io/ingress.class
annotation is deprecated and replaced by the IngressClass resource. To specify
NSX
as the Ingress controller in the IngressClass resource, set the controller parameter to
k8s.io/nsx
. For example:
apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: nsx-lb annotations: ingressclass.kubernetes.io/is-default-class: true spec: controller: k8s.io/nsx
To specify a third-party Ingress controller, set the controller parameter to
k8s.io/<controller name>
. For example:
apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: nginx-lb annotations: ingressclass.kubernetes.io/is-default-class: false spec: controller: k8s.io/ingress-nginx
To set the IngressClass as the default for the cluster, set the annotation
ingressclass.kubernetes.io/is-default-class
to
true
. See the example above.
You must not use both the
kubernetes.io/ingress.class
annotation and the IngressClass resource.
If you manually configure an
Endpoints
object, be sure to set the
targetRef
parameter. The value should be the UID of the backend pod. For example:
apiVersion: networking.k8s.io/v1 kind: Endpoints metadata: name: tea-svc subsets: - addresses: - ip: 172.26.0.2 targetRef: uid: 4378e0ae-5837-49c6-a0b2-178dced8eb1e ...

Feature Annotations

The following table lists annotations that NCP supports:
Annotation
Description
Supported Values
Default Value
kubernetes.io/ingress.allow-http
Enables HTTP requests in addition to HTTPS
true, false
true
ncp/use-regex
Enables path pattern matching
true, false
false
ingress.kubernetes.io/rewrite-target
Rewrites path of incoming request
  • Path starting with ‘/’
  • Path with a numbered placeholder for a group captured in a regular expression, for example, /$1
No default value
ncp/http-redirect
Redirects HTTP requests to HTTPS
true, false
false
kubernetes.io/ingress.class
Indicates which Ingress controller is responsible for this Ingress
nsx, nginx, etc.
nsx
nsx/loadbalancer
Places an Ingress on a dedicated load balancer
Name of a LoadBalancer CRD
No default value
ncp/allowed-source-range
Limits Ingress access by request source IP
Comma-separated list of CIDRs, IP addresses, or IP ranges.
No default value
ncp/ssl-mode
Selects SSL mode for all the rules in an Ingress
offload, reencrypt, passthrough
offload
ncp/jwt-alg
Determines the algorithm to be used to validate JWT signature. Enables JWT client authentication when set with ncp/jwt-secret.
HS256, RS256
No default value
ncp/jwt-secret
Specifies the name of the Kubernetes secret that contains the JWT secret or public key used for signature validation. Enables JWT client authentication when set with ncp/jwt-alg.
Name of a Kubernetes secret
No default value
ncp/jwt-token
Additional location to search for JWT in the HTTP request.
_arg_<param_name>, _cookie_<cookie_name>
No default value
ncp/jwt-realm
Specifies the Realm header returned with 401 when authentication failed.
String indicating the realm
Hostname of the ingress rule
ncp/jwt-preserve
Option to keep JWT and pass to backend after successful authentication.
true, false
true
ncp/connection_multiplexing_enabled
Enables TCP multiplexing.
true, false
false
ncp/connection_multiplexing_number
If TCP multiplxing is enabled, this specifies the number of idle TCP connections that are available.
0 - 2147483647
6
Details about the annotations:
  • Path Regex (Regular Expression) Matching
    • You can enable or disable regular expression matching of the Ingress
      path
      (but not
      host
      ) parameter using the annotation
      ncp/use-regex
      . If set to
      false
      , exact path matching will be performed by doing the
      equals
      match. If set to
      true
      , regular expression matching will be performed by adding the start of string character (^) and end of string character ($) to the path so that the entire request URI matches the pattern. Note that when using the
      OR
      operator (|), always specify the scope with parentheses so that ^ and $ apply to all the operands. For example,
      path: /(tea|coffee|milk)
      . An Ingress specification example:
      apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: kubernetes.io/ingress.class: "nsx" #/tea/cup will be served by tea-svc:80 ncp/use-regex: "True" spec: rules: - host: cafe.example.com http: paths: # for this tea and coffee NCP will configure regex rule - path: /tea/(.*) pathType: ImplementationSpecific backend: service: name: tea-svc port: number: 80 - path: /coffee/(.*) pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 80 # for juice and alcohol NCP will configure Exact and Prefix rules - path: /juice pathType: Exact backend: service: name: juice-svc port: number: 80 - path: /alcohol pathType: Prefix backend: service: name: bar number: 80
    • Updating Ingresses prior to upgrading NCP
      This is required only if you have Ingresses requiring all sub-path matching using the characters '.' and '*'.
      1. Update the Ingresses to include the annotation
        ncp/use-regex: true
        .
      2. For all sub-path matching, if you have paths such as
        /coffee/*
        or
        /*
        , change them to
        /coffee/.*
        and
        /.*
        .
        /coffee/.*
        will match
        /coffee/
        ,
        /coffee/a
        ,
        /coffee/b
        ,
        /coffee/a/b
        , and so on.
        /.*
        will match
        /coffee
        ,
        /tea
        ,
        /coffee/a
        , and so on. Omitting the path will produce the same behavior as
        path: /.*
        .
  • In the following example of the annotation
    ingress.kubernetes.io/rewrite-target
    , the paths
    /tea
    and
    /coffee
    will be rewritten to
    /
    before the URL is sent to the backend service.
    apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: ingress.kubernetes.io/rewrite-target: / spec: rules: - host: cafe.example.com http: paths: - path: /tea pathType: ImplementationSpecific backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 80
    If
    path
    is specified using a regular expression, the captured groups are saved in numbered placeholders in the form $1, $2, and so on. These placeholders can be used as parameters in the
    ingress.kubernetes.io/rewrite-target
    annotation. Named capture groups and named placeholders are not supported. For example,
    apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: kubernetes.io/ingress.class: "nsx" ncp/use-regex: "true" #/tea/cup will be rewritten to /cup before sending request to endpoint ingress.kubernetes.io/rewrite-target: /$1 spec: rules: - host: cafe.example.com http: paths: - path: /tea/(.*) pathType: ImplementationSpecific backend: service: name: tea-svc port: number: 80 - path: /coffee/(.*) pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 80
  • About the annotation
    kubernetes.io/ingress.allow-http
    :
    • If the annotation is set to
      false
      , rules will be created for the HTTPS virtual server.
    • If the annotation is set to
      true
      or missing, rules will created for both HTTP and HTTPS virtual servers. Note that HTTPS rules will be created only if the TLS section is present in the Ingress specification.
  • About the annotation
    ncp/http-redirect
    :
    • If the annotation is set to
      false
      , Incoming HTTP traffic (port 80) to HTTP Virtual Server will not be redirected to HTTPS Virtual Server.
    • If the annotation is set to
      true
      , Incoming HTTP traffic (port 80) to HTTP Virtual Server will be redirected to HTTPS Virtual Server (port 443).
    This annotation is only valid if the TLS section is present. This annotation takes precedence over
    kubernetes.io/ingress.allow-http
    . When set to
    true
    , it will direct matched HTTP traffic to HTTPS, regardless of how
    kubernetes.io/ingress.allow-http
    is set.
  • About the annotation
    kubernetes.io/ingress.class
    :
    • If the value is
      nsx
      , this ingress will be handled by NCP. If any other value is specified, the Ingress will be ignored by NCP. For more info see Third-party Ingress Controllers.
  • For more information about the annotation
    nsx/loadbalancer
    , see LoadBalancer CRDs to Handle Ingress Scaling.
  • About the annotation
    ncp/allowed-source-range
    :
    • Supported in Policy mode only.
    • When set, an incoming request will only be accepted if its source IP is included in this annotation. Otherwise, traffic will be dropped.
    • You can specify a combination of CIDR, IP addresses, and IP ranges. For example, 1.1.1.1/24, 2.2.2.2, 3.3.3.3-4.4.4.4.
    • YAML example:
      apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: kubernetes.io/ingress.class: "nsx" #Source IP addresses allowed to access ingress URL ncp/allowed-source-range: "192.168.128.0/17, 192.168.64.0-192.168.64.255, 192.168.16.32" spec: tls: - hosts: - cafe.example.com rules: - host: cafe.example.com http: paths: - path: /tea pathType: ImplementationSpecific backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 80
  • About the annotation
    ncp/ssl-mode
    :
    • Supported in Policy mode only.
    • This annotation applies to all the rules in an Ingress and the load balancer will operate in the selected SSL mode for these rules.
      If
      ncp/ssl-mode
      is set to
      reencrypt
      or
      passthrough
      ,
      kubernetes.io/ingress.allow-http
      must be set to
      False
      . This annnotation cannot be set to
      passthrough
      if the
      ingress.kubernetes.io/rewrite-target
      ,
      ncp/use-regex
      or
      ncp/allowed-source-range
      annotation is set.
      If
      ncp/ssl-mode
      is set to
      passthrough
      , the attribute
      path
      in the
      rules
      specification is not supported.
    • YAML example for SSL mode
      reencrypt
      :
      apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress namespace: qe annotations: kubernetes.io/ingress.class: "nsx" kubernetes.io/ingress.allow-http: "False" ncp/ssl-mode: "reencrypt" ingress.kubernetes.io/rewrite-target: "/$1" ncp/use-regex: "true" spec: tls: - hosts: - cafe.example.com secretName: cafe-secret defaultBackend: service: name: default-svc port: number: 443 rules: - host: cafe.example.com http: paths: - path: /fo{2}/[abc]+(r|R)/(.*) pathType: ImplementationSpecific backend: service: name: tea-svc port: number: 443 - path: /coffee pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 443
  • YAML example for SSL mode
    passthrough
    :
    apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: kubernetes.io/ingress.class: "nsx" #Loadbalancer in passthrough mode ncp/ssl-mode: "passthrough" spec: tls: - hosts: - cafe.example.com rules: - host: cafe.example.com http: paths: - backend: service: name: tea-svc port: number: 443 pathType: ImplementationSpecific
  • About JWT client authentication:
    • To enable JWT client authentication for all rules under an Ingress, both
      ncp/jwt-alg
      and
      ncp/jwt-secret
      need to be set to a valid value. When enabled, incoming HTTP traffic will only be passed to the backend if bearing valid JSON web token. Otherwise, traffic will be rejected with the 401 status.
    • This feature is incompatible with the following annotations:
      • kubernetes.io/ingress.allow-http: true
      • ncp/http-redirect: true
      • ncp/ssl-mode: passthrough
    • ncp/jwt-alg
      :
      • Supported symmetrical algorithms: HS256
      • Supported asymmetrical algorithms: RS256
    • ncp/jwt-secret
      :
      • A symmetrical key or public certificate must be configured in a Kubernetes secret with the name specified in this annotation under the same namespace as the Ingress.
      • For symmetrical algorithms, the secret must be stored in the
        jwt.key
        field.
      • For asymmetrical algorithms, the public cert must be stored in the
        tls.crt
        field.
      • JWT client authentication will be disabled if the symmetrical secret or public certificate is not stored in the locations mentioned above, or if the data stored in the secret is invalid.
    • ncp/jwt-token
      :
      • Only one item can be configured in this annotation.
      • _arg_<param_name>
        : For JWT passed in as a URI parameter. Specify the parameter name that contains JWT.
      • _cookie_<cookie_name>
        : For JWT passed in as a cookie. Specify the cookie name that contains JWT.
  • About the annotations
    ncp/connection_multiplxing_enabled
    and
    ncp/connection_multiplxing_number
    :
    • TCP multiplexing cannot coexist with HTTP NTLM. In NSX, if if a load balancer pool with TCP multiplxing enabled is bound to an NTLM-enabled layer-7 server, the NTLM configuration will take precedence over TCP multiplexing.
    • YAML example:
      apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: ncp/connection_multiplexing_enabled: "true" ncp/connection_multiplexing_number: 100 spec: tls: - hosts: - cafe.example.com rules: - host: cafe.example.com http: paths: - path: /tea backend: service: name: tea-svc port: number: 80 - path: /coffee backend: service: name: coffee-svc port: number: 80

Errors

Errors are annotated to the Ingress resource. The error key is
ncp/error.loadbalancer
and the warning key is
ncp/warning.loadbalancer
. The possible error and warning are:
  • ncp/error.loadbalancer
    :
    DEFAULT_BACKEND_IN_USE
    This error indicates that an Ingress with a default backend already exists. The Ingress will be inactive. This error will occur if (1) this Ingress is for HTTP and another Ingress for HTTP with a default backend exists; (2) this Ingress is for HTTPS and another Ingress for HTTPS with a default backend exists; or (3) this Ingress is for HTTP and HTTPS and another Ingress for HTTP and HTTPS with a default backend exists. To fix the error, delete and recreate the Ingress with a correct specification.
  • ncp/warning.loadbalancer
    :
    SECRET_NOT_FOUND
    This error indicates that the secret specified in the Ingress specification does not exist, or if
    ncp/jwt-alg
    and
    ncp/jwt-secret
    are anootated but the secret cannot be found under the same namespace as the Ingress. The Ingress will be partially active. To fix the error, create the missing secret. Note that once a warning is in the annotation, it will not be cleared during the life cycle of the Ingress resource.
  • ncp/warning.loadbalancer
    :
    INVALID_INGRESS
    This error indicates that one of the following conditions is true. The Ingress will be inactive. To fix the error, delete and recreate the Ingress with a correct specification.
    • An Ingress rule conflicts with another Ingress rule in the same Kubernetes cluster. Conflicts are determined only for Ingresses with the same match strategy, that is, the same
      ncp/use-regex
      annotation value.
    • The
      kubernetes.io/ingress.allow-http
      annotation is set to
      false
      and the Ingress does not have a TLS section.
    • The
      ncp/http-redirect
      annotation is set to
      true
      and the Ingress does not have a TLS section.
    • An Ingress rule does not have
      host
      and
      path
      specified. Such an Ingress rule has the same functionality as the Ingress default backend. Use the Ingress default backend instead.
    • The Ingress has JWL annotations that cannot be correctly processed. For example:
      • Either
        ncp/jwt-alg
        or
        ncp/jwt-secret
        is missing.
      • ncp/jwt-alg
        is configured with an unsupported algorithm.
      • ncp/jwt-alg
        and
        ncp/jwt-secret
        are configured with other HTTP enabling annotations, or with ssl passthrough.