Ingress
NCP will create one layer-7 load balancer for Ingresses with TLS specification, and
one layer-7 load balancer for Ingresses without TLS specification. You can also create CRDs
(CustomResourceDefinitions) to handle Ingress scaling.
Note the following:
- All Ingresses will get a single IP address.
- The Ingress resource is allocated an IP address from the external IP pool specified by theexternal_ip_poolsoption in the[nsx_v3]section inncp.ini. The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).
- The Ingress resource is allocated an IP address from the external IP pool specified by theexternal_ip_pools_lboption in the[nsx_v3]section inncp.ini. If theexternal_ip_pools_lboption does not exist, the pool specified byexternal_ip_poolsis used. The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).Note that after you set theexternal_ip_pools_lboption, you can add more IP pools. However, updating the IP pools is not supported. Removing IP pools is also not supported.
- You can change to a different IP pool by changing the configuration and restarting NCP.
- You can specify a default certificate for TLS. See below for information about generating a certificate and mounting a certificate into the NCP pod.
- Ingresses without TLS specification will be hosted on HTTP virtual server (port 80).
- Ingresses with TLS specification will be hosted on HTTPS virtual server (port 443). The load balancer will act as an SSL server and terminate the client SSL connection.
- Modification of Ingress by adding or removing the TLS section is supported. When thetlskey is removed from the Ingress specification, the Ingress rules will be transferred from the HTTPS virtual server (port 443) to the HTTP virtual server (port 80). Similarly, when thetlskey is added to Ingress specification, the Ingress rules are transferred from the HTTP virtual server (port 80) to the HTTPS virtual server (port 443).
- If there are duplicate rules in Ingress definitions for a single cluster, only the first rule will be applied. The other Ingresses with the duplicate rules will be annotated with error. For example, if you create two Ingresses with the samehostandpath, and one Ingress is TLS while and the other is non-TLS, andkubernetes.io/ingress.allow-httpisfalse, the two rules will be created on different virtual servers and will not conflict with each other. However, ifkubernetes.io/ingress.allow-httpistrue, the Ingress that is applied later will be annotated with an error. See the "Errors" section below for more information.
- Only a single Ingress with a default backend is supported per cluster. Traffic not matching any Ingress rule will be forwarded to the default backend.
- If there are multiple Ingresses with a default backend, only the first one will be configured. The others will be annotated with an error. See the "Errors" section below for more information.
- The rules are applied in the following order:
- Rules with bothhostandpathspecified.
- Rules with theExactpathType.
- Rules with thePrefixpathType.
- Rules with theRegexpathType.
- Rules withhostorpathspecified.
- Rules with theExactpathType.
- Rules with thePrefixpathType.
- Rules with theRegexpathType.
Note: If multiple paths match a request, precedence will be given to the longest matching path.AboutpathType:- Starting with Kubernetes 1.19pathTypeis mandatory.
- Ifuse-regexis set, it will only take effect ifpathTypeisImplementationSpecific.
- Ifuse-regexis not set,ImplementationSpecificis treated the same asExact.pathTypetakes precedence over other NCP annotations.
- Two matching paths with differentpathTypes can co-exist.
- For thePrefixtype,/foowill match/foo/,/foo/barbut not/fooor /foobar. To match/foo, add theExactpath/footo the Ingress rule.
- Hostname wildcardsWhen specifying hostname, you can specify the exact name, such as abc.example.com, or use a wildcard, such as *.example.com. Note that the wildcard will match one or more DNS labels. The following table shows the match results if you specify *.example.com.Host HeaderMatch Resultabc.example.comMatchabc.def.example.comMatchexample.comNo match
- (This is applicable if you use Policy mode.) If a TLS Ingress is created with a default backend, it is recommended that you set up the default backend to respond to both HTTP and HTTPS requests because:
- The load balancer will terminate TLS and send HTTP requests to the default backend server if there is a TLS Ingress (in the cluster for the Kubernetes/TKGI use case, or in the same namespace for the Project Pacific use case) with host which matches the host in the request.
- The load balancer will re-encrypt and send HTTPS request to the backend server if there is no TLS Ingress (in the cluster for the Kubernetes/TKGI use case, or in the same namespace for the Project Pacific use case) with host which matches the host in the request.
- TheIngressClassresource is supported.
- ThepathTypeattribute is supported.
- JWT (JSON Web Token) client authentication is supported. This feature requiresNSX3.0.0 or later.
- The HTTP and HTTPS request body size is unlimited. NCP does not support setting a limit for the request body size.
Starting with Kubernetes 1.18, the
kubernetes.io/ingress.class
annotation is deprecated and
replaced by the IngressClass resource. To specify NSX
as the Ingress controller in the IngressClass resource, set the
controller parameter to k8s.io/nsx
. For
example:apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: nsx-lb annotations: ingressclass.kubernetes.io/is-default-class: true spec: controller: k8s.io/nsx
To specify a third-party Ingress controller,
set the controller parameter to
k8s.io/<controller name>
. For
example:apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: nginx-lb annotations: ingressclass.kubernetes.io/is-default-class: false spec: controller: k8s.io/ingress-nginx
To set the IngressClass as the default for
the cluster, set the annotation
ingressclass.kubernetes.io/is-default-class
to
true
. See the example above.You must not use both the
kubernetes.io/ingress.class
annotation and the IngressClass
resource.If you manually configure an
Endpoints
object, be sure to set the
targetRef
parameter. The value should be the UID of the backend
pod. For
example:apiVersion: networking.k8s.io/v1 kind: Endpoints metadata: name: tea-svc subsets: - addresses: - ip: 172.26.0.2 targetRef: uid: 4378e0ae-5837-49c6-a0b2-178dced8eb1e ...
Feature Annotations
The following table lists annotations
that NCP supports:
Annotation | Description | Supported Values | Default Value |
---|---|---|---|
kubernetes.io/ingress.allow-http | Enables HTTP requests in addition to HTTPS | true, false | true |
ncp/use-regex | Enables path pattern matching | true, false | false |
ingress.kubernetes.io/rewrite-target | Rewrites path of incoming request |
| No default value |
ncp/http-redirect | Redirects HTTP requests to HTTPS | true, false | false |
kubernetes.io/ingress.class | Indicates which Ingress controller is responsible for this Ingress | nsx, nginx, etc. | nsx |
nsx/loadbalancer | Places an Ingress on a dedicated load balancer | Name of a LoadBalancer CRD | No default value |
ncp/allowed-source-range | Limits Ingress access by request source IP | Comma-separated list of CIDRs, IP addresses, or IP
ranges. | No default value |
ncp/ssl-mode | Selects SSL mode for all the rules in an Ingress | offload, reencrypt, passthrough | offload |
ncp/jwt-alg | Determines the algorithm to be used to validate JWT signature.
Enables JWT client authentication when set with
ncp/jwt-secret. | HS256, RS256 | No default value |
ncp/jwt-secret | Specifies the name of the Kubernetes secret that contains the JWT
secret or public key used for signature validation. Enables JWT client
authentication when set with ncp/jwt-alg. | Name of a Kubernetes secret | No default value |
ncp/jwt-token | Additional location to search for JWT in the HTTP
request. | _arg_<param_name>, _cookie_<cookie_name> | No default value |
ncp/jwt-realm | Specifies the Realm header returned with 401 when authentication
failed. | String indicating the realm | Hostname of the ingress rule |
ncp/jwt-preserve | Option to keep JWT and pass to backend after successful
authentication. | true, false | true |
ncp/connection_multiplexing_enabled | Enables TCP multiplexing. | true, false | false |
ncp/connection_multiplexing_number | If TCP multiplxing is enabled, this specifies the number of idle
TCP connections that are available. | 0 - 2147483647 | 6 |
Details about the annotations:
- Path Regex (Regular Expression) Matching
- You can enable or disable regular expression matching of the Ingresspath(but nothost) parameter using the annotationncp/use-regex. If set tofalse, exact path matching will be performed by doing theequalsmatch. If set totrue, regular expression matching will be performed by adding the start of string character (^) and end of string character ($) to the path so that the entire request URI matches the pattern. Note that when using theORoperator (|), always specify the scope with parentheses so that ^ and $ apply to all the operands. For example,path: /(tea|coffee|milk). An Ingress specification example:apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: kubernetes.io/ingress.class: "nsx" #/tea/cup will be served by tea-svc:80 ncp/use-regex: "True" spec: rules: - host: cafe.example.com http: paths: # for this tea and coffee NCP will configure regex rule - path: /tea/(.*) pathType: ImplementationSpecific backend: service: name: tea-svc port: number: 80 - path: /coffee/(.*) pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 80 # for juice and alcohol NCP will configure Exact and Prefix rules - path: /juice pathType: Exact backend: service: name: juice-svc port: number: 80 - path: /alcohol pathType: Prefix backend: service: name: bar number: 80
- Updating Ingresses prior to upgrading NCPThis is required only if you have Ingresses requiring all sub-path matching using the characters '.' and '*'.
- Update the Ingresses to include the annotationncp/use-regex: true.
- For all sub-path matching, if you have paths such as/coffee/*or/*, change them to/coffee/.*and/.*./coffee/.*will match/coffee/,/coffee/a,/coffee/b,/coffee/a/b, and so on./.*will match/coffee,/tea,/coffee/a, and so on. Omitting the path will produce the same behavior aspath: /.*.
- In the following example of the annotationingress.kubernetes.io/rewrite-target, the paths/teaand/coffeewill be rewritten to/before the URL is sent to the backend service.apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: ingress.kubernetes.io/rewrite-target: / spec: rules: - host: cafe.example.com http: paths: - path: /tea pathType: ImplementationSpecific backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 80Ifpathis specified using a regular expression, the captured groups are saved in numbered placeholders in the form $1, $2, and so on. These placeholders can be used as parameters in theingress.kubernetes.io/rewrite-targetannotation. Named capture groups and named placeholders are not supported. For example,apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: kubernetes.io/ingress.class: "nsx" ncp/use-regex: "true" #/tea/cup will be rewritten to /cup before sending request to endpoint ingress.kubernetes.io/rewrite-target: /$1 spec: rules: - host: cafe.example.com http: paths: - path: /tea/(.*) pathType: ImplementationSpecific backend: service: name: tea-svc port: number: 80 - path: /coffee/(.*) pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 80
- About the annotationkubernetes.io/ingress.allow-http:
- If the annotation is set tofalse, rules will be created for the HTTPS virtual server.
- If the annotation is set totrueor missing, rules will created for both HTTP and HTTPS virtual servers. Note that HTTPS rules will be created only if the TLS section is present in the Ingress specification.
- About the annotationncp/http-redirect:
- If the annotation is set tofalse, Incoming HTTP traffic (port 80) to HTTP Virtual Server will not be redirected to HTTPS Virtual Server.
- If the annotation is set totrue, Incoming HTTP traffic (port 80) to HTTP Virtual Server will be redirected to HTTPS Virtual Server (port 443).
This annotation is only valid if the TLS section is present. This annotation takes precedence overkubernetes.io/ingress.allow-http. When set totrue, it will direct matched HTTP traffic to HTTPS, regardless of howkubernetes.io/ingress.allow-httpis set. - About the annotationkubernetes.io/ingress.class:
- If the value isnsx, this ingress will be handled by NCP. If any other value is specified, the Ingress will be ignored by NCP. For more info see Third-party Ingress Controllers.
- For more information about the annotationnsx/loadbalancer, see LoadBalancer CRDs to Handle Ingress Scaling.
- About the annotationncp/allowed-source-range:
- Supported in Policy mode only.
- When set, an incoming request will only be accepted if its source IP is included in this annotation. Otherwise, traffic will be dropped.
- You can specify a combination of CIDR, IP addresses, and IP ranges. For example, 1.1.1.1/24, 2.2.2.2, 3.3.3.3-4.4.4.4.
- YAML example:apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: kubernetes.io/ingress.class: "nsx" #Source IP addresses allowed to access ingress URL ncp/allowed-source-range: "192.168.128.0/17, 192.168.64.0-192.168.64.255, 192.168.16.32" spec: tls: - hosts: - cafe.example.com rules: - host: cafe.example.com http: paths: - path: /tea pathType: ImplementationSpecific backend: service: name: tea-svc port: number: 80 - path: /coffee pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 80
- About the annotationncp/ssl-mode:
- Supported in Policy mode only.
- This annotation applies to all the rules in an Ingress and the load balancer will operate in the selected SSL mode for these rules.Ifncp/ssl-modeis set toreencryptorpassthrough,kubernetes.io/ingress.allow-httpmust be set toFalse. This annnotation cannot be set topassthroughif theingress.kubernetes.io/rewrite-target,ncp/use-regexorncp/allowed-source-rangeannotation is set.Ifncp/ssl-modeis set topassthrough, the attributepathin therulesspecification is not supported.
- YAML example for SSL modereencrypt:apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress namespace: qe annotations: kubernetes.io/ingress.class: "nsx" kubernetes.io/ingress.allow-http: "False" ncp/ssl-mode: "reencrypt" ingress.kubernetes.io/rewrite-target: "/$1" ncp/use-regex: "true" spec: tls: - hosts: - cafe.example.com secretName: cafe-secret defaultBackend: service: name: default-svc port: number: 443 rules: - host: cafe.example.com http: paths: - path: /fo{2}/[abc]+(r|R)/(.*) pathType: ImplementationSpecific backend: service: name: tea-svc port: number: 443 - path: /coffee pathType: ImplementationSpecific backend: service: name: coffee-svc port: number: 443
- YAML example for SSL modepassthrough:apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: kubernetes.io/ingress.class: "nsx" #Loadbalancer in passthrough mode ncp/ssl-mode: "passthrough" spec: tls: - hosts: - cafe.example.com rules: - host: cafe.example.com http: paths: - backend: service: name: tea-svc port: number: 443 pathType: ImplementationSpecific
- About JWT client authentication:
- To enable JWT client authentication for all rules under an Ingress, bothncp/jwt-algandncp/jwt-secretneed to be set to a valid value. When enabled, incoming HTTP traffic will only be passed to the backend if bearing valid JSON web token. Otherwise, traffic will be rejected with the 401 status.
- This feature is incompatible with the following annotations:
- kubernetes.io/ingress.allow-http: true
- ncp/http-redirect: true
- ncp/ssl-mode: passthrough
- ncp/jwt-alg:
- Supported symmetrical algorithms: HS256
- Supported asymmetrical algorithms: RS256
- ncp/jwt-secret:
- A symmetrical key or public certificate must be configured in a Kubernetes secret with the name specified in this annotation under the same namespace as the Ingress.
- For symmetrical algorithms, the secret must be stored in thejwt.keyfield.
- For asymmetrical algorithms, the public cert must be stored in thetls.crtfield.
- JWT client authentication will be disabled if the symmetrical secret or public certificate is not stored in the locations mentioned above, or if the data stored in the secret is invalid.
- ncp/jwt-token:
- Only one item can be configured in this annotation.
- _arg_<param_name>: For JWT passed in as a URI parameter. Specify the parameter name that contains JWT.
- _cookie_<cookie_name>: For JWT passed in as a cookie. Specify the cookie name that contains JWT.
- About the annotationsncp/connection_multiplxing_enabledandncp/connection_multiplxing_number:
- TCP multiplexing cannot coexist with HTTP NTLM. In NSX, if if a load balancer pool with TCP multiplxing enabled is bound to an NTLM-enabled layer-7 server, the NTLM configuration will take precedence over TCP multiplexing.
- YAML example:apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress annotations: ncp/connection_multiplexing_enabled: "true" ncp/connection_multiplexing_number: 100 spec: tls: - hosts: - cafe.example.com rules: - host: cafe.example.com http: paths: - path: /tea backend: service: name: tea-svc port: number: 80 - path: /coffee backend: service: name: coffee-svc port: number: 80
Errors
Errors are annotated to the Ingress
resource. The error key is
ncp/error.loadbalancer
and the
warning key is ncp/warning.loadbalancer
. The possible error and
warning are: - ncp/error.loadbalancer:DEFAULT_BACKEND_IN_USEThis error indicates that an Ingress with a default backend already exists. The Ingress will be inactive. This error will occur if (1) this Ingress is for HTTP and another Ingress for HTTP with a default backend exists; (2) this Ingress is for HTTPS and another Ingress for HTTPS with a default backend exists; or (3) this Ingress is for HTTP and HTTPS and another Ingress for HTTP and HTTPS with a default backend exists. To fix the error, delete and recreate the Ingress with a correct specification.
- ncp/warning.loadbalancer:SECRET_NOT_FOUNDThis error indicates that the secret specified in the Ingress specification does not exist, or ifncp/jwt-algandncp/jwt-secretare anootated but the secret cannot be found under the same namespace as the Ingress. The Ingress will be partially active. To fix the error, create the missing secret. Note that once a warning is in the annotation, it will not be cleared during the life cycle of the Ingress resource.
- ncp/warning.loadbalancer:INVALID_INGRESSThis error indicates that one of the following conditions is true. The Ingress will be inactive. To fix the error, delete and recreate the Ingress with a correct specification.
- An Ingress rule conflicts with another Ingress rule in the same Kubernetes cluster. Conflicts are determined only for Ingresses with the same match strategy, that is, the samencp/use-regexannotation value.
- Thekubernetes.io/ingress.allow-httpannotation is set tofalseand the Ingress does not have a TLS section.
- Thencp/http-redirectannotation is set totrueand the Ingress does not have a TLS section.
- An Ingress rule does not havehostandpathspecified. Such an Ingress rule has the same functionality as the Ingress default backend. Use the Ingress default backend instead.
- The Ingress has JWL annotations that cannot be correctly processed. For example:
- Eitherncp/jwt-algorncp/jwt-secretis missing.
- ncp/jwt-algis configured with an unsupported algorithm.
- ncp/jwt-algandncp/jwt-secretare configured with other HTTP enabling annotations, or with ssl passthrough.