Spring Cloud Data Flow for Kubernetes 1.6

1.6 1.5

Configuring Monitoring for VMware Spring Cloud Data Flow for Kubernetes

Last Updated February 15, 2025

Here you will learn how to configure monitoring for VMware Spring Cloud Data Flow for Kubernetes (SCDF for Kubernetes). SCDF for Kubernetes can use either Wavefront or Prometheus.

Configure monitoring with Wavefront

You can configure SCDF for Kubernetes to send metrics from the Data Flow server application, along with metrics from deployed stream and task applications, to the Wavefront service.

Sign up for a Wavefront account

To use Wavefront with SCDF for Kubernetes, you must have a Wavefront account.

  1. Sign up for an account at the VMware Tanzu Observability by Wavefront website.

  2. Generate an API token for your account. Go to your Wavefront user profile page and click the API Access tab.

Wavefront API secret

To send metrics to the Wavefront service, you must create a Kubernetes secret containing the API token generated in the previous section. This secret must be created in the namespace where you will install SCDF for Kubernetes.

  1. Create the secret using the following command, where <WAVEFRONT_API_TOKEN> is your API token:

    $ kubectl create secret generic wavefront-api --from-literal=wavefront-api-token=<WAVEFRONT_API_TOKEN>

  2. Mount the secret, to make it available for use by the Data Flow server configuration. Add the secret mount settings in the deployment-patch.yaml file for the Data Flow server application:

    containers:
      - name: scdf-server
        volumeMounts:
          - name: wavefront-api
            mountPath: /workspace/runtime/secrets/wavefront-api
            readOnly: true
  ...

      volumes:
        - name: wavefront-api
          secret:
            secretName: wavefront-api

Configure monitoring with Wavefront

This section contains an example of the configuration to use when enabling Wavefront for SCDF for Kubernetes applications. After the Data Flow server starts up with this configuration, metrics from the applications are sent to Wavefront.

To enable Wavefront for the Data Flow Server and stream and task applications:

  1. Edit the application.yaml file for the Data Flow application. Add the following configuration:

    management:
  metrics:
    web:
      server:
        request:
          autotime:
            enabled: true
        metric-name: spring.cloud.dataflow.http.server.requests
    export:
      wavefront:
        enabled: true
        api-token: ${wavefront-api-token}
        uri: https://<YOUR_WAVEFRONT_URI>.wavefront.com
        source: <YOUR_SOURCE_NAME>

    Where:

    • <YOUR_WAVEFRONT_URI> is the URI for your installation of Wavefront.
    • <YOUR_SOURCE_NAME> is the name of your Wavefront source.
    • api-token is automatically resolved from the secret.

Configure monitoring with Prometheus

SCDF supports both short-lived application monitoring and long-lived application monitoring with Prometheus. Spring Cloud Task and Spring Cloud Stream applications deployed by Data Flow and Skipper can use the Prometheus RSocket Proxy.

SCDF for Kubernetes also supports other monitoring technologies that use Micrometer. This section describes how to configure the Prometheus RSocket Proxy only.

The SCDF for Kubernetes dashboard provides a link to custom Streams and Tasks dashboards for use with Grafana. You can download these dashboards and install them into your Grafana server.

The following example configures monitoring for all deployed Spring Cloud Task and Spring Cloud Stream applications that use Prometheus over an RSocket gateway. Add this configuration to the application.yaml file for the Data Flow application:

spring:
  cloud:
    dataflow:
      applicationProperties:
        stream:
          management:
            metrics:
              export:
                prometheus:
                  enabled: true
                  rsocket:
                    enabled: true
                    host: prometheus-proxy
                    port: 7001
        task:
          management:
            metrics:
              export:
                prometheus:
                  enabled: true
                  rsocket:
                    enabled: true
                    host: prometheus-proxy
                    port: 7001
      grafana-info:
        url: 'https://grafana:3000'

Where:

  • url is the Grafana location.
  • host is the Prometheus host for your environment.
  • port is the Prometheus port for your environment.

See the Spring Cloud Data Flow microsite for more information about stream monitoring and task monitoring.

These common application properties are passed to the deployed Spring Cloud Task and Spring Cloud Stream applications. If you are using an alternative monitoring tool that is also supported by Micrometer, you can set the appropriate Spring Boot configuration values for your monitoring infrastructure. Add this configuration to the applicationProperties.stream and applicationProperties.task sections of the application.yaml file for the Data Flow application.

Next: Configure multiple platforms support

To configure multiple platform support in SCDF for Kubernetes, see Configuring Multiple Platforms Support for SCDF for Kubernetes.