Tanzu for Valkey on Cloud Foundry 4.0

PDF

Troubleshooting Valkey Instances

Last Updated February 19, 2025

This topic for app developers gives you basic instructions for troubleshooting on-demand VMware Tanzu for Valkey on Cloud Foundry service instances.

Troubleshooting Errors

Start here if you are responding to a specific error or error messages.

Common service errors

The following errors occur in multiple services:


Apps Fail to Connect to the Service Instance

Symptom Apps fail to connect to the Valkey service instance.
Cause

The Valkey on-demand service broker now binds apps to service instances using BOSH DNS. Service bindings return the DNS address instead of the IP address. If an operator enables the BOSH HotSwaps feature, any apps with service bindings that do not use BOSH DNS will fail to connect to the Valkey service instance.

For more information, see Enable BOSH HotSwaps to Reduce Downtime.

For more information about service bindings, see Service Bindings in the Cloud Foundry documentation.

Solution

Convert service instance bindings to use BOSH DNS. To do so, unbind, rebind, and restage all apps that were bound to a service instance using an IP address as follows:

  • Unbind the app: 
    cf unbind-service APP-NAME SERVICE-INSTANCE-NAME
  • Rebind the app: 
    cf bind-service APP-NAME SERVICE-INSTANCE-NAME
  • Restage the app: 
    cf restage APP-NAME


No Metrics from Log Cache

Symptom You receive no metrics when running the cf tail command.
Cause Depending on your versions of Tanzu Platform for CF and Tanzu for Valkey on Cloud Foundry, this can occur when the Firehose is deactivated in the Tanzu Platform for CF tile.
Solution Ask your operator to ensure that the V2 Firehose checkbox is selected, and the Enable Log Cache syslog ingestion check box is cleared in the Tanzu Platform for CF tile. For more information about configuring these check boxes, see Activate syslog forwarding.

Tanzu for Valkey on Cloud Foundry Specific Errors

The following errors are specific to Tanzu for Valkey on Cloud Foundry:

Certain errors are returned to the Valkey client instead of being recorded in the logs. The Valkey protocol represents errors as simple strings beginning with a - character.


Maximum Number of Clients Reached

Symptom

You receive the following error:

-ERR max number of clients reached
Cause This is usually caused by apps opening multiple client connections to Valkey.
Solution Share or pool Valkey connections within an app. Tanzu for Valkey on Cloud Foundry configures Valkey to accept 10000 client connections. This can be confirmed by running the INFO command using the Redis CLI.


Maxmemory Limit Reached

Symptom

You receive the following error:

-OOM command not allowed when used memory > 'maxmemory'.
Cause This occurs when the Valkey server has reached its maxmemory limit.
Solution

Consider changing your maxmemory-policy. You can update this using the cf update-service parameters. For how to do this, see Customize an On-Demand Service Instance.


Error When Running the Save Command

Symptom

You receive the following error message when running redis-cli SAVE or issuing the save command using another Valkey client: 

-ERR
Cause This might occur when the Valkey server's disk is full.
Solution A more informative message might be logged in the syslog. For more information, see Syslog Errors.


Unknown Command Error

Symptom

You receive the following error message when running redis-cli COMMAND or issuing a command using another Valkey client: 

-ERR unknown command
Cause For security reasons, certain commands such as CONFIG, SAVE, BGSAVE and ACL are not available by default.
Solution Talk to your operator about the availability of the command.

Techniques for troubleshooting

See the following sections for troubleshooting techniques when using the Cloud Foundry Command-Line Interface (cf CLI) to perform basic operations on a Tanzu for Valkey on Cloud Foundry service instance.

Basic cf CLI operations include create, update, bind, unbind, and delete.

Debug using the CF CLI

See the following table for Cloud Foundry Command Line Interface (cf CLI) commands commonly used while debugging:

To view the... Command
API endpoint, org, and space cf target
Service offerings available in the targeted org and space cf marketplace
Apps deployed to the targeted org and space cf apps
Service instances deployed to the targeted org and space cf services
GUID for a specific service instance cf service SERVICE-INSTANCE --guid
Service instance or application logs cf tail SERVICE-INSTANCE/APP

Parse a Cloud Foundry (CF) error message

Failed operations (create, update, bind, unbind, delete) cause an error message. You can retrieve the error message later by running the cf CLI command cf service INSTANCE-NAME. 

$ cf service myservice

Service instance: myservice
Service: super-db
Bound apps:
Tags:
Plan: dedicated-vm
Description: Dedicated Instance
Documentation url:
Dashboard:

Last Operation
Status: create failed
Message: Instance provisioning failed: There was a problem completing your request.
     Please contact your operations team providing the following information:
     service: redis-acceptance,
     service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089,
     broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac,
     task-id: 442,
     operation: create
Started: 2017-03-13T10:16:55Z
Updated: 2017-03-13T10:17:58Z

Use the information in the Message field to debug further. Provide this information to Support when filing a ticket.

The task-id field maps to the BOSH task ID. For more information about a failed BOSH task, use the bosh task TASK-ID.

The broker-request-guid maps to the portion of the On-Demand Service Broker log containing the failed step. Access the broker log through your syslog aggregator, or access BOSH logs for the broker by typing bosh logs broker 0. If you have more than one broker instance, repeat this process for each instance.

Retrieve Service Instance information

To retrieve information about the service instance that you can use for debugging:

  1. Log into the space containing the instance or failed instance.

    $ cf login

  2. If you do not know the name of the service instance, you can view a listing of all service instances in the space by running:

    cf services

    The service instances are listed in the name column.

    For example:

    $ cf services
Getting services in org my-org / space my-space as user@example.com...
OK
name          service      plan               bound apps    last operation
my-instance   p.redis      on-demand-cache                  create succeeded

  3. Retrieve more information about a specific service instance by running:

    cf service SERVICE-INSTANCE-NAME

  4. Retrieve the GUID of the service instance by running:

    cf service SERVICE-INSTANCE-NAME --guid

    This is useful for debugging.

  5. If the Log Cache CLI plugin is enabled, you can retrieve logs for the service instance by running:

    cf tail SERVICE-INSTANCE-NAME/APP-NAME

    For more information, see Log Cache CLI plug-in.

Retrieve the password for a Valkey Service Instance

If you want to access the Valkey server for troubleshooting, you can find a Valkey service instance password by creating a new service key.

VMware recommends that you use this key for troubleshooting only, and that you delete the key after troubleshooting by running the command cf delete-service-key SERVICE-INSTANCE KEY-NAME.

For instructions on how to retrieve the password, see Retrieve the Password for a Valkey Service Instance.

Temporary outages

Tanzu for Valkey on Cloud Foundry service instances can become temporarily inaccessible during upgrades and VM or network failures.

Knowledge Base (Community)

Find the answer to your question and navigate product discussions and solutions by searching Broadcom Support.

File a Support Ticket

You can file a support ticket here. Include the error message from cf service YOUR-SERVICE-INSTANCE.

To expedite troubleshooting, provide your service broker logs, service instance logs, and BOSH task output. Your cloud operator can obtain these from your error message.