Where to find Logs

Installer logs provides vital information on what went wrong. Access the installer logs - logs_installer.log in the root folder from where the installer command is run.

Service Status

Check status of all packages being installed by running following commands.

kubectl -n tanzusm get packageInstalls

Describe particular package Installs

kubectl -n tanzusm describe packageInstalls <package name>

List all apps with kapp command

kapp ls -n tanzusm --column Name

List app and resource status

kapp inspect -a  <app name listed in above command>  -n tanzusm

Prometheus Data Collection

Prometheus data can offer insights to RED metrics of the TPSM services. Below are the steps to collect this data.

Prerequisite: jq (https://jqlang.github.io/jq/download/) mostly this will be available in jumper box if not install it and make it executable.

To create a dump of Prometheus data:

1. Export the cluster config file to KUBECONFIG environment variable

   export KUBECONFIG=<cluster kube config file location full path>
   

2. Find out the Prometheus pod name and store it in a variable

   TP_PROMETHEUS_POD="$(kubectl get pods -l app.kubernetes.io/part-of=prometheus -l app.kubernetes.io/component=server -n tanzusm -o json | jq -r .'items[0].metadata.name')"  
   

3. Port forward the Prometheus admin API to invoke the Prometheus data snapshot creation from localhost

   kubectl  port-forward  pods/$TP_PROMETHEUS_POD 9090:9090 -n tanzusm
   

4. Open a new terminal and export the KUBECONFIG (same as step 1) and run the below commands
5. Create the Prometheus data snapshot in Prometheus server and store the snapshot name in a variable

   TP_PROMETHEUS_SNAPSHOT_NAME=$(curl -X POST -s http://localhost:9090/prometheus/api/v1/admin/tsdb/snapshot | jq -r .data.name)
   

6. Create a directory in local to copy the snapshot from cluster

   mkdir -p prometheus-data/$TP_PROMETHEUS_SNAPSHOT_NAME
   

7. Find out the Prometheus pod name and store it in a variable

   TP_PROMETHEUS_POD="$(kubectl get pods -l app.kubernetes.io/part-of=prometheus -l app.kubernetes.io/component=server -n tanzusm -o json | jq -r .'items[0].metadata.name')"
   

8. Copy the snapshot to the local

   kubectl cp -n tanzusm $TP_PROMETHEUS_POD:/bitnami/prometheus/data/snapshots/$TP_PROMETHEUS_SNAPSHOT_NAME   prometheus-data/$TP_PROMETHEUS_SNAPSHOT_NAME
   

9. Create a tar of the local directory and upload it

   tar -czvf prometheus-$TP_PROMETHEUS_SNAPSHOT_NAME.tar.gz prometheus-data/$TP_PROMETHEUS_SNAPSHOT_NAME
   

10. Go to the previous terminal session and close the port forwarding by pressing control + c in the keyboard

Troubleshooting common issues

Use the following table to list Troubleshooting common issues.

Installer command-line help

To get help with the installer commands, use the --help flag. For example:

tanzu-sm-installer  --help  

You can also use this flag with sub-commands. For example:

tanzu-sm-installer install --help 

Installer sample commands

tanzu-sm-installer verify \
  -f config.yaml \
  -u "${ARTIFACTORY_USER}:${ARTIFACTORY_API_TOKEN}" \
  -r ${DOCKER_REGISTRY}/hub-self-managed/${TANZU_SM_VERSION}/repo \
  --kubeconfig ${KUBECONFIG}

tanzu-sm-installer install \
  -f config.yaml \
  -u "${ARTIFACTORY_USER}:${ARTIFACTORY_API_TOKEN}" \
  -r ${DOCKER_REGISTRY}/hub-self-managed/${TANZU_SM_VERSION}/repo \
  --yes

tanzu-sm-installer post-verify \
  --kubeconfig ${KUBECONFIG}

tanzu-sm-installer push collectors \
  -a "${REGISTRY_USERNAME}:{$REGISTRY_PASSWORD}" \
  -r "${REGISTRY_ENDPOINT}" \
  -f tanzusm-collector.tar -s

tanzu-sm-installer push tanzu-plugins \
  -u "${REGISTRY_USERNAME}:${REGISTRY_PASSWORD}" \
  -r "${REGISTRY_ENDPOINT}/${REPO_PATH}" \
  -i tanzu-bundle/tpsm-plugin-bundle.tar.gz

tanzu-sm-installer push tmc-extensions \
  -a "${REGISTRY_USERNAME}:${REGISTRY_PASSWORD}" \
  -r "${REGISTRY_ENDPOINT}/${REPO_PATH}" \
  -f agent-images.tar

tanzu-sm-installer log \
  --kubeconfig ${KUBECONFIG}

tanzu-sm-installer reset \
  --kubeconfig ${KUBECONFIG} -p

tanzu-sm-installer velero install \
  --provider aws \
  --image harbor.tanzu.io:8443/library/velero:v1.14.1 \
  --plugins harbor.tanzu.io:8443/library/velero/velero-plugin-for-aws:v1.10.0 \
  --bucket <BUCKET_NAME> \
  --secret-file <PATH_TO_CREDENTIAL_FILE> \
  --use-volume-snapshots=false \
  --features=EnableCSI \ 
  --use-node-agent \
  --backup-location-config region=<OBJECT_STORAGE_SERVICE_REGION>,s3ForcePathStyle="true",s3Url=<OBJECT_STORAGE_SERVICE_PATH>

tanzu-sm-installer velero backup create <KAPP_ARTIFACT_BACKUP_NAME> \ 
  --snapshot-move-data \
  --include-resources=apps.kappctrl.k14s.io,packageinstalls.packaging.carvel.dev \ 
  --include-namespaces tanzusm

tanzu-sm-installer velero restore create <SECRET_RESTORE_NAME> \ 
  --snapshot-move-data \
  --from-backup  <FULL_BACKUP_NAME>  \
  --include-resources=secrets

tanzu-sm-installer velero schedule create kapp-backup-schedule \ 
  --snapshot-move-data \
  --schedule="@every 4h" \
  --include-resources=apps.kappctrl.k14s.io,packageinstalls.packaging.carvel.dev \ 
  --include-namespaces tanzusm

Create an installation log bundle

To create an installation log bundle, run the following command:

tanzu-sm-installer log

By default, this command creates the log bundle in /tmp/tanzusm.

The following table describes the options of the tanzu-sm-installer log command.