VMware vSphere SDKs and Tools 8.0

8.0 7.0 6.7 6.5

Troubleshooting Changed Block Tracking (CBT)

This section provides two hints for helping customers with CBT failures.

Verify Change Tracking Enabled

If you reconfigure a virtual machine to set
changeTrackingEnabled
, but the property remains false, check that you have queried the virtual machine status with
VirtualMachine->config()
 after reconfiguration with
VirtualMachine->reconfigure()
 and not before. Also make sure that virtual machine compatibility is hardware version 7 or higher, and that the virtual machine has undergone a stun-unstun cycle since reconfiguration.

Run the vixDiskCheck Utility

If customers encounter CBT issues when restoring files on a destination VM, they can use
vixDiskCheck
 to ascertain whether there was data corruption after backing up the source VM. The
vixDiskCheck
 utility should be run on the backup proxy VM, either Linux or Windows.
When you unpack the VDDK distribution tarball or ZIP file, the
bin/vixDiskCheck
 utility is available in the resulting folder. Add the pathname of this folder to the
LD_LIBRARY_PATH
 environment on Linux, or to the PATH environment on Windows. Examples below use Bash and PowerShell with install locations
/opt/vddk
 and VMware\VDDK.
$ export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/opt/vddk/lib64"
> $env:Path += 'C:\Program Files\VMware\VDDK\bin'
You can run
vixDiskCheck
 or
vixDiskCheck.exe
 to see its options with explanations. Follow these steps to diagnose:
  1. Delete previous snapshots, if any. Then reset CBT by deactivating and enabling it again.
  2. Power on the VM. Collect the following files and look for the
    power-on
     tag.
    • CTK files in the
      vmfs/volumes/datastore/
      VM
       folder, for example
      BaseDisk
      -ctk.vmdk
    • hostd.log
       in
      var/run/log
       and
      vmware.log
       in
      vmfs/volumes/datastore/
      VM
  3. Take a snapshot (snapshot-1) using the vSphere Client. Collect:
    • CTK file in the VM folder under
      vmfs/volumes/datastore
      ,
      BaseDisk
      -ctk.vmdk
       again.
  4. Run
    vixDiskCheck
     to verify consistency of active full backup. Redirect
    stdout
     and
    stderr
     into
    vixDiskCheck.log
     in the current folder, as below. The CBT backup file must be large enough to hold the entire disk.
    vixDiskCheck cbtCheck -host 
    IPaddr
     -user 
    username
     -password 
    password
     -vm moref=
    VmMoref
    
  -ssmoref 
    SnapshotMoRef
     -libdir 
    VDDKlibDir
     -thumb 
    SSLhostThumbprint
     -cbtdir 
    CBTbackupFile
    
  -disk Base-disk.vmdk -transportMode nbdssl 2>&1 | tee vixDiskCheck-01.log
  5. Search for CBT DIFF in the log. If any lines are output, then the CBT file is corrupted.
    grep "CBT DIFF:" vixDiskCheck.log
  6. Delete the previous snapshot and take a new snapshot (snapshot-2). Collect the CTK file and logs tagged snapshot-2.
  7. Run
    vixDiskCheck
     again to verify consistency of the incremental backup with snapshot-2. Use the same command options as before, except for the snapshot MoRef and the log file path.
    vixDiskCheck cbtCheck -host IPaddr -user username -password password -vm moref=VmMoref
  -ssmoref Snapshot2MoRef -libdir VDDKlibDir -thumb SSLhostThumbprint -cbtdir CBTbackupFile
  -disk Base-disk.vmdk -transportMode nbdssl 2>&1 | tee vixDiskCheck-02.log
  8. Repeat the previous step to check consistency of subsequent incremental backups. Make sure to change the snapshot MoRef and the log file name accordingly.
The
-cbtdir
 flag specifies the CBT working directory;
vixDiskCheck
 creates
backup.dat
 and
cbt.inf
 files in that folder. Large file
backup.dat
 is the full backup of the target disk as a sparse local file, which means only allocated areas are written to it, while other areas are holes. Small text file
cbt.inf
 stores the
changeId
 of the snapshot in the latest run, and is read in the next run, so that the utility can
queryChangedDiskAreas
 against the snapshot with this
changeId
 for incremental backup.
In
vixDiskCheck
 logs,
  • "compare in range" means the range has no changes reported by CBT between the two snapshots. It compares data in the created file with data on the source disk.
  • "write in range" means the range has changes reported by CBT between the two snapshots. It writes data on the created file from the source disk.

Content feedback and comments