VMware vSphere SDKs and Tools 8.0

8.0 7.0 6.7 6.5

Query Allocated Blocks Function

As of vSphere 6.7, developers can call
VixDiskLib_QueryAllocatedBlocks
 to accelerate the backup process, especially with large virtual disks. By knowing which blocks are allocated, an application can avoid reading irrelevant sectors and thus reduce data transfer during backup.

Function Signatures

The
VixDiskLib_QueryAllocatedBlocks
 function synchronously retrieves a list of allocated blocks. On an open VMDK, it queries for allocated blocks by chunks. A chunk is a defined as a block of consecutive sectors. The calling program can specify chunk size to control granularity, along with starting sector and extent. 
VixError
VixDiskLib_QueryAllocatedBlocks(VixDiskLibHandle diskHandle,
                                VixDiskLibSectorType startSector,
                                VixDiskLibSectorType numSectors,
                                VixDiskLibSectorType chunkSize,
                                VixDiskLibBlockList **blockList )
Parameters
  • diskHandle
     [in] Handle to an open virtual disk.
  • startSector
     [in] Absolute offset, possibly zero (0).
  • numSectors
     [in] Number of sectors to query.
  • chunkSize
     [in] Minimum number of sectors covered by a chunk containing data.
  • blockList
     [out] Buffer containing a
    VixDiskLibBlockList
    .
Return:
VIX_OK
 on success, otherwise a suitable VIX error code.
Backup programs must call the
VixDiskLib_FreeBlockList
 function to deallocate the returned
blockList
 when done with it. 
VixError 
VixDiskLib_FreeBlockList(VixDiskLibBlockList *blockList)
Parameters
  • blockList
     [out] Buffer with above returned
    VixDiskLibBlockList
    .
Return:
VIX_OK
 on success, otherwise a suitable VIX error code.

About Query Allocated Blocks

This function supplements
QueryChangedDiskAreas
, the vSphere API for changed block tracking (CBT). Backup software can start by calling
VixDiskLib_QueryAllocatedBlocks
 to get a list of allocated blocks, instead of calling
QueryChangedDiskAreas
 with special change ID ("*") to return only active portions of a virtual disk. Calling
VixDiskLib_QueryAllocatedBlocks
 is easier, and its implementation is simpler. Consequently the special change ID ("*") is deprecated. It has issues with pre-existing snapshots, and is slow when used on extremely large disks.
Query allocated blocks does not depend on CBT. You can enable CBT either before or after the call. New in vSphere 6.7,
VixDiskLib_QueryAllocatedBlocks
 is backward compatible to vSphere 6.0.
On a Windows proxy, a read/write thread that calls
VixDiskLib_QueryAllocatedBlocks
 must be created after the
VixDiskLib_InitEx
 call. On a Linux proxy there is no such limitation.
The storage system may allocate more space than needed by a guest OS. For example, suppose VMFS block size is 1MB and the guest OS has 4KB block size. When the guest requests 4KB to be written on disk, vSphere allocates 1MB but only 64KB is recorded as changed in CBT (64KB is CBT's minimum sector size). So
VixDiskLib_QueryAllocatedBlocks
 may report pre-allocated blocks on which nothing is yet written.
If disk size is not a multiple of the chunk size passed to
VixDiskLib_QueryAllocatedBlocks
, it may not check whether the final part is allocated. For example, with thick-provisioned disk 1024.25 MB in size, query allocated blocks returns the entire disk (1024.25MB) if the chunk size is 64KB, but if chunk size is 1MB, it returns only 1024MB and does not check that the final 0.25 MB is allocated. VMware recommends that developers treat the remaining 0.25 MB as allocated.
VixDiskLib_QueryAllocatedBlocks
 can be used on object-oriented datastores such as vSAN and vVols (virtual volumes). It is especially helpful for backing up thin-provisioned disks. To skip copying non-allocated regions, programs call
VixDiskLib_QueryAllocatedBlocks
 to get the list of allocated blocks returned in the final parameter. This works for large sparsely allocated virtual disks even when CBT is not enabled.
VixDiskLib_QueryAllocatedBlocks
 uses an underlying NFC session (NBD) to get allocated block information. Network connectivity, including DNS address resolution, is required for NFC protocol between the backup proxy and ESXi hosts on port 902. DNS name may not be required if using direct IP addresses without FQDN set on ESXi hosts.
VMware Cloud does not support NBD transport mode, so backup applications on VMware Cloud must use
QueryChangedDiskAreas("*")
 instead.
Another limitation is that
VixDiskLib_QueryAllocatedBlocks
 returns the whole VMDK for NFS, rather than just the allocated blocks as for VMFS, vSAN, and vVol datastores.
In file transport mode,
VixDiskLib_QueryAllocatedBlocks
 depends on the disk library to support local VMDK files. In NBD(SSL) transport mode, an NFC message queries the bitmap of allocated sectors. In SAN and HotAdd transport modes, raw disk in the guest OS does not contain allocated sector information, so the function depends on the aforementioned NFC message to get a bitmap of allocated sectors.

Use Cases for Query Allocated Blocks

To accelerate the initial full backup of allocated disk:
  1. Open the VMDK file.
  2. Query allocated blocks.
  3. Read and save the allocated blocks.
  4. Close the VMDK file. Repeat steps for subsequent VMDK files.
For an incremental or differential backup:
  1. Call
    QueryChangedDiskAreas
     with
    changeId
     of the epoch (point in time of last backup) to find all areas that have changed since the last backup. This is
    A1
    .
  2. Use
    VixDiskLib_QueryAllocatedBlocks
     to find all valid data sectors. This is
    A2
    .
  3. The intersection of the two sets (A1 ∩ A2) yields the regions that have changed and are allocated. The backup application should save the data in these regions (the intersection).
  4. The area A1 minus the intersection (A1 – ( A1 ∩ A2)) yields the regions that were changed but unmapped. The backup application can record those regions as zeroed without reading them.
This is highly useful when backing up an
AUTO-UNMAP
 enabled VMDK. Since unmapped blocks can be recorded as "changed block" by CBT, without the
VixDiskLib_QueryAllocatedBlocks
 call, those unmapped blocks get backed up unnecessarily. With the features together, as described in above, the real changed blocks (namely excluding unmapped) can be calculated.

Code Sample for Query Allocated Blocks

In the development kit, see the
vixDiskLibSample.cpp
 program for its sample code related to
chunkSize
 . The
DoGetAllocatedBlocks
 routine calls the query allocated blocks function, after being requested by the
-getallocatedblocks
 command line argument.

Content feedback and comments