Using the Highstate workspace

The
Highstate
workspace is used to manage and monitor your system's configuration. You can see a historical view of highstate runs across your infrastructure to get insights and trends related to configuration drift across all minions.
Going forward,
Automation Config
is no longer included in the Aria Automation suite of products. The new name of this product is VMware Tanzu Salt and this product is available as part of the VMware Tanzu Platform suite of products. See Using and Managing Tanzu Salt for more information.
As a systems administrator, you are aware of the importance of keeping your system's configuration in compliance, identifying any systems with failures or changes, and bringing those systems back into compliance as quickly as possible. The
Highstate
workspace provides valuable insight into your system's configuration and includes tools that can help you manage configuration drift.

What is highstate?

Instead of manually configuring each minion and each application one-by-one, you can use the
Automation Config
state management system to create state files that you can apply to many minions simultaneously. These state files can include a set of instructions that tell
Automation Config
which operations to run on the minion and in which order. The state management system also ensures each minion is configured properly and as efficiently as possible.
However, if you have hundreds of state files and thousands of minions, running each state file takes time. A faster, more intuitive way to apply state files to your minions is to run highstate. When you run highstate, the
state.highstate
function is called. The
state.highstate
function applies all Salt states outlined in a
top.sls
file that you create on your Salt master.
Running highstate can be useful in the following scenarios:
  • You are setting up your infrastructure for the first time and need to standardize configurations across your environment.
  • You have critical systems that must remain in compliance. The
    Schedules
    workspace in
    Automation Config
    can be used to schedule a highstate job that runs once a week to ensure that your machines are configured properly.
The
top.sls
file maps Salt states to a group of targets and consists of the Salt environment, target criteria, and state files to run. The minions download the
top.sls
file and attempt to match the expressions in it. If a minion matches the expression, it runs the specified state files. For example, you might organize your
top.sls
file to apply certain state files to all minions, some to CentOS minions, and others to minions whose IDs contain the string
web
.
base: '*': - vim - scripts 'os:CentOS': - match: grain - centos-config '*web*': - apache - django
While running highstate, if a configuration has drifted,
Automation Config
puts minions and applications back into its compliant configuration state. If a configuration must change,
Automation Config
quickly deploys those changes to the affected minions. You can then use the
Highstate
workspace to see what changes were made during the last highstate run. Charts and reports in the workspace also help visualize the changes across your infrastructure.
Consider testing your changes in a development environment or staging environment. If your states succeed, you can apply them in your production environment.
您也可以在试运行模式下运行 highstate。如果在试运行模式下运行 highstate,
Automation Config
将生成作业返回,显示应用状态时将在每个工作节点上进行哪些更改。要了解有关试运行模式的详细信息,请参见如何创建作业
You must create and configure the
top.sls
file on your Salt master before you run highstate for the first time. See Organizing states in the Salt user guide for more information about Salt states and the
top.sls
file.

Run highstate

After you configure the
top.sls
file, you can run your first highstate.
To run highstate:
  1. Click the
    States
    tab in the
    Highstate
    workspace.
  2. Click
    Run Highstate
    .
  3. In the dialog box, select the target to run highstate on.
  4. Click
    Run Highstate
    .
After the highstate run is finished, you can view the results of your first highstate run at several levels:
  • Global level: see highstate returns across your environment
  • Target level: see highstate returns by target group
  • Minion level: see highstate returns by minion

Global level

You can see highstate compliance across your environment from the
States
tab and
Minions
tabs. This global level helps you assess the overall state of your system's configuration since the latest highstate run.
The Highstate workspace in the SaltStack Config user interface after a recent
                    highstate run
Across all levels, the latest highstate data is pulled from each minion, whether the data is from a recent highstate run or from many highstate runs. You can use the
Limit by
drop-down menu in the
States
tab to ensure that the
Highstate
workspace only displays the most recent data.
The global level includes the following information for all states:
Term
Description
State
The ID declaration for the state, for example,
common-centos-packages
.
SLS file
The file that contains the state declaration, for example,
centos-config.sls
.
Minions
The numbers of minions that ran the state.
Errors
Indicates if the state run failed. The state fail for many reasons. For example:
  • A required file does not exist or is inaccessible due to filesystem permissions.
  • A network resource or service is unavailable.
  • There is a package dependency conflict so the state cannot be applied.
The
Comments
column provides more details about the cause of the state failure.
Succeeded
Indicates the overall percentage of states that succeeded.
Changed
Indicates the overall percentage of minions that reported changes for each state.
Changes
Indicates if the most recent highstate run resulted in changes.
The
Comments
column provides more details about the changes that were made.
Avg Duration
Average duration for the state run in milliseconds (ms).
Environments
The Salt environment that contains the state file, for example,
base
or
dev
.
JID
The job identification number for the
state.highstate
run.
If you click a state from the
States
tab, you can view the most recent highstate run on each of your Salt minions.
All previous highstate runs for the default html page state in SaltStack
                    Config
The following information is provided for each state:
Term
Description
Minion ID
The ID for the minion that ran the state.
Status
Indicates if the state run succeeded or failed.
Changes
Indicates if the most recent highstate run resulted in changes.
The
Comments
column provides more details about the changes that were made.
Environment
The Salt environment that contains the state file, for example,
base
or
dev
.
Cluster/Master
The Salt master associated with the minion.
Duration
Duration for the state run in milliseconds (ms).
Comment
A message that provides further details about the success or failure of the state run.
JID
The job identification number for the
state.highstate
run.
You can also run an ad-hoc highstate job at the global level:
  1. In
    Automation Config
    , open the
    Highstate
    workspace.
  2. Click
    Run Highstate
    .
  3. In the dialog box, select the target to run highstate on.
  4. Click
    Run Highstate
    .
You can monitor the job's progress in the
Activity
tab of the
Highstate
workspace. After the job is complete, you can view the updated reports and details of the highstate run at the global, target, or minion levels.

Target level

To view highstate returns at the target level, you can open the
Targets
workspace. If you click a target, you can see all previous state runs for all minions in the target group.
Previous highstate runs for a CentOS target group in SaltStack Config
The target level provides the following information about each minion in the target group:
Term
Description
Minion ID
The ID of the minion.
State Runs
The number of state runs for a minion.
Errors
Indicates if the state run failed. The state might fail for many reasons. For example:
  • A required file does not exist or is inaccessible due to filesystem permissions.
  • A network resource or service is unavailable.
  • There is a package dependency conflict so the state cannot be applied.
The
Comments
column provides more details about the cause of the state failure.
Succeeded
Indicates the overall percentage of states that succeeded.
Changed
Indicates the overall percentage of states that resulted in changes.
Changes
Indicates if the most recent highstate run resulted in changes.
The
Comments
column provides more details about the changes that were made.
Duration
Duration for the state run in milliseconds (ms).
Environments
The Salt environment that contains the state file, for example,
base
or
dev
.
JID
The job identification number for the
state.highstate
run.
To run an ad-hoc highstate job at the target level:
  1. In
    Automation Config
    , open the
    Targets
    workspace.
  2. Select the target group you want to run highstate on.
  3. Click the
    Highstate
    tab.
  4. Click
    Run Highstate
    .
  5. In the dialog box, confirm the target group you want to run highstate on.
  6. Click
    Run Highstate.

Minion level

To view highstate returns at the minion level, you can open the
Minions
tab in the
Highstate
workspace.
Highstate returns for all minions in a SaltStack Config environment
The minion level provides the following information for all minions:
Term
Description
Minion ID
The ID of the minion.
State Runs
The number of state runs for a minion.
Errors
Indicates if the state run failed. The state might fail for several reasons:
  • A required file does not exist or is inaccessible due to filesystem permissions.
  • A network resource or service is unavailable.
  • There is a package dependency conflict so the state cannot be applied.
The
Comments
column provides more details about the cause of the state failure.
Succeeded
Indicates the overall percentage of states that succeeded.
Changed
Indicates the overall percentage of states that resulted in changes for the minion.
Changes
Indicates if the most recent highstate run resulted in changes.
If you click a minion ID, you can view all previous state runs for a minion.
All state runs for one Salt minion in the Highstate workspace
The following information is provided:
Term
Description
Succeeded
Indicates if the state run succeeded or failed.
Changes
Indicates if the most recent highstate run resulted in changes.
Environment
The Salt environment that contains the state file, for example,
base
or
dev
.
SLS file
The file that contains the state declaration, for example,
centos-config.sls
.
State
The ID declaration for the state, for example,
common-centos-packages
.
Duration
Duration for the state run in milliseconds (ms).
Comment
A message that provides further details about the success or failure of the state run.
JID
The job identification number for the
state.highstate
run.
To run an ad-hoc highstate job at the minion level:
  1. In
    Automation Config
    , open the
    Highstate
    workspace.
  2. Select the minion or minions you want to run highstate on.
  3. Click
    Run Highstate.
  4. In the dialog box, click
    Run Highstate
    .

View highstate activity

The
Activity
tab in the
Highstate
workspace is used to monitor the status of highstate runs. The
Activity
tab gives visibility to many types of events and activities, such as scheduled jobs or ad-hoc jobs.
An interactive chart in the Highstate workspace that shows the number of
                    minions with and without changes over the past 30 days
The interactive chart in the
Activity
tab includes several reports to provide an overview of previous highstate runs in your
Automation Config
environment. You can view results from the past 24 hours up to the past 30 days for each report.
Report
Description
Minion: average duration
Displays the average duration for state runs across all minions, over time.
Minions: changes vs no changes
Displays the number of minions that reported changes versus minions that did not report changes, over time.
Minions: success vs failure
Displays the numbers of minions that reported successful state runs versus minions that reported failed state runs, over time.
States: changes vs no changes
Displays the number of states that resulted in changes versus states that did not result in changes, over time.
States: success vs failure
Displays the number of states that succeeded versus states that failed.
The
Activity
tab has three sections that sort jobs by status:
  • Completed
    : used to monitor the status of completed highstate jobs.
  • In Progress
    : used to monitor the status of highstate jobs that are currently running.
  • Upcoming
    : used to monitor the status of upcoming highstate jobs.
The table in the
Activity
tab shows information about the job's status, origin, schedule (if available), job, target group, and associated job ID (JID). You can filter the columns as needed.