Here are instructions for integrating your VMware Tanzu Application Service for VMs (TAS for VMs) apps with OMS Log Analytics.
Operations Management Suite (OMS) Log Analytics is a monitoring service for Microsoft Azure. The OMS Log Analytics Firehose Nozzle is a TAS for VMs component that forwards metrics from the Loggregator Firehose to OMS Log Analytics.
This topic assumes you are using the latest version of the Cloud Foundry Command Line Interface (cf CLI) and a working TAS for VMs deployment on Azure.
Step 1: Create an OMS workspace in Azure
To create an OMS workspace, see Get started with Log Analytics in the Microsoft Azure documentation.
Step 2: Deploy the nozzle to TAS for VMs
To deploy the OMS Log Analytics Firehose nozzle to TAS for VMs:
-
Authenticate to your TAS for VMs instance. For more information, see Creating and Managing Users with the UAA CLI (UAAC) and Orgs, Spaces, Roles, and Permissions. Run:
cf login -a https://api.YOUR-DOMAIN -u YOUR-USERNAME --skip-ssl-validation
Where:
YOUR-DOMAIN
is your domain.YOUR-USERNAME
is your TAS for VMs username.
-
To create a new TAS for VMs user and grant it access to the Loggregator Firehose using the UAA CLI (UAAC):
-
Target your UAA server by running:
uaac target uaa.YOUR-DOMAIN --skip-ssl-validation
Where
YOUR-DOMAIN
is your domain. -
Obtain an access token for the admin client by running:
uaac token client get admin
-
Create a new user by running:
uaac user add USERNAME -p PASSWORD --email EMAIL
Where:
USERNAME
is a new username.PASSWORD
is a password.EMAIL
is an email address.
-
Grant the new user admin permissions by running:
uaac member add cloud_controller.admin USERNAME
Where
USERNAME
is the username you set in the previous step. -
Grant the new user permission to read logs from the Loggregator Firehose endpoint by running:
uaac member add doppler.firehose USERNAME
Where
USERNAME
is the username you set.
-
-
Download the OMS Log Analytics Firehose Nozzle BOSH release from Github. Clone the repository and navigate to the
oms-log-analytics-firehose-nozzle
directory by running:git clone https://github.com/Azure/oms-log-analytics-firehose-nozzle.git cd oms-log-analytics-firehose-nozzle
-
Set the following environment variables in the OMS Log Analytics Firehose Nozzle manifest:
Environment Variable Description applications: - name: oms_nozzle ... env: OMS_WORKSPACE: YOUR-WORKSPACE-ID OMS_KEY: YOUR-OMS-KEY
Enter the ID and key value for your OMS workspace. OMS_POST_TIMEOUT: 10s
(Optional) Set the HTTP post timeout for sending events to OMS LogmAnalytics. The default value is 10 seconds. OMS_BATCH_TIME: 10s
(Optional) Set the interval for posting a batch to OMS. The default value is 10 seconds.
For more information, see Configure Additional Logging.OMS_MAX_MSG_NUM_PER_BATCH: 1000
(Optional) Set the maximum number of messages to include in an OMS batch. The default amount is 1000.
For more information, see Configure Additional Logging.FIREHOSE_USER: YOUR-FIREHOSE-USER FIREHOSE_USER_PASSWORD: YOUR-FIREHOSE-PASSWORD
Enter the username and password for the Firehose user you created in Step 2c. API_ADDR: https://api.YOUR-DOMAIN
Enter the URL of your API endpoint. DOPPLER_ADDR: wss://doppler.YOUR-DOMAIN:443
Enter the URL of your Loggregator Traffic Controller endpoint. EVENT_FILTER: YOUR-LIST
(Optional) Enter the event types you want to filter out in a comma-separated list. The valid event types are METRIC
,LOG
, andHTTP
.IDLE_TIMEOUT: 60s
(Optional) Set the duration for the Firehose keep-alive connection. The default time is 60 seconds. SKIP_SSL_VALIDATION: TRUE-OR-FALSE
Set this value to TRUE
to allow insecure connections to the UAA and the Traffic Controller. To block insecure connections to the UAA and Traffic Controller, set this value toFALSE
.LOG_LEVEL: INFO
(Optional) Change this value to increase or decrease the amount of logs. Valid log levels in increasing order include INFO
,ERROR
, andDEBUG
. The default value isINFO
.LOG_EVENT_COUNT: TRUE-OR-FALSE
Set this value to TRUE
to log the total count of events that the nozzle has sent, received, and lost. OMS logs this value asCounterEvents
.
For more information, see Configure Additional Logging.LOG_EVENT_COUNT_INTERVAL: 60s
(Optional) Set the time interval for logging the event count to OMS. The default interval is 60 seconds.
For more information, see Configure Additional Logging. -
Push the app by running:
cf push
Step 3: View logs in OMS Portal
Import the TAS for VMs OMS view to your OMS Portal to view visualized logs and metrics. You can also create alert rules for specific events.
The OMS view of TAS for VMs is not yet available in the OMS Solutions Gallery. You can add it manually to view your logs in OMS Portal.
Import the OMS view
To import the OMS view:
-
From the main OMS Overview page, go to View Designer.
-
Click Import.
-
Click Browse.
-
Select the Cloud Foundry (Preview).omsview file.
-
Save the view. The main OMS Overview page displays the Tile.
-
Click the Tile to view visualized metrics.
For more information, see Create custom views by using View Designer in Azure Monitor in the Azure documentation.
Create alert rules
For more information about OMS Log Analytics alerts, see Overview of alerts in Microsoft Azure in the Azure documentation.
Set alert queries
This section includes example queries that operators can set in the OMS Portal.
-
The following query alerts the operator when the nozzle sends a
slowConsumerAlert
to OMS:Type=CF_ValueMetric_CL Name_s=slowConsumerAlert
-
The following query alerts the operator when Loggregator sends an
LGR
to indicate problems with the logging process:Type=CF_LogMessage_CL SourceType_s=LGR MessageType_s=ERR
-
The following query alerts the operator when the number of lost events reaches a certain threshold, specified in the OMS Portal:
Type=CF_CounterEvent_CL Job_s=nozzle Name_s=eventsLost
-
The following query alerts the operator when the nozzle receives the
TruncatingBuffer.DroppedMessages
CounterEvent:Type=CF_CounterEvent_CL Name_s="TruncatingBuffer.DroppedMessages"
Step 4: Configure additional logging (optional)
OMS Log Analytics Firehose Nozzle forwards metrics from the Loggregator Firehose to OMS with minimal processing, but the nozzle can push additional metrics to OMS.
Logs sent and received, and events lost
If you set the LOG_EVENT_COUNT
environment variable to TRUE
in the manifest, the nozzle periodically sends the count of sent, received, and lost events to OMS. The value you set for the LOG_EVENT_COUNT_INTERVAL
determines how frequently the nozzle sends the count.
The nozzle does not count CounterEvents themselves in the sent, received, or lost event count.
The nozzle sends the count as a CounterEvent with a CounterKey of one of the following:
CounterEvent | CounterKey |
---|---|
nozzle.stats.eventsReceived | The number of events the Firehose has received during the interval |
nozzle.stats.eventsSent | The number of events the nozzle has successfully sent to OMS during the interval |
nozzle.stats.eventsLost | The number of events the nozzle has tried to send to OMS during the interval, but failed to send after 4 attempts |
In most cases, the total count of eventsSent
plus eventsLost
is less than the total eventsReceived
at the same time. The nozzle buffers some messages and posts them in a batch to OMS. Operators can adjust the buffer size by adjusting the OMS_BATCH_TIME
and OMS_MAX_MSG_NUM_PER_BATCH
environment variables in the manifest).
Log slow consumer alerts
The nozzle does not count ValueMetrics in the sent, received, or lost event count.
Loggregator sends the nozzle a slowConsumerAlert
in the following situations:
-
WebSocket sends the error code
ClosePolicyViolation (1008)
. -
The nozzle receives a CounterEvent with the value
TruncatingBuffer.DroppedMessages
.
In either case, the nozzle sends the slowConsumerAlert
event to OMS as the following ValueMetric:
ValueMetric | MetricKey |
---|---|
nozzle.alert.slowConsumerAlert | 1 |
For more information, see the Slow Nozzle Alerts section of the Loggregator Guide for TAS for VMs Operators topic.
Step 5: Scale the deployment (optional)
Scale the nozzle
If the nozzle is unable to keep up with processing logs from the Firehose, Loggregator alerts the nozzle. When the nozzle receives the alert, it sends a slowConsumerAlert
to OMS. If this happens, scaling up the nozzle minimizes data loss.
If an operator chooses to scale up their deployment, the Firehose evenly distributes events across all instances of the nozzle. For more information, see the Scaling Nozzles section of the Loggregator Guide for TAS for VMs Operators topic.
Operators can create an alert rule for the slowConsumerAlert
message. For more information, see Create Alert Rules.
Scale Loggregator
Loggregator sends LGR
log messages to indicate problems with the logging process. For more information, see Scaling Loggregator in Loggregator Guide for TAS for VMs Operators.
Operators can create an alert rule for the LGR
message. For more information, see Create Alert Rules.
Content feedback and comments