• Overview
• AMA - Azure Monitoring Agent
• DCR - Data Collection Rule
  • Transformation
• DCE - Data Collection Endpoint
• LAW - Log Analytics Workspace
• Streams and Data Sources
• Misc References

Overview

DCR –> AMA –> DCE –> LAW

AMA - Azure Monitoring Agent

• Installed when you create an Azure VM
• Can be updated / controlled via Azure Portal for in VM Extensions

DCR - Data Collection Rule

• Applies via Azure Monitor (portal, PS, API, etc)
• Defines the VMs (or services/resources) the DCR applies to
• Defines the things it collections
• Defines the destination for those things
• Can define transformations of those things to the destination.
  • Most common transformation is to reduce the amount of data being imported after it has been collected
• Must be in same region as the Log Analytics Workspace it is sending things to

Transformation

TBC - Most common transformation is: Data ingestion duplication avoidance

• free if not adding data
• Easiest way to modify
  1. Make in portal
  2. Deploy
  3. Go to export template
  4. download a copy
  5. Find the dataflow section
  6. add in your transformation in a new element. eg

          "dataFlows": [
                    {
                        "streams": [
                            "Microsoft-CommonSecurityLog"
                        ],
                        "destinations": [
                            "DataCollectionEvent"
                        ],
                        "transformKql": "  source\n    |  where ProcessName !contains \"CEF\"\n"
                    }

This didn’t work for me… So, I tried this:

          "dataFlows": [
                    {
                        "streams": [
                            "Microsoft-CommonSecurityLog"
                        ],
                        "destinations": [
                            "DataCollectionEvent"
                        ],
                        "transformKql": "  source\n    |  where isnotempty(DeviceVendor)\n"
                    }
          ]

Because DeviceVendor is part of the CEF Standard. More details in an upcoming post.

Microsoft-CommonSecurityLog

My Microsoft-CommonSecurityLog was underlined yellow and saying it wasn’t valid. It still accepted it and it still worked. This may have been an issue since I moved my sentinel-enabled LAW around. I also noted that when typing it in I had to add a space that before the transformkql came up in the drop down so I knew what I was doing was ‘somewhat’ valid.

Notes about KQL for Transformations: Supported KQL features in Azure Monitor transformations

DCE - Data Collection Endpoint

Apparently not needed unless using Azure Private Link. I think can be used if you have some machines that can’t see the internet without private link but not sure.

LAW - Log Analytics Workspace

TBC

Streams and Data Sources

From Structure of a data collection rule (DCR) in Azure Monitor

Input-streams:

• If this is a standard data type such as a Windows event, then the stream will be in the form Microsoft-<TableName>. If it’s a custom type, then it will be in the form Custom-<TableName>
• Valid Source Types - there are more
• Full list of 240 possible streams values (Oct 20, 2022)

Azure Monitor data sources and data collection methods

Misc References

https://learn.microsoft.com/en-us/azure/azure-monitor/agents/troubleshooter-ama-windows?tabs=WindowsPowerShell#linux-troubleshooter
https://learn.microsoft.com/en-us/azure/azure-monitor/agents/troubleshooter-ama-linux?tabs=redhat%2CGenerateLogs
https://learn.microsoft.com/en-us/azure/sentinel/connect-cef-syslog-ama?tabs=portal#test-the-connector - contains commands to test syslog and the connector