TL;DR

Introduction

So you’ve got CEF Messages you want to go into Azure Sentinel. That sounds like a neat idea doesn’t it! BULLSHIT.

SAY WHAT AGAIN

When reviewing a setup to get CEF messages into Sentinel, I found there was far more messages going in than their should be and that most of those messages were not valid. That sounds like $$$ going down the drain. So I went over the setup trying to find this excess.

Thats where this came from. Hopefully it will help someone.

"SYSLOG, I say no, no, no!"

In this scenario i’m not sending SYSLOG messages into Sentinel. That can be done but wasn’t the point of this post. However some of the steps and/or thinking may still be useful.

Overview and Scenario

In this scenario we had a NGFW in an private network and a NGAV/EDR SaaS solution configured to send message to a new Ubuntu based syslog server we created running rsyslog and using AMA for Linux 1.33 (more importantly above 1.28). We then have a Azure Monitor Data Collection Rule getting syslog messages with a facility of USER/LOG_USER into the Log Analytics Workspace that is used by Sentinel. Thats a mouthful of crap. Lets break it down.

Technobabble

  1. The NGFW can send CEF messages to a Syslog server and to a specific facility over a private network.
  2. The NGAV/EDR solution has a Firehose API Client that routes messages to a syslog server (without facility control).
  3. The syslog server is an Ubuntu 22 LTS server in Azure, running rsyslog and AMA.
  4. The version of AMA matters as versions 1.28 and greater simplify the ‘pickup’ process. Earlier versions had different agents and methods, including unix sockets and the (now legacy) Log Analytics Agent and all sorts of other crap. Don’t get me wrong AMA isn’t great but its better than what we had.
  5. AMA includes an rsyslog configuration that routes a copy of all syslog messages to the AMA.
  6. An Azure Monitor Data Collection Rule (DCR) was created by Azure Sentinel that collects everything sent to USER/LOG_USER. That DCR gets deployed to the syslog server via AMA.
  7. The DCR then routes the syslog messages it collects to the public Data Collection Endpoint (DCE).
  8. The DCE processes the other part of the DCR which is its destination. Since this DCR was created by sentinel, its going to the same workspace sentinel is running in.

Device –> Rsyslog –> AMA –> DCE –> DCR –> LAW
API Client –> Rsyslog –> AMA –> DCE –> DCR –> LAW

flowchart LR
  Device --> Rsyslog;
  subgraph SysLog Server
    direction LR
    APIClient --> Rsyslog --> AMA
    DCR1[DCR] --> AMA
  end
  AMA --> DCE;
  DCE --> DCR2[DCR];
  DCR2[DCR] --> LAW;

There were dreams of having this one syslog server being used for everything by including some other rsyslog configs for log management but that was scaled back to focus on noise reduction and until we can refine the rsyslog configs.

Data Collection Endpoint

I mention Data Collection Endpoint’s (DCE) above. Mostly to be aware of the component. You can create Data Collections Endpoints but you don’t need to unless you’re using Azure Private Links.

Now we have syslog messages going into our Log Analytics Workspace and there is a lot, and I think a fair bit of it is noise.

Syslog Noise

No SYSLOG for Sentinel

In this scenario i’m not sending SYSLOG messages into Sentinel. That can be done but wasn’t the point of this post.

Here is how I started to determine the level of noise. DeviceVendor is part of the CEF standard. So try this KQL Query in the Log Analytics Workspace and adjust your time period/limit per your logging load (Start small then increase either the limit or the time range to get an idea):

CommonSecurityLog
| summarize Count=count() by DeviceVendor

Messages where the DeviceVendor is blank is syslog noise (Noise may be unfair but in this scenario they shouldn’t be there as we are not sending regular SYSLOG messages to Sentinel. So they are not CEF messages). Also note that those that know syslog facilities better than me may not have chosen the USER facility. I didn’t have a choice given the Firehose API client. Maybe I could roll my own client? ARE YOU HIGH?

ARE YOU HIGH

Anyway back on topic…

Where is the noise coming from? I had to muck around with syslog configs to get an idea. But here are some examples I found:

So we need to reduce the noise. There are two approaches I came up with. DCR Transformation & Rsyslog config modification.

DCR Transformation

MS suggest modifying the DCR with something like this:

          "dataFlows": [
                    {
                        "streams": [
                            "Microsoft-CommonSecurityLog"
                        ],
                        "destinations": [
                            "DataCollectionEvent"
                        ],
                        "transformKql": "  source\n    |  where ProcessName !contains \"CEF\"\n"
                    }
          ]

I did steal the above from a specific example scenario but from what I could see CEF doesn’t use the ProcessName field. So, we’re using this transformation instead:

          "dataFlows": [
                    {
                        "streams": [
                            "Microsoft-CommonSecurityLog"
                        ],
                        "destinations": [
                            "DataCollectionEvent"
                        ],
                        "transformKql": "  source\n    |  where isnotempty(DeviceVendor)\n"
                    }
          ]

Note that not all KQL is supported in Transformations. Refer to Supported KQL features in Azure Monitor transformations

Editing DCRs

Creating and editing DCR’s usually requires hard-coding JSON and submitting it via the API.

“Cool Story Bro, but I really don’t want to fuck around that much…”.

Cool Story Bro

Me either so there is a cheat if your Transformation is simple.

  1. Go to Azure Monitor in the Azure Portal
  2. Go to Settings –> Data Collection Rules
  3. Select your DCR that gets your syslog’s/CEF messages
  4. Go to Automation –> Export Template
  5. Click Download to take a backup
  6. Click Deploy at the top
  7. Click Edit Template
  8. Find the dataflow section
  9. Add in your transformation in a new element. similar to the above example

How can I check the DCR Transformation?

One thing I found is that it can be hard to check the DCR Transformation exactly . Here are some ways I found:

  1. Metrics of the DCR - this method I found best as you can see Logs coming in, log errors, transformation time and the main one for me was Log Rows Dropped. This one moved a lot.
  2. Workbooks on usage of the LAW, especially if its Sentinel-enabled.
  3. Trying to mirror / watch / capture what the syslog server output. Too hard.
  4. Try the noisy KQL query from above again (this was my litmus test).

Seeing the log rows dropped metric really indicated this was working. But…

Is this really a fix?

Technically yes…

Technically Correct

but I understand what you mean. You’re still sending a lot of data in the first place, then filtering it out with the DCR Transformation. Depending on the scale, thats money somewhere.

The goal was to use this as a single (or HA) syslog server for all needs instead of running multiple servers.

In my situation I got there by setting the CEF Connection via AMA for Sentinel, in which you must choose a SYSLOG facility to bring in and a level. One of my input devices I could change the facility and one I couldn’t. So I may be stuck with User/Log_User Facility. But we could get ‘creative’ with the RSyslog config file as mentioned above. Thats the next step to try.

Rsyslog conf modification

If you setup log forwarding via:

  1. The Azure Monitoring Agent and a DCR
  2. Setup the Sentinel CEF Connector and run Forwarder_AMA_installer.py

on your syslog server and you’re running Azure Monitoring Agent v1.28 or newer, you will end up with /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf being created. It looks like:

# Azure Monitor Agent configuration: forward logs to azuremonitoragent

template(name="AMA_RSYSLOG_TraditionalForwardFormat" type="string" string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
# queue.workerThreads sets the maximum worker threads, it will scale back to 0 if there is no activity
# Forwarding all events through TCP port
*.* action(type="omfwd"
template="AMA_RSYSLOG_TraditionalForwardFormat"
queue.type="LinkedList"
queue.filename="omfwd-azuremonitoragent"
queue.maxFileSize="32m"
queue.maxDiskSpace="1g"
action.resumeRetryCount="-1"
action.resumeInterval="5"
action.reportSuspension="on"
action.reportSuspensionContinuation="on"
queue.size="25000"
queue.workerThreads="100"
queue.dequeueBatchSize="2048"
queue.saveonshutdown="on"
target="127.0.0.1" Port="28330" Protocol="tcp")

Azure Monitoring Agent before 1.28

Before 1.28 you will end up with 2 files in /etc/rsyslog.d/. This is because AMA uses a Unix Socket to get the syslog’s, where after 1.28 its back to like it was in the Log Analytics Agent (something running on a port)
Before 1.28: File Missing
After 1.28: Forwarder After 1.28

One option to consider is adjusting the rsyslog file. One Idea I was considering stealing was back from when they used the Log Analytics Agent with syslog forwarding:

if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226

So how does this merging look? Here is my attempt but note i’m not an rsyslog expert.

# Azure Monitor Agent configuration: forward logs to azuremonitoragent

template(name="AMA_RSYSLOG_TraditionalForwardFormat" type="string" string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
# queue.workerThreads sets the maximum worker threads, it will scale back to 0 if there is no activity
# Forwarding all events through TCP port
if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then {
  action(type="omfwd"
  template="AMA_RSYSLOG_TraditionalForwardFormat"
  queue.type="LinkedList"
  queue.filename="omfwd-azuremonitoragent"
  queue.maxFileSize="32m"
  queue.maxDiskSpace="1g"
  action.resumeRetryCount="-1"
  action.resumeInterval="5"
  action.reportSuspension="on"
  action.reportSuspensionContinuation="on"
  queue.size="25000"
  queue.workerThreads="100"
  queue.dequeueBatchSize="2048"
  queue.saveonshutdown="on"
  target="127.0.0.1" Port="28330" Protocol="tcp")
}

Things you may need to consider with the above script.

How can I check the new Rsyslog config working?

Using the same Azure Metrics I used for the DCR, you should have seen a drop, but I didn’t see anything significant change since the DCR Transformation drop, Especially given what I was seeing before the DCR Transformation. The metric of log rows dropped is still high. As such I may need to disable the transformation to assess the impact of the rsyslog changes.

Summary

So we haven’t worked out the syslog part (yet) but we’re making progress. Some reading indicates that Syslog-NG may be able to modify messages before being processed which may be a better way to attack the problem but I haven’t looked into that yet.

Some of the problem may be the choice of USER/LOG_USER facility level (where I had the choice) and a lack of common usages or lack of understanding the intent of specific facility levels. I get the feeling USER/LOG_USER is a dumping ground. My goal of just ignoring the facility level and only forwarding CEF messages should have taken care of this but the Rsyslog config didn’t work.

"No SYSLOG, No Cry"

Another reminder that in this scenario i’m not sending SYSLOG messages into Sentinel. That can be done but wasn’t the point of this post. However some of the steps and/or thinking may still be useful.

Here are the takeaways:

References

https://learn.microsoft.com/en-us/azure/sentinel/cef-syslog-ama-overview?tabs=single#data-ingestion-duplication-avoidance
https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-transformations-kql
https://www.rsyslog.com/storing-messages-from-a-remote-system-into-a-specific-file/
https://www.rsyslog.com/normalizing-cisco-asa-messages/
https://www.rsyslog.com/doc/configuration/index.html
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/deployment_guide/s1-basic_configuration_of_rsyslog#s2-Filters
https://en.wikipedia.org/wiki/Syslog
KQL Queries| WTF
AMA DCR LAW | WTF
Misc References | WTF