Security Filters with the Cloud SIEM API
Overview
The Cloud SIEM product analyzes your ingested logs to detect threats in real time, such as by matching logs with threat intelligence, or by applying detection rules to detect attacks or anomalies.
Datadog charges for analyzed logs based on the total number of gigabytes ingested and analyzed by the Datadog Cloud SIEM service. By default, Cloud SIEM analyzes all your ingested logs to maximize detection coverage. However, by using the Cloud SIEM API, you can programmatically setup Security Filters to configure which subset of ingested logs to analyze.
The following examples are covered in this guide:
Note: Security Filters are only required to control logs analyzed by the Cloud SIEM product. You do not need to write Security Filters to exclude logs generated by the Datadog Agent as part of the Cloud Security Management Threats (source:runtime-security-agent) and Cloud Security Management Misconfigurations (source:compliance-agent) products, as they’re not billed as analyzed logs regardless.
Prerequisites
An API key and an application key from an admin user is required to use the API. These are available in your Datadog account API key page. Replace <DATADOG_API_KEY> and <DATADOG_APP_KEY> with your Datadog API key and your Datadog application key.
This guide features curl examples. Install cURL if you do not have it installed, or reference additional language examples for this API endpoint in the API documentation.
Examples
Add an exclusion
By default, a single Security Filter exists that analyzes all ingested logs. It’s named all ingested logs and has a query of *. You can customize it by adding an exclusion to exclude a subset of logs based on their tags. To do so, you first need to retrieve the list of Security Filters in order to get the filter’s id.
API call:
curl -L -X GET 'https://api.
Response:
{
"data": [
{
"attributes": {
"is_enabled": true,
"is_builtin": true,
"name": "all ingested logs",
"filtered_data_type": "logs",
"exclusion_filters": [],
"version": 1,
"query": "*"
},
"type": "security_filters",
"id": "l6l-rmx-mqx"
}
]
}
In this example, the filter’s id is "l6l-rmx-mqx". You can then modify it to add an exclusion, for example exclude all the logs tagged with env:staging.
Note: version indicates the current version of the filter you want to update. This field is optional. If it is not provided, the latest version is updated.
API call:
curl -L -X PATCH 'https://api.
Response:
{
"data": {
"attributes": {
"is_enabled": true,
"is_builtin": true,
"name": "all ingested logs",
"filtered_data_type": "logs",
"exclusion_filters": [
{
"query": "env:staging",
"name": "exclude env:staging"
}
],
"version": 2,
"query": "*"
},
"type": "security_filters",
"id": "l6l-rmx-mqx"
}
}
Create a custom filter
You can also create custom Security Filters in order to restrict analysis to explicitly specified logs. For example, you can choose to analyze logs from AWS CloudTrail with a filter that matches only source:cloudtrail.
API call:
curl -L -X POST 'https://api.
Response:
{
"data": {
"attributes": {
"is_enabled": true,
"is_builtin": false,
"name": "cloudtrail",
"filtered_data_type": "logs",
"exclusion_filters": [],
"version": 1,
"query": "source:cloudtrail"
},
"type": "security_filters",
"id": "qa6-tzm-rp7"
}
}
Note: version indicates the current version of the filter you want to update. This field is optional. If it is not provided, the latest version is updated.
Security Filters are inclusive, which means a given log is analyzed if it matches at least one Security Filter. If you’re aiming to specify a subset of logs to analyze, you likely also would want to disable the default built-in filter named all ingested logs. You would do so by setting its is_enabled attribute to false, as follows:
API call:
curl -L -X PATCH 'https://api.
Response:
{
"data": {
"attributes": {
"is_enabled": false,
"is_builtin": true,
"name": "all ingested logs",
"filtered_data_type": "logs",
"exclusion_filters": [
{
"query": "env:staging",
"name": "exclude env:staging"
}
],
"version": 3,
"query": "*"
},
"type": "security_filters",
"id": "l6l-rmx-mqx"
}
}
Note: version indicates the current version of the filter you want to update. This field is optional. If it is not provided, the latest version is updated.
If you aim to only analyze explicitly specified categories of logs, be cautious not to exclude logs that contain valuable security-relevant users and entities, or key sources of security logs. The tables below provide useful examples.
Key users and entities
| Name | Query |
|---|
| All named events | @evt.name:* |
| All client IPs | @network.client.ip:* |
| All destination IPs | @network.destination.ip:* |
| All users | @usr.id:* OR @usr.name:* @usr.email:* |
| All hosts | host:* OR instance-id:* |
Key security sources
| Name | Query |
|---|
| AWS Security Logs | source:(cloudtrail OR guardduty OR route53) |
| AWS Network Logs | source:(vpc OR waf OR elb OR alb) |
| Google Cloud Logs | source:gcp* |
| Azure Logs | source:azure* |
| Kubernetes Audit Logs | source:kubernetes.audit |
| Identity Provider Logs | source:(okta OR gsuite OR auth0) |
| CDN Logs | source:(cloudfront OR cloudflare OR fastly) |
| Web Server Logs | source:(nginx* OR apache OR iis) |
