Log Sources define how raw events are parsed and transformed before rule matching. Each Log Source contains:

Parsing Script – DSL query to extract and transform fields with functions chained using pipes (|).
Field Mapping – YAML mapping to align parsed fields with Sigma rule fields

Log Source configuration is added to a Pipeline to ensure proper event parsing and field mapping.

Create Log Source

Click the Create Log Source button on the Settings → Log Sources page.
Fill in the configuration Name. It's recommended to name it according to the log source of the topic to be used in the Pipeline.
Select one or multiple topics in the Test Topic dropdown. These are the topics from which a sample of events will be taken.
Click the Get Events button to view a sample of events from the selected topics. On the Event Sample panel, click the Show Text icon next to an event to see its full text.
Parse the sample events:
- Review the events.
- Write a parsing script using the parsing DSL in the Parsing Script editor to get field-value pairs.
- Click Run Test to apply the script to the event sample.
- Click the arrow icon next to a raw event to expand its parsed field-value pairs.
- Once the script parses all the field-value pairs of interest, go to the next step.
Expand the Detect section and select the repositories or folders to be used in the Pipeline with the topics selected in the Transform section.
Click the Paste fields from Repositories button to get all the field names used in detection rules from the selected Repositories. Note that at this stage they are commented out with #.
Click the AI Generate button to open the flow of automated mapping generation.
In the modal that appears, click:
- Generate with Uncoder AI if you have the SOC Prime Platform API key with AI features permission (see the SOC Prime Repositories section). A request to Uncoder AI's LLM is made to map the original Sigma fields to the fields from sample events. You'll get the resulting mapping in the editor that appears in the modal. Review it thoroughly.
- Copy Prompt if you don't have the API key or Internet connection. The prompt for mapping generation (instructions to the model together with the original Sigma and sample event field sets) is copied to the clipboard. Use it with an LLM of your choice and paste the resulting mapping in the YAML format in the editor that appears.
Click the Apply Mapping button to apply the mapping to the Mapping (YAML) editor. The mapping has a YAML format where the Sigma field name is key and log event field name is value. If the same Sigma field name is mapped to multiple log event field names, the following format is used:
```
Sigma_field:
  - first_log_event_field
  - second_log_event_field
```
Click Run Test to test the mapping with the sample events.
Expand the field-value pairs to review the mapped fields.
Click the Create button to save the Log Source configuration.

Edit or Delete Log Source

To edit a Log Source:

Go to the Settings → Log Sources page and click the tile of the Log Source you want to edit.
Make updates to the configuration and click the Update button.

To delete a Log Source:

Go to the Settings → Log Sources page and click the tile of the Log Source you want to delete.
Scroll down to the bottom of the page and click the Delete button.
Confirm the action.

Note:

You cannot delete a Log Source that's added to a Pipeline. First, remove the Log Source from all Pipelines, and then delete it.

Parsing DSL

DetectFlow provides a domain-specific language (DSL) for parsing event data. Functions are chained using pipes (|) and support nested field paths using dot notation (e.g., "user.profile.name"):

function_name(parameter="value", parameter2="value2")
| another_function(field="data")

Available Functions

parse_json

Parses a JSON string from a field.

Parameters:

field (required): Name of the field containing the JSON string. Supports nested paths (e.g., "event.raw").
in_place (optional, default: False): If True, replaces the field value with parsed JSON. If False, returns only the parsed JSON value (must be a dict).

Examples:

# Parse JSON and return the parsed value
parse_json(field="raw")

# Parse JSON in place (replaces field with parsed value)
parse_json(field="raw", in_place=True)

# Parse nested field
parse_json(field="event.raw", in_place=True)

# Case-insensitive boolean values are supported
parse_json(field="raw", in_place=true)

Behavior:

When in_place=False: Returns only the parsed JSON (must be a dict)
When in_place=True: Modifies the input data dictionary and returns it
Silently ignores if field is missing, None, empty string "", empty array [], or empty dict {}
Raises ParseJsonFunctionError if JSON parsing fails or parsed value is not a dict when in_place=False

regex

Extracts data from a field using a regular expression pattern.

Parameters:

pattern (required): Regular expression pattern with named groups
field (required): Name of the field to apply the regex to. Supports nested paths (e.g., "event.log")

Examples:

# Pattern first, field second
regex(pattern="^(?P<ip>\\S+) .*", field="log")

# Field first, pattern second (both orders supported)
regex(field="log", pattern="^(?P<ip>\\S+) .*")

# Apply regex to nested field
regex(field="event.log", pattern="^(?P<ip>\\S+) .*")

Returns: Dictionary with named groups from the regex pattern.

extract

Extracts a nested dictionary from a field and merges it with the parent dictionary.

Parameters:

field (required): Name of the field containing the nested dictionary to extract. Supports nested paths (e.g., "winlog.event_data")

Examples:

# Extract a nested user object
extract(field="user")

# Example transformation:
# Input:  {'user': {'user_name': 'test', 'user_id': '1'}, 'some_field': 'some_value'}
# Output: {'user_name': 'test', 'user_id': '1', 'some_field': 'some_value'}

# Extract from nested path
extract(field="winlog.event_data")

# Empty dict is also extracted (field is removed):
# Input:  {'empty': {}, 'other': 'value'}
# Output: {'other': 'value'}

Behavior:

Extracts the nested dictionary from the specified field (supports nested paths)
Merges all fields from the nested dictionary into the parent
Removes the original nested field
Silently ignores if field is missing, None, empty string "", or empty array []
Empty dict {} is valid and will be extracted (field removed, nothing merged)
Raises an error only if field exists and is not a dictionary

parse_win_event_log

Parses Windows Event Log text format (typically exported from Splunk or Windows Event Viewer).

Parameters:

field (required): Name of the field containing the Windows Event Log text

Input Format:

The function expects Windows Event Log text in the following format:

01/01/2025 10:00:00 AM
LogName=Security
EventCode=4648
EventType=0
ComputerName=example.com
SourceName=Microsoft Windows security auditing.
Type=Information
RecordNumber=12345
Keywords=Audit Success
TaskCategory=Logon
OpCode=Info
Message=A logon was attempted using explicit credentials.

Subject:
    Security ID:        NT AUTHORITY\SYSTEM
    Account Name:        EXAMPLE$
    Account Domain:        EXAMPLE
    Logon ID:        0x3E7

Network Information:
    Network Address:    10.10.10.10
    Port:            57200

How it works:

Parses key-value pairs in the format Key=Value or Key: Value
Extracts the EventID from the EventCode field
Handles structured sections (e.g., "Subject:", "Network Information:")
Maps section fields to standardized field names based on event type
Skips the first line (timestamp)
Handles multi-line values (continuation lines)

Returns: Dictionary with parsed event fields. Field names are standardized based on the event type and section mapping.

Examples:

# Basic usage
parse_win_event_log(field="log_text")

# In a query chain
query = """
parse_win_event_log(field="raw_log")
"""

Output Example:

For the input above, the function returns:

{
    "EventID": "4648",
    "LogName": "Security",
    "EventType": "0",
    "ComputerName": "example.com",
    "SourceName": "Microsoft Windows security auditing.",
    "Type": "Information",
    "RecordNumber": "12345",
    "Keywords": "Audit Success",
    "TaskCategory": "Logon",
    "OpCode": "Info",
    "Message": "A logon was attempted using explicit credentials.",
    "SubjectUserSid": "NT AUTHORITY\\SYSTEM",
    "SubjectUserName": "EXAMPLE$",
    "SubjectDomainName": "EXAMPLE",
    "SubjectLogonId": "0x3E7",
    "IpAddress": "10.10.10.10",
    "IpPort": "57200"
}

Notes:

Field names in sections are mapped to standardized names based on event ID and section type
Fields outside of sections are preserved as-is
Empty logs return an empty dictionary
The parser handles various Windows Event Log formats and event types

parse_wincollect_win_event_log

Parses Windows Event Log text as forwarded by WinCollect (AgentDevice=WindowsLog). WinCollect flattens the multi-line Windows Event Viewer text into a single-line, tab- and double-space-delimited string; this function reshapes that string back into the layout expected by parse_win_event_log and reuses its parsing and field-mapping logic.

Parameters:

field (required): Name of the field containing the WinCollect-forwarded Windows Event Log text

Input Format:

Typical WinCollect payload (tabs between top-level fields; section fields packed with double spaces after Message=):

<13>Jul 13 15:18:32 HOST.sub.domain.name AgentDevice=WindowsLog	AgentLogFile=Security	PluginVersion=7.3.1.43	Source=Microsoft-Windows-Security-Auditing	Computer=HOST.sub.domain.name	OriginatingComputer=10.10.10.10	User=	Domain=	EventID=4662	EventIDCode=4662	EventType=8	EventCategory=12804	RecordNumber=3703867	TimeGenerated=1783945048	TimeWritten=1783945048	Level=Log Always	Keywords=Audit Success	Task=SE_ADT_OBJECTACCESS_OTHER	Opcode=Info	Message=An operation was performed on an object.  Subject :  Security ID:  NT AUTHORITY\LOCAL SERVICE  Account Name:  LOCAL SERVICE  Account Domain:  NT AUTHORITY  Logon ID:  0x3E5  Object:  Object Server:  LSA  Object Type:  SecretObject  Object Name:  Policy\Secrets\$MACHINE.ACC  Handle ID:  0x27058dfe690  Operation:  Operation Type:  Query  Accesses:  Query secret value       Access Mask:  0x2  Properties:  -  Additional Information:  Parameter 1:  -  Parameter 2:  -

How it works:

Reshapes the flattened WinCollect line back into the standard Windows Event Log layout
Splits the packed Message= value into individual fields (including section fields such as Subject/Object)
Ignores free-form message text so it isn’t mistaken for fields
Delegates to parse_win_event_log for the actual parsing and field mapping

Returns: Dictionary with parsed event fields. Section fields are mapped to standardized names the same way as parse_win_event_log.

Examples:

parse_wincollect_win_event_log(field="log_text")

Output Example:

For an object-access event like the input above, the function returns (some fields omitted):

{
    "EventID": "4662",
    "AgentLogFile": "Security",
    "Computer": "HOST.sub.domain.name",
    "Message": "An operation was performed on an object.",
    "SubjectUserSid": "NT AUTHORITY\\LOCAL SERVICE",
    "SubjectUserName": "LOCAL SERVICE",
    "SubjectDomainName": "NT AUTHORITY",
    "SubjectLogonId": "0x3E5",
    "ObjectServer": "LSA",
    "ObjectType": "SecretObject",
    "ObjectName": "Policy\\Secrets\\$MACHINE.ACC",
    "HandleId": "0x27058dfe690",
    "OperationType": "Query",
    "AccessList": "Query secret value",
    "AccessMask": "0x2",
}

Notes:

Prefer this function for QRadar-forwarded Windows logs; use parse_win_event_log for native Event Viewer / Splunk multi-line text
Missing fields return an empty dictionary; non-string field values raise ValueError
Trailing Microsoft explanatory sentences after section fields are dropped so they do not corrupt the last parsed value

Query Syntax

Queries are composed of function calls separated by pipes (|). Functions can be written with or without whitespace around parameters.

Basic Syntax

function_name(parameter="value", parameter2="value2")

Multiple Functions

Chain multiple functions using pipes:

parse_json(field="raw")
| regex(field="raw", pattern="^test")

Whitespace Support

All functions support flexible whitespace:

# All of these are valid:
parse_json(field="raw")
parse_json( field = "raw" )
parse_json(field="raw", in_place=True)
parse_json( field = "raw" , in_place = True )

Comments

Queries support # comments. Comments are stripped from the end of lines, but # characters inside string literals are preserved:

parse_json(field="raw") # This is a comment