Filtering and Deduplication
Metrics and Logic
During ingestion, VRM compares each batch of findings against the most recently processed export to detect and exclude previously seen data. This ensures only new or modified findings are enriched.
Metrics tracked include:
- number_unseen_findings: Number of findings not present in the previous export
- deduplication_count: Number of duplicates excluded based on export comparison
- excluded_findings_count: Entries excluded through custom filtering logic
- message: Description of export status (for example, New Export or Existing Export used)
VRM - Filtering Activity Record
The VRM - Filtering Activity record logs ingestion filtering outcomes for each batch. It provides detailed insights into deduplication and exclusion logic and supports tuning of ingestion performance. key fields include:
- Script_Internal_Runtime: Time taken by the Python component to compare findings
- Memory_Used_Bytes: Memory consumed during the filtering process
- Unique_Excluded_Values_Count: Number of unique values that triggered exclusion
- Excluded_Findings_Count: Total findings skipped due to custom exclusion rules
- number_unseen_findings, deduplication_count, message
This record is created automatically during each ingestion batch and is accessible via the Filtering Activity app in Turbine. Reviewing this data helps teams fine-tune ingestion batch sizes and exclusion rules.
Tuning Recommendations
- Begin with a page size of 1000 for typical finding volumes.
- Use number_unseen_findings and deduplication_count trends to optimize the page size.
- Adjust filtering criteria if too many findings are skipped or duplicated.