Filtering and Deduplication

metrics and logic during ingestion, vrm compares each batch of findings against the most recently processed export to detect and exclude previously seen data this ensures only new or modified findings are enriched metrics tracked include number unseen findings number of findings not present in the previous export deduplication count number of duplicates excluded based on export comparison excluded findings count entries excluded through custom filtering logic message description of export status (for example, new export or existing export used ) vrm filtering activity record the vrm filtering activity record logs ingestion filtering outcomes for each batch it provides detailed insights into deduplication and exclusion logic and supports tuning of ingestion performance key fields include script internal runtime time taken by the python component to compare findings memory used bytes memory consumed during the filtering process unique excluded values count number of unique values that triggered exclusion excluded findings count total findings skipped due to custom exclusion rules number unseen findings , deduplication count , message this record is created automatically during each ingestion batch and is accessible via the filtering activity app in turbine reviewing this data helps teams fine tune ingestion batch sizes and exclusion rules tuning recommendations begin with a page size of 1000 for typical finding volumes use number unseen findings and deduplication count trends to optimize the page size adjust filtering criteria if too many findings are skipped or duplicated