---
title: Filtering and Deduplication
slug: solutions/filtering-and-deduplication
docTags: 
createdAt: 2025-08-11T11:13:45.688Z
---

**Metrics and Logic**

During ingestion, VRM compares each batch of findings against the most recently processed export to detect and exclude previously seen data. This ensures only new or modified findings are enriched.

**Metrics tracked include:**

- **number\_unseen\_findings:** Number of findings not present in the previous export
- **deduplication\_count:** Number of duplicates excluded based on export comparison
- **excluded\_findings\_count:** Entries excluded through custom filtering logic
- **message:** Description of export status (for example, *New Export* or *Existing Export used*)

**VRM - Filtering Activity Record**

The **VRM - Filtering Activity** record logs ingestion filtering outcomes for each batch. It provides detailed insights into deduplication and exclusion logic and supports tuning of ingestion performance.
key fields include:

- **Script\_Internal\_Runtime**: Time taken by the Python component to compare findings
- **Memory\_Used\_Bytes**: Memory consumed during the filtering process
- **Unique\_Excluded\_Values\_Count**: Number of unique values that triggered exclusion
- **Excluded\_Findings\_Count**: Total findings skipped due to custom exclusion rules
- **number\_unseen\_findings**, **deduplication\_count**, **message**

This record is created automatically during each ingestion batch and is accessible via the **Filtering Activity** app in Turbine. Reviewing this data helps teams fine-tune ingestion batch sizes and exclusion rules.

**Tuning Recommendations**

- Begin with a page size of 1000 for typical finding volumes.
- Use **number\_unseen\_findings** and **deduplication\_count** trends to optimize the page size.
- Adjust filtering criteria if too many findings are skipped or duplicated.