Correlation

you can correlate records within a single application within swimlane turbine upon ingestion of new application records, turbine compares the new record to previous records that have correlation keys record correlation executes two tasks compares correlation key fields across records in a single application, looking for configurable similarities updates a correlation field in the associated records the following correlation field types have the below default configurations and will be compared with the matching preferences on the settings tab correlation field type value and default information ipv4 public addresses plus or minus 255 values example 10 10 11 0 – 10 10 10 254 = 2 ipv4 private addresses exact match between addresses ipv6 public addresses within 65535 values domains, urls, file names, and email addresses has a threshold of 90% similarity, calculated via “lengthwise levenshtein” of less than 1 character substitution per 10 characters of string length list elements lists of raw text strings as well as ipv4 public, ipv6 public, domain, url, email, md5, sha1, sha256, ssdeep, filename list values file attachments fuzzy hash similarity greater than or equal to 90% configure record correlations to configure record correlations in your application, complete the following steps begin by creating an application select the plus icon , and then select create a new application on the create application screen, complete the name field you can select the general , administration , records , or workspace tabs to provide more information, or use the next and prev buttons to navigate through tabs select +create a new application the application opens in the application builder the form layout section will be empty in the form layout section, add the following field types text , ip , url , and correlation tip to access the ip and url field types, select the arrow in the bottom right corner of the text field type the form will now include all required field types for correlation at this point, record correlation is enabled the field named correlation results will display matched results select manage correlation settings this opens the record correlation window for information about how correlation works and default matching behavior for certain field types, select the documentation tab after reviewing the information, return to the settings tab configure the following matching preferences matching timeframe specify how far back in time (in days) the system should check for matches matching threshold enter the percentage of similarity required to determine a match from the correlation field list, select a field to use for matching from the expected value type list, select the value type (for example, domain, email, or ip) to apply filters that refine the results further, select a field from the filters section you can define up to two filters in the correlation action section, choose the playbook to run when a correlation is found (optional) select the only trigger playbook if correlated records have been found checkbox to limit execution to matched cases only if not selected, the playbook will always run select either a classic or canvas playbook classic triggers the selected playbook as usual canvas choose a flow from the select a flow list, or select +create new to define one select apply to save your correlation settings once you apply these correlation settings and save your application, correlation will begin immediately however, correlation fields can be edited later as needed attachments not linked in correlation in turbine on prem in turbine platform (tp), the attachment correlation functionality is not working as expected when correlation is configured for attachments in an application, the attachments are not being linked or correlated across records steps connect to the postgresql pod kubectl exec it postgresql 0 bash switch to the postgres os user su postgres connect to the postgresql database psql execute the following sql commands to create the tlsh comparison function create or replace function public tlsh compare(text, text) returns integer as '/usr/lib/postgresql/14/lib/tlsh psql so', 'pg tlsh compare' language c; verify the function is available by listing functions in the public schema select from pg extension; select routine name as function name from information schema routines where routine type = 'function' and routine schema = 'public';