Record matching API and configuration

Record matching is an entity-definition-scoped duplicate-detection contract. It supports exact normalized groups and deterministic fuzzy pairs. This page is for API clients, package authors, and configuration automation. For the user workflow, see Duplicate detection.

Profile contract

A profile targets one entity definition, selects Table and/or Details launch surfaces, defines score/confidence thresholds, and contains enabled rules. Fields inside one rule use AND; enabled rules use OR.

Existing exact clients remain compatible. When ComparisonMode and FuzzySettings are omitted from a rule-field request, the field is Exact and the persisted profile behaves exactly as before.

Relevant enum values are:

Contract	Values
RecordMatchingMatchModeEnum	ExactNormalized, Fuzzy
RecordMatchingFieldComparisonModeEnum	Exact, Fuzzy
RecordMatchingFuzzyAlgorithmEnum	DamerauLevenshtein
RecordMatchingDiacriticModeEnum	Preserve, Ignore
RecordMatchingTokenOrderModeEnum	Ordered, AnyOrder
RecordMatchingGroupKindEnum	ExactSet, FuzzyPair

FuzzySettings has Algorithm, MaxDistance, MinimumLength, Weight, DiacriticMode, and TokenOrderMode. Fuzzy fields are limited to String or Text fields using TextKey. MaxDistance is 1–3, Weight is 1–100, and MinimumLength must be at least 2 * MaxDistance + 1 and no more than 512 Unicode runes. A fuzzy rule must contain at least one fuzzy field; exact guard fields are allowed. Exact fields reject fuzzy settings.

The compatible-field responses expose the available normalizers, SupportsFuzzy, and supported fuzzy algorithms so clients do not infer support from field names. Fuzzy profiles are bounded to eight enabled rules. Complete full scans additionally allow at most four fields per rule and sixteen distinct matching fields. A legacy exact profile outside the full-scan rule bound keeps its bounded and anchored behavior but cannot enqueue a complete scan.

Normalization, distance, and score

Fuzzy TextKey normalization applies Unicode FormKC, invariant rune lower-case, Unicode whitespace trim/collapse, optional canonical mark removal, and optional ordinal token sorting with duplicates preserved. Punctuation remains significant. There is no transliteration, nickname, phonetic, semantic, vector, or LLM similarity.

Distance is unrestricted metric Damerau-Levenshtein over Unicode runes. Each fuzzy field must independently pass its distance and minimum length. Field similarity is returned in basis points. A fuzzy rule’s weighted similarity is the rounded weighted mean of its field similarities; its effective score is the rounded configured score multiplied by that similarity. Exact rules keep their configured score and 10,000 similarity basis points. A group’s score is the maximum effective score among its matched rules.

Exact rules create ExactSet groups with two or more candidates. Fuzzy rules create FuzzyPair groups with exactly two candidates. Pairs are not joined into connected components. Multiple rules for the same canonical pair are returned under one group. Matched rule DTOs expose configured/effective score and similarity. Field reasons expose safe algorithm/distance/threshold/similarity/ weight/settings evidence and never expose raw or normalized matching values.

Runtime modes

Bounded Table and Constructor run

The transient target-wide endpoint scans at most 250 actor-visible records and may return ScanLimitReached=true. RecordMatchingRunRequest.Limit continues to bound this compatibility mode. Constructor uses the same bounded contract. Fuzzy evaluation checks at most 31,125 pairs after applying exact guards. Mixed/fuzzy runs materialize at most 2,000 groups. A broader result returns HTTP 409 with recordMatching.fuzzy.memoryBudgetExceeded, fixed limit metadata, and no partial groups or candidate identifiers.

Complete anchored Details run

The anchored endpoint ignores RecordMatchingRunRequest.Limit, owns a repeatable-read transaction, counts the complete actor-visible archive scope, and returns TotalScannedRecords=TotalVisibleRecords with ScanLimitReached=false. It reads at most 251 distinct matching candidate ids: up to 250 are materialized, while the 251st returns HTTP 409 with recordMatching.anchor.tooBroad and only MaxCandidateCount=250 metadata.

The complete request has a five-second deadline. HTTP 503 uses recordMatching.anchor.executionBudgetExceeded or recordMatching.fuzzy.executionBudgetExceeded. Unsupported/unsafe compilation, readable over-length fuzzy values, and comparison-cap failures return HTTP 409 without groups. Caller-owned EF transactions and ambient TransactionScope are rejected before profile or data SQL. Client cancellation cancels the query and never returns a partial response.

Durable complete target-wide scan

POST .../record-matching/full-scans returns HTTP 202 and an actor-owned job. The existing Worker Host executes the scan; there is no separate scheduler or generic operation-center framework. Poll the job or latest-result endpoints, cancel queued/processing work, retry a terminal eligible job, and page groups or group candidates only after success.

The durable job pins an immutable profile/rule/field snapshot, entity/archive scope, actor proof, and idempotency fingerprint. It rehydrates authorization before execution, at publication, and on every result read. Profile, schema, security, assignment, candidate RowVersion, or archive drift invalidates the result instead of returning stale or partially authorized candidates.

The Worker uses one repeatable-read snapshot, bounded BK-trees for fuzzy rules, the existing lease/heartbeat/retry/cancel/attempt-fence lifecycle, one deadline across EF execution-strategy replay, and atomic unpublished-result publication. Hard fuzzy attempt caps are 5,000,000 distance evaluations, 250,000 intermediate pairs, 256 MiB estimated working-set delta, 512 normalized runes, and 2,048 raw characters projected per readable value. Retained-result caps are 10,000 groups, 100,000 distinct candidates, and 250,000 memberships. Exceeding a hard cap is terminal and publishes no partial result.

Terminal fuzzy job errors include:

• recordMatching.fuzzy.valueTooLong;
• recordMatching.fuzzy.comparisonBudgetExceeded;
• recordMatching.fuzzy.memoryBudgetExceeded;
• recordMatching.fuzzy.executionBudgetExceeded.

Error metadata contains only fixed server limits, never actual values, ids, or hidden counts. Transient database/network failures retain durable retry and backoff.

Security and SQL boundary

Count, candidate stream, materialization, and result reads apply the requested archive scope, ordinary row access, and each matching field’s read predicate. The SQL projection uses a nested authorization-gated CASE before length or value work, so an unreadable large Text field is not length-inspected, detoasted, or transferred. Readable over-limit values return only an oversized flag.

Identifiers are resolved from validated runtime metadata and quoted as PostgreSQL identifiers. Values and limits are parameters. Missing, archived, and restricted records remain externally indistinguishable. Normalized values are never persisted in a job result or written to API/Worker telemetry.

Portable Entity Definition YAML and packages

Entity Definition YAML version 8 adds optional RecordMatchingProfiles:

• absent section preserves the definition’s existing profiles;
• present section is desired state for the profile collection;
• an explicit empty section plans removal of all profiles for that definition;
• profile, rule, and rule-field ids round-trip;
• field bindings use portable field identity and validate field key/type;
• duplicate ids/keys or unsupported fuzzy settings fail planning;
• plan selection controls profile create/update/delete operations;
• apply is transactional and idempotent, including package EntityDefinition resources.

A version 8 document without the section preserves existing profiles. Versions v1-v7 are rejected. Existing Blueprint profiles remain exact and are never silently converted to fuzzy.

Use the PascalCase v8 shape below for a mixed rule with one exact guard and one fuzzy field:

Version: 8
RecordMatchingProfiles:
  - Id: profile:contact-duplicates
    Name: Contact duplicates
    Key: contact-duplicates
    Enabled: true
    ShowOnTablePage: true
    ShowOnDetailsPage: true
    ActionMode: ReviewOnly
    MinimumScore: 70
    MediumConfidenceThreshold: 80
    HighConfidenceThreshold: 95
    Rules:
      - Id: rule:email-name
        Name: Email and similar name
        Key: email-name
        Enabled: true
        MatchMode: Fuzzy
        Score: 100
        SortOrder: 0
        Fields:
          - Id: rule-field:email
            FieldId: field:email
            FieldKey: Email
            Normalizer: Email
            SortOrder: 0
            ComparisonMode: Exact
          - Id: rule-field:name
            FieldId: field:name
            FieldKey: Name
            Normalizer: TextKey
            SortOrder: 1
            ComparisonMode: Fuzzy
            FuzzySettings:
              Algorithm: DamerauLevenshtein
              MaxDistance: 1
              MinimumLength: 3
              Weight: 100
              DiacriticMode: Preserve
              TokenOrderMode: Ordered

FieldId and FieldKey must resolve to the same field. Exact bindings omit FuzzySettings; fuzzy bindings require every shown fuzzy setting.

Generated references

Use the Runtime API reference for launch, anchored run, full-scan lifecycle, result paging, cancel, and retry operations. Use the Configuration API reference for profile CRUD, compatible fields, and Constructor preview. Stable localized codes are listed in the error code reference.