Skip to content

PRD
Owner Agus
Status Draft, pending review
Created 2026-09-01
Updated 2026-09-03 (domain and scenarios split out, D28; v0 review: E6-E7, M1, milestones; D22-D27)
Supporting docs Architecture, Module Interfaces, Decision log, R1 Context Embedding Report, Excalidraw board a8d17080-29dd-47c2-98d7-b1fcaee9c0ab

This document states what Indra must do and why, with testable requirements. How it does it (concepts, storage model, invariants, matching design) is the architecture doc; the contracts modules build against are the interfaces doc; the rulings behind both are the decision log.

1. Problem

Helmi's products see the same people and organizations through many disconnected windows: emails, meeting recordings, calendar events, documents, notes. Nothing today can say "these five mentions are one person" with evidence attached. Every product that needs identity re-derives it locally and badly, and no correction made in one place propagates anywhere else.

Opportunity. One standalone service that owns the problem: callers push sources, Indra composes them into events, extracts partial entities with claims, normalizes the queryable claims, and (last, once real data exists) matches and resolves them into canonical entities, every merge carrying the evidence that drove it.

2. Goals and success metrics

Goal Metric Target
One identity across windows Given real mentions of one person across two or more source types, Indra returns one entity with links to every mention Passes on real Helmi org data
Evidence attached, mistakes reversible Every member link records its driving claim; a wrong merge is undone by superseding links, with no data loss 100% of links; an undo touches links only
Human review stays minimal Human workload per resolved entity at fixed precision: queued review decisions plus wrong-merge corrections Target parameterized from the M1 report
Matching designed on data, not guesses The extraction metrics (requirement M1) exist as a report before any matching code Report produced from a real ingested corpus
Context-rich corpus from day one Sources and events carry versioned situating summaries 100% of ingested sources and events

The two costs in the review-workload metric are not equal. A queued case, prepared with its context bundle, costs one accept/deny decision. A wrong merge that escapes to production costs an investigation: find it, read the context, hunt the correct match, fix it. So precision outweighs queue-minimization: when in doubt, queue (D10).

Progress is measured by milestone completion (section 8), never by time.

3. Non-goals and deferred work

Non-goals, permanent scope boundaries:

  • Product UX. Products own UX; Indra owns truth, behind an API. Dev-only inspection UI (seeing the data and pipeline state during development and testing) is in scope and expected.

Deferred beyond v1, wanted but not now:

  • Soft-claim processing, claim truthfulness, profile generation. Soft claims are stored and otherwise untouched until an entity reaches canonical.
  • Semi-hard (dynamic) claim types. Adds complexity, not required for v1.
  • Event context grouping (relating a meeting to the email thread after it): v2, do not build against it.
  • Right-to-erasure. v1 cannot erase a person; accepted risk (D14).

4. Users and context

Callers are Helmi's products; keshi is the first integration. The service is API-first and multi-tenant: every row is scoped by organization_id from the first migration.

The audience of this PRD is the implementing team: two engineers building v1, with Benve on forward research. The work is decomposed into five modules (ingestion, extraction, matching, entity store + applier, bootstrap), each with a contract in the interfaces doc. Five modules do not mean five owners (D15): the two engineers take the first wave (ingestion and extraction), and the remaining modules are unassigned until their block starts. The contract rule holds regardless of who builds what: a module is built against the contracts, never against another module's implementation.

5. Proposed solution

A pipeline of three stages. Ingestion: sources compose into events, and events are summarized and parsed. Extraction: an LLM extracts partial entities carrying claims, and hard claims are normalized into an indexed projection. Matching: partials are resolved into entities. These are the only stages. Summarization and extraction are the pipeline's only LLM calls; everything else is deterministic and testable without a model.

The design rests on one storage idea: entities are never rows that accumulate edits, they are views over an append-only set of links to partial entities. Merging is creating a link. Unmerging is superseding one. Data is never destroyed, only composed.

The human-capability ceiling. Indra auto-resolves only where a human could disambiguate the reference from the same context. That sets the design focus: capture the context a human would use (the situating summaries, D5) and put it in front of the model, so the model's resolution reasoning approximates the human's. What still cannot be resolved after the deterministic, heuristic, and model layers queues for human review, and the queue must stay small (goal table): a review backlog that outruns its users kills adoption of the whole service.

Matching and resolution are deliberately built last, against the distribution of data extraction actually produces. Match confidence bands and the critical-mass model are parameterized from measured data (M1), never hand-tuned in advance. The vocabulary and the ten invariants are in the domain doc; the pipeline and the directional matching design are in the architecture doc; worked cases are in scenarios.

6. Requirements

P0 blocks v1. P1 ships after the P0 core works, parameterized by M1's numbers.

Ingestion and composition (module 1)

# Requirement Pri Acceptance criterion
I1 Source ingestion API with external_id/external_system/content_hash dedup; identical re-ingest is a no-op, changed content mints a new row P0 The same payload ingested twice yields one source row; a changed payload yields a second
I2 Sources compose into events P0 A meeting recording plus its calendar event yield one event holding both sources
I3 Source and event summaries generated eagerly, keyed by id plus summarizer_version, small model via OpenRouter P0 Every source and event has a summary row; re-running under a bumped version adds rows and replaces none
I4 A changed source (same external_id/external_system, new content) recomposes into the SAME event and forces re-extraction; reconciliation is diff-based: unchanged output is kept, changed output supersedes the stale claims and links P0 A rescheduled calendar event yields one event, re-extracted; the removed attendee's stale claims are superseded and no duplicate entity members appear

The driving case for I4 is calendars: events mutate constantly (reschedules, attendee changes), and corrected emails and edited documents follow the same path. Claims from a superseded source version stop contributing to critical mass (interacts with S4). The reconciliation mechanics, including how they relate to extractor_version re-runs (E1), are owed by the architecture doc (D11).

Extraction and normalization (module 2)

# Requirement Pri Acceptance criterion
E1 LLM extraction of partial entities with claims from events, versioned and re-runnable P0 Re-extracting an event under a new extractor_version leaves prior output intact
E2 Every claim is typed against the shared registry, carrying both classification dimensions (queryability with identifying class; temporality with cardinality) P0 A claim with an unregistered type is rejected at the contract boundary
E3 A partial entity carries at least one hard claim P0 Extractor output violating this is rejected, not stored
E4 Hard claims normalize into an indexed, org-scoped projection P0 A normalized email query returns its partial entity; a cross-org query returns nothing
E5 Relation claims: the extractor emits claims with target_partial_id referencing a partial of the same event, typed against the relation-type registry with per-type attributes (design/relationships.md, D16-D19) P0 An event stating an employment yields a typed relation claim between the two partials; an unregistered relation type or a cross-event target is rejected at the contract boundary
E6 Identifying claims are grounded: the value appears literally in the event's source text, modulo the claim type's canonicalization (D23) P0 An extractor output containing an identifying claim whose value is absent from the source text is rejected, not stored
E7 Partial entities are typed: kind from the registry (v0: person, organization), with identifying class keyed by (claim type, entity kind) where it differs (D22) P0 A domain claim resolves as identifying on an organization partial and non-identifying on a person partial; an unregistered kind is rejected at the contract boundary

Measurement (deliverable, not overhead)

# Requirement Pri Acceptance criterion
M1 Extraction metrics report: hard-claim yield per source type, share of partials carrying an identifying claim, alias collision rate, candidate-set size per claim type, relation-claim yield and precision per relation type (including other label distribution, D18/D21), alignment stability under re-extraction, identifying-claim error rate per source type (D25), and review-driver analysis (which claim patterns produce ambiguity a human would have to resolve) P0 Report computed from a real ingested corpus against the labeled eval set (D25); matching design review and the review-workload target (goal table) are set against it

Entity store and resolution (modules 3, 4, 5; designed directionally, built last)

# Requirement Pri Acceptance criterion
S1 Entity store on the EntityLink model: member/evidence kinds, supersede instead of delete, driving_claim_id on every link P1 A superseded link remains queryable; no code path deletes a link
S2 Matching with tier routing: identifying exact hit merges, non-identifying hit attaches evidence, no hit creates an unresolved entity P1 Each of the three routes produces its link kind or entity, per the architecture design
S3 Per-org FIFO applier P1 Two concurrent resolutions naming the same unknown person yield one unresolved entity, not two
S4 Promotion at both levels: evidence to member on critical mass; unresolved to canonical only on clear no-match plus identifying claim from a trusted source or critical mass P1 Promotion transactions are atomic; a failed search never enables promotion
S5 Resolution judge in shadow mode until it clears an eval dataset P1 Verdicts are logged with model and prompt version, not applied, until the gate clears
S6 Bootstrap import of canonical entities from authoritative systems, through the applier P1 Imported entities carry member links with driving claims, like resolved ones

Human review and auditability

# Requirement Pri Acceptance criterion
R1 Cases no resolution layer can decide (including judge inconclusive verdicts) land in a review queue exposed via API; each case carries the context bundle needed to decide it P1 A queued case is decidable without opening source systems; deciding it is one accept/deny API action
R2 Every mutating action (merge, link create/supersede, promotion, bootstrap import, review verdict) is recorded in an action log with actor, timestamp, and inputs; humans and the system write the same log, distinguished by actor P1 Any entity's current state is explainable from its action history alone
R3 Resolution actions (merge, link create/supersede, promotion, review verdict) expose one-step undo; an undo is itself a logged action. Ingestion and bootstrap import are re-runnable, not undoable P1 Undoing a merge restores prior query results, appends an action, and deletes nothing

Observability and cost (cross-cutting)

Tool choices (PostHog, Langfuse, BetterStack from the start) are a stack decision (D9); this section states only what must be observable and why.

# Requirement Pri Acceptance criterion
O1 Pipeline traceability: a source is traceable end to end through ingestion, extraction, and matching, with stage-level failure visibility P0 Given a source id, an operator answers "where did it stop and why" without reading code
O2 Every LLM call records prompt version, model, tokens, cost, and latency; extraction and judge calls are linkable to their eval datasets. Traces carry real org data: tracing destinations sit inside the production data boundary P0 A month of extraction spend is attributable per stage and per prompt version; S5 shadow verdicts are auditable
O3 Goal metrics emitted continuously, not only in the M1 report: review rate, merge rate, undo rate (the production precision proxy), unresolved backlog P1 Each metric queryable over time; regressions visible without a manual report
C1 Per-stage LLM cost metrics with caps and alerts P0 A runaway stage triggers an alert before it exhausts budget
C2 All LLM calls go through one interface that hides both the provider (OpenRouter or a specific vendor) and the call mode; the interface is batch-shaped, so sequential calls (testing) and batch calls (production) are a configuration switch, not a code change P0 Swapping provider or call mode changes no caller
C3 Batch-mode processing for summarization and extraction in production P1 No full-corpus run starts on sequential calls

Backup, security, and access

# Requirement Pri Acceptance criterion
B1 Backups with RPO 24h and RTO of half a working day. Sources are re-ingestable from external systems, but derived LLM output costs real money to re-create: the database is the asset P0 A restore drill has actually run before v1 is declared done
X1 Per-product service credentials; every call org-scoped P0 No credential can query across organizations
X2 PII posture stated and enforced: Indra is Helmi's most PII-dense service by design (emails, names, employment, relationships, cross-referenced). Encryption at rest; access logging on entity and claim reads P0 Reads of entity data are attributable to a caller

Erasure is explicitly not supported in v1 (section 3, D14).

7. Milestones

These are system gates, not work assignments: each marks a point where one of the three stages of section 5 joins the one before it and something end to end becomes true (D15). Modules carry their own internal milestones, owned with the module; this section tracks only the joins. Sequential, each done when its criterion holds, no time estimates.

The first wave is the ingestion and extraction stages. The entity store, applier, bootstrap, and matching are deferred as one block: matching needs entities in the graph to match against, and its thresholds are parameterized from M1. Observability and cost requirements (O1, O2, C1, C2) land with the gate that introduces their stage, not at the end.

  1. Contracts ratified. The interfaces doc is agreed, the registry package exists at v0, and the modules of the first wave have owners. Done when a module can be started from the interfaces doc alone.
  2. Corpus ingested and summarized (I1-I3). Done when a real corpus of emails plus calendar events is ingested with source and event summaries, behind the C2 LLM interface.
  3. Corpus extracted (E1-E7, and I4 at the claim level). Done when partial entities from that corpus are queryable by hard claim, a changed-source re-ingest reconciles claims correctly, and the M1 eval-set plan (size, sampling, who labels) exists before the measured corpus is extracted (D25). Link supersession is out of scope here: links do not exist yet.
  4. Measurement (M1). Done when the report exists, the matching design has been reviewed against it, and the review-workload target is set from it.
  5. Matching and resolution (S1-S6, R1-R3, O3), parameterized by M1's numbers. Internal order: entity store and applier, then bootstrap import (organizations first, D26), then matching. Done when the goal-table identity test passes on real Helmi org data and I4 is re-verified with links present.

Gate 5 holds nine requirements across three modules, far more than any other. It is decomposed into its own milestones when the block starts, against M1's numbers; splitting it earlier would mean guessing the parameters that decomposition depends on.

8. Decisions

Decisions live in the decision log: append-only, one entry per decision, each carrying a status line that records supersession and why. Do not re-litigate a decision without naming its entry and saying what changed.

9. Appendix