Knowledge tool reference

Overview

The knowledge tool is one tool with actions. Instead of a separate tool for search, reading, listing and computing, the model gets one name, search_knowledge_base, and picks what it does with the action argument. One description carries the decision rules once, and discover tells the model what exists before it guesses.

There is one implementation behind every door. engine.search_knowledge_base, call_knowledge_tool and the MCP tool served by create_mcp_app all call the same function, so an answer cannot differ between them.

Calling it

From your own code, call the engine method. principals works as it does everywhere in the engine: a list of the caller's principals, [] for anonymous, or TRUSTED for a trusted in-process job. principals=None is deprecated: it means trusted, warns, and a later release refuses it. scope has no default and must be given.

from context_engine import ContextEngine, TRUSTED

caller = ["group:hr"]          # from your authenticated session; [] is anonymous

answer = await engine.search_knowledge_base(
    action="search",
    query="annual leave",
    principals=caller,
    scope=["hr"],              # required: the host's ceiling of source ids
)
if not answer["success"]:
    print(answer["error"])     # relay it to the model; it is written for one

import { ContextEngine, TRUSTED } from "@promptev/context-engine";

const caller = ["group:hr"]; // from your authenticated session; [] is anonymous

const answer = await engine.searchKnowledgeBase({
  action: "search",
  query: "annual leave",
  principals: caller,
  scope: ["hr"], // required: the host's ceiling of source ids
});
if (!answer.success) console.log(answer.error); // relay it to the model

In your own agent loop

knowledge_tool_definition() (knowledgeToolDefinition()) returns the tool as data: name, description, input_schema as JSON Schema with required: ["action"], and annotations. Hand that to your model, then run what it asks for with call_knowledge_tool (callKnowledgeTool). readOnlyHint is false on purpose: most actions only read, but compute runs generated code.

call_knowledge_tool takes the engine, then keyword arguments: action, principals (already resolved: it never invents a caller), scope, the model arguments listed below, and the host-only redaction, secret_key, compute and map_reduce. The TypeScript form takes one object with the same snake_case keys.

from context_engine import call_knowledge_tool, knowledge_tool_definition

tool = knowledge_tool_definition()
# {"name": "search_knowledge_base", "description": "...",
#  "input_schema": {"type": "object", "properties": {...}, "required": ["action"]},
#  "annotations": {"readOnlyHint": False, "openWorldHint": False}}

# Your agent loop: hand "tool" to your model, then run what it asked for.
# Take ONLY the model's arguments from the model; principals and scope are yours.
result = await call_knowledge_tool(
    engine,
    **model_args,                 # action, query, source_ids, ...
    principals=caller,            # already resolved, never defaulted here
    scope=["hr", "finance"],
)

import { callKnowledgeTool, knowledgeToolDefinition } from "@promptev/context-engine";

const tool = knowledgeToolDefinition();
// { name: "search_knowledge_base", description, input_schema, annotations }

// Take ONLY the model's arguments from the model; principals and scope are yours.
const result = await callKnowledgeTool(engine, {
  ...modelArgs, // action, query, source_ids, ... (snake_case)
  principals: caller,
  scope: ["hr", "finance"],
});

Arguments

Every argument the model may set. The exported schema and the MCP tool are both built from this one property table, so they cannot disagree. Only action is required by the schema; each action checks its own required arguments.

A search hit's document_id is a system id. Content identifiers written inside documents, such as invoice numbers, belong in query, never in document_id.

The 13 actions

The order below is the order of KNOWLEDGE_ACTIONS. Examples show the engine call; the model sends the same arguments as JSON. Response shapes are abridged: ... marks elided content, and // comments are explanations, not part of the payload.

discover

Needs: Always available

When to use it. First, whenever the agent does not already know what is in scope. One call returns the documents, what is inside each one and which actions this deployment can run, so later calls can use real sheet, column, section and field names.

answer = await engine.search_knowledge_base(
    action="discover",
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "discover",
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "documents": [
    {"id": "...", "name": "costs.xlsx", "source_id": "finance",
     "kind": "spreadsheet",            // "spreadsheet" or "text", decided by MIME
     "document_type": "...", "mode": "hybrid", "failure_reason": null,
     "structure": {...}}               // sheets / sections + last_page / keys / chunks
  ],
  "count": 50, "has_more": true,
  "next_cursor": "...", "next_page": {"action": "discover", "cursor": "..."},
  "document_types": [...],             // census of the whole scope
  "fields_by_type": {...},             // extracted field names, types, counts
  "available_actions": {"search": "...", "compute": "... (OFF)", ...},
  "next_action": {"for a clause or a fact": {"action": "search", "query": "..."}, ...}
}

search

Needs: Always available

When to use it. Questions answered by reading text. To look up an ID, code, SKU or invoice number, send the bare identifier alone (for example "2525"), not the whole question.

answer = await engine.search_knowledge_base(
    action="search",
    query="annual leave carry over",
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "search",
  query: "annual leave carry over",
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "hits": [
    {"document_id": "...", "document_name": "HR leave policy", "chunk_text": "...",
     "score": 0.83, "source_id": "hr", "chunk_idx": 4}
  ],
  "usage": {...},
  "hint": {"action": "compute", "reason": "aggregate question over a spreadsheet"}  // only on a mixed scope
}

get_doc

Needs: Always available

When to use it. Reading one whole document by its system id, taken from discover, list or a search hit. Never pass an invoice number or other content identifier here.

answer = await engine.search_knowledge_base(
    action="get_doc",
    document_id="3f2b8c1e-5a4d-4e8f-9b61-2c7d0a9e4f10",
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "get_doc",
  document_id: "3f2b8c1e-5a4d-4e8f-9b61-2c7d0a9e4f10",
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "document": {
    "id": "...", "name": "...", "text": "...", "acl": [...], "status": "completed",
    ...,
    "failure_reason": null,      // one FAILURE_REASONS code, or null
    "failure_message": null      // the fixed sentence for that code
  }
}

get_docs

Needs: Always available

When to use it. Several whole documents at once, by id.

answer = await engine.search_knowledge_base(
    action="get_docs",
    document_ids=["3f2b8c1e-5a4d-4e8f-9b61-2c7d0a9e4f10", "8d1e7f02-6b3c-4a59-8e2d-1f0c9b7a6e35"],
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "get_docs",
  document_ids: ["3f2b8c1e-5a4d-4e8f-9b61-2c7d0a9e4f10", "8d1e7f02-6b3c-4a59-8e2d-1f0c9b7a6e35"],
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "documents": [{...}, {...}],   // same per-document shape as get_doc
  "count": 2,
  "remaining_document_ids": ["..."],                               // only when the budget ran out
  "next_page": {"action": "get_docs", "document_ids": ["..."]}
}

get_chunks

Needs: Always available

When to use it. Walking one long document in order, a range at a time, or reading around a search hit by starting at its chunk_idx.

answer = await engine.search_knowledge_base(
    action="get_chunks",
    document_id="3f2b8c1e-5a4d-4e8f-9b61-2c7d0a9e4f10",
    start=4,
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "get_chunks",
  document_id: "3f2b8c1e-5a4d-4e8f-9b61-2c7d0a9e4f10",
  start: 4,
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "document_id": "...", "total_chunks": 120, "start": 4, "end": 28,
  "chunks": [{"position": 4, "text": "...", "language": "en"}, ...],
  "has_more": true, "next_start": 29,
  "next_page": {"action": "get_chunks", "document_id": "...", "start": 29}
}

list

Needs: Always available

When to use it. Browsing documents without searching. Cheaper than discover: it does not fetch structure, fields or the census.

answer = await engine.search_knowledge_base(
    action="list",
    source_ids=["hr"],
    limit=20,
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "list",
  source_ids: ["hr"],
  limit: 20,
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "documents": [{"id": "...", "name": "...", "source_id": "hr", "kind": "text",
                 "document_type": null, "mode": "hybrid", "failure_reason": null,
                 "status": "completed", "parked": null}],
  "count": 20, "has_more": true,
  "next_cursor": "...", "next_page": {"action": "list", "cursor": "..."}
}

query_meta

Needs: An LLM configured

When to use it. The structured fields a question names (dates, amounts, parties) and each document’s value for them. It does not filter or compare by value: read the values and compare them yourself.

answer = await engine.search_knowledge_base(
    action="query_meta",
    query="invoice amount and invoice date",
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "query_meta",
  query: "invoice amount and invoice date",
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "question": "...",
  "resolved_fields": [...],
  "documents": [{"id": "...", "source_id": "...", "name": "...",
                 "document_type": "...", "structured_data": {...}, ...}],
  "count": 3
}

compute

Needs: An LLM and enable_code_execution, or a host compute runner

When to use it. Any figure derived from spreadsheets (total, average, count, ranking, margin, comparison) and finding the exact row that matches one id or value. It runs over every row of the sheets in scope, not a sample.

answer = await engine.search_knowledge_base(
    action="compute",
    query="total amount by region in costs.xlsx",
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "compute",
  query: "total amount by region in costs.xlsx",
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,              // whether the generated code ran
  "result": ...,
  "code": "...", "stdout": "", "error": null,
  "execution_time": 0.41,       // executionTime in TypeScript
  "documents_used": ["..."],    // documentsUsed in TypeScript
  "provider_tokens": {"llm_input": 812, "llm_output": 164},  // providerTokens in TypeScript
  "attempts": 1                 // 1 or 2 code-generation calls
}

map_reduce

Needs: An LLM configured, or a host map_reduce runner

When to use it. Asking the same question of every document in scope and getting one answer per document, for example "which contracts mention X", where search would return a handful of passages and miss the rest. Not for figures: that is compute.

answer = await engine.search_knowledge_base(
    action="map_reduce",
    query="Does this contract have an auto-renewal clause?",
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "map_reduce",
  query: "Does this contract have an auto-renewal clause?",
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "results": [
    {"document_id": "...", "document_name": "...", "data": {...}},
    {"document_id": "...", "document_name": "...", "error": "..."}
  ],
  "processed": 24, "failed": 1, "considered": 25,
  "hint": {"action": "compute", "reason": "..."}   // only when some targets are spreadsheets
}

get_neighbors

Needs: graph.enabled

When to use it. What is one step from a named thing, and which way each link points.

answer = await engine.search_knowledge_base(
    action="get_neighbors",
    entity="Acme Corp",
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "get_neighbors",
  entity: "Acme Corp",
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "entity": {"id": "...", "name": "acme corp", "type": "ORG"},
  "neighbors": [{"id": "...", "name": "...", "type": "PERSON", "category": "MEMBERSHIP",
                 "label": "works_for", "evidence": "...", "direction": "incoming"}],
  "count": 7
}

traverse

Needs: graph.enabled

When to use it. Everything within a few steps of a named thing.

answer = await engine.search_knowledge_base(
    action="traverse",
    entity="Acme Corp",
    depth=2,
    category="HIERARCHICAL",
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "traverse",
  entity: "Acme Corp",
  depth: 2,
  category: "HIERARCHICAL",
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "entity": {"id": "...", "name": "...", "type": "ORG"},
  "paths": [{"target_id": "...", "target": "...", "target_type": "PERSON",
             "category": "HIERARCHICAL", "label": "reports_to", "evidence": "...", "hops": 2}],
  "count": 12
}

find_related

Needs: graph.enabled

When to use it. Connections of a kind across the corpus, with no starting entity (an entity is refused; for how two named things connect, traverse from one). Also the way in when every name in an answer is masked: take a source_id or target_id from its result.

answer = await engine.search_knowledge_base(
    action="find_related",
    category="CREATION",
    entity_type="PRODUCT",
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "find_related",
  category: "CREATION",
  entity_type: "PRODUCT",
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "relationships": [{"source_id": "...", "source": "...", "source_type": "ORG",
                     "target_id": "...", "target": "...", "target_type": "PRODUCT",
                     "category": "CREATION", "label": "makes", "evidence": "..."}],
  "count": 9
}

community_summary

Needs: graph.enabled and graph.community_summaries

When to use it. The themes each source groups into, ranked against a question.

answer = await engine.search_knowledge_base(
    action="community_summary",
    query="supplier risk",
    principals=caller,
    scope=["hr", "finance"],
)

const answer = await engine.searchKnowledgeBase({
  action: "community_summary",
  query: "supplier risk",
  principals: caller,
  scope: ["hr", "finance"],
});

{
  "success": true,
  "communities": [{"summary": "...", "entity_count": 14, "relationship_count": 22,
                   "hierarchy_level": 0, "relevance": 0.7121}],
  "count": 3
}

Refusals and off actions

Which actions can run is decided per call, not when the tool is built, so an engine whose config resolves per request stays correct:

An off action is advertised as off by discover and, if called anyway, returns a fixed sentence. The other common refusals:

// Aggregate question over a scope that is all spreadsheets (search, query_meta)
{"success": false,
 "error": "these documents are spreadsheets; searching their rows would answer from a sample. Use compute over the whole sheet.",
 "next_action": {"action": "compute", "query": "total spend by region",
                 "suggestion": {"frame_key": "...", "measure": "amount", "group_by": "region"}}}

// An action this deployment cannot run
{"success": false, "error": "compute is off: code execution is not enabled on this deployment. Quote the rows you found instead."}

// document_ids sent to an action that does not honour it
{"success": false, "error": "document_ids is not honored by get_doc ..."}

In the aggregate refusal, next_action.query is the question handed back so the redirect is a call the model can make. suggestion is vocabulary, not code: frame_key is the name compute will give the newest in-scope sheet, measure its first numeric column, group_by its first text column. Any of them can be null. A scope that mixes prose and spreadsheets is still searched and only gets an additive hint.

Answers may also carry next_page or next_action: the next call, already filled in. The tool description tells the model to follow it rather than guess.

Scope and ceilings

scope is the ceiling the host sets: the documents this tool may ever reach. It uses only nouns the engine already knows, source ids and optionally document ids within them. Map your own idea onto it: a project to its source ids, a matter to its source ids, an agent to the documents it was handed.

from context_engine import Scope, UNSCOPED

scope = ["hr", "finance"]                         # a list: these source ids
scope = Scope(source_ids=("hr",),                 # sources, and specific documents in them
              document_ids=("3f2b8c1e-5a4d-4e8f-9b61-2c7d0a9e4f10",))
scope = UNSCOPED                                  # the whole corpus, on purpose
scope = None                                      # raises ValueError: scope is required

from context_engine.knowledge_tool import narrow_to_ceiling, resolve_scope
narrow_to_ceiling(["hr", "legal"], ("hr", "finance"))   # -> ["hr"]   ("legal" dropped)
narrow_to_ceiling(["legal"], ("hr", "finance"))         # -> []       (nothing in scope)
narrow_to_ceiling(None, ("hr", "finance"))              # -> ["hr", "finance"]
narrow_to_ceiling(None, None)                           # -> None     (no filter at all)

import { narrowToCeiling, resolveScope, UNSCOPED, type ScopeInput } from "@promptev/context-engine";

let scope: ScopeInput = ["hr", "finance"];        // an array: these source ids
scope = { sourceIds: ["hr"], documentIds: ["3f2b8c1e-5a4d-4e8f-9b61-2c7d0a9e4f10"] };
scope = UNSCOPED;                                 // the whole corpus, on purpose
resolveScope(null);                               // throws: scope is required

narrowToCeiling(["hr", "legal"], ["hr", "finance"]); // -> ["hr"]
narrowToCeiling(["legal"], ["hr", "finance"]);       // -> []
narrowToCeiling(null, ["hr", "finance"]);            // -> ["hr", "finance"]
narrowToCeiling(null, null);                         // -> null (no filter at all)

Who may name documents

A document-id ceiling is a whitelist. get_doc and get_chunks check it and the source ceiling too, and a document outside either reads exactly like one that does not exist. discover and list drop listed documents outside a document-id ceiling. Within the ceiling, the caller's principals still apply: the ACL is enforced inside the SQL of every read.

Only six actions honour document_ids (search, compute, get_docs, map_reduce, discover, list). The others refuse it rather than answer over the whole scope as if it had been honoured.

resolve_scope and narrow_to_ceiling live in context_engine.knowledge_tool; only Scope is exported at the Python package root. TypeScript exports resolveScope and narrowToCeiling at the root.

Redaction and host runners

redaction overrides the deployment's configured policy for one call, with the same contract engine.search has. Without it, a host with a per-project or per-customer policy would get masked text from every other read path and unmasked text from this one. secret_key is the other half: the key hash rules use for this call. A per-tenant policy hashed with the engine-wide key gives every tenant the same token for the same value, so tokens would join across tenants. Omitted, the engine's own policy and key apply.

Every action applies them: document text, chunks, hits, computed results and graph answers.

Replacing compute and map_reduce

A host can pass its own compute and map_reduce callables. They replace the built-in ones, which is where a host applies its own rules for permission, billing and approval, and they make the action available whatever the deployment settings say. A runner is never handed the redaction policy or the hash key. What it returns is masked on the engine side, the whole value including any documents_used. What the runner's own model saw is the host's responsibility.

async def my_compute(instruction, *, source_ids, document_ids, principals) -> dict:
    ...   # your sandbox, your billing, your approval

async def my_map_reduce(instruction, *, source_ids, document_ids, principals, limit) -> dict:
    ...

app.mount("/mcp", create_mcp_app(engine, principals=principals, scope=scope,
                                 compute=my_compute, map_reduce=my_map_reduce))

# or per call, without MCP:
await engine.search_knowledge_base(action="compute", query="...", principals=caller,
                                   scope=["finance"], compute=my_compute)

const myCompute = async (
  instruction: string,
  opts: { sourceIds: string[] | null; documentIds: string[] | null; principals: unknown },
) => ({ /* your sandbox, your billing, your approval */ });

const myMapReduce = async (instruction: string, opts: Record<string, unknown>) =>
  ({ /* opts: sourceIds, documentIds, principals, limit */ });

const handler = await createMcpApp(engine, {
  principals, scope, compute: myCompute, mapReduce: myMapReduce,
});

The spreadsheet refusal for map_reduce runs before a host runner is called. When the engine resolves the documents a run would read, the runner receives those ids; if that set is empty, it receives the caller's original document_ids instead of an empty list.

Serving over MCP

create_mcp_app (createMcpApp) serves the knowledge tool over MCP streamable HTTP, with no stdio. It needs the [mcp] extra in Python and the @modelcontextprotocol/server and @modelcontextprotocol/node peers in TypeScript. Beside search_knowledge_base it always registers the connector-tool pair search_tools and execute_tool. The served description adds one sentence saying results are scoped to the caller's permissions.

from fastapi import FastAPI
from context_engine import create_mcp_app

app = FastAPI()

def principals():            # zero-argument, sync or async, called on EVERY tool call
    return current_user_groups()           # [] for anonymous; never None

def scope():                 # same contract; or pass a plain value
    return sources_for_current_project()   # e.g. ["hr", "finance"]

app.mount("/mcp", create_mcp_app(
    engine,
    principals=principals,   # required, keyword-only
    scope=scope,             # required in practice: None fails every call
    redaction=policy_for_current_tenant,   # optional: a policy, or a callable returning one
    secret_key=key_for_current_tenant,     # optional: hash key for this call
))

import { createServer } from "node:http";
import { createMcpApp } from "@promptev/context-engine";

// createMcpApp is async: await it. It returns a (req, res) handler.
const handler = await createMcpApp(engine, {
  principals: () => currentUserGroups(),       // required; [] is anonymous
  scope: () => sourcesForCurrentProject(),     // required; throws TypeError if missing
  redaction: () => policyForCurrentTenant(),   // optional, per call
  secretKey: () => keyForCurrentTenant(),      // optional, per call
});

createServer((req, res) => {
  void handler(req, res);
}).listen(8080);

The context-engine mcp command serves one database this way with anonymous principals and UNSCOPED. See the CLI reference. For HTTP routes see the HTTP API reference.

Python and TypeScript parity

Both packages run the same engine over the same Postgres schema; a corpus ingested by one is searchable by the other. Names follow each language: snake_case in Python, camelCase in TypeScript. Wire payloads the model or an MCP host reads (knowledge tool results, failure_reason) are snake_case in both. The tables list what each package exports from its root; a name marked for one side often exists on the other in a submodule, which the table says.

Engine methods

Exported by both roots

Only at the TypeScript root

Only in Python, or different by design

Configuration fields are covered field by field in the configuration reference; full signatures are in the Python API and TypeScript API pages.

Further reading

Cookbooks

Each package ships a runnable multi-tenant RAG recipe: cookbook/multi_tenant_rag.py and js/cookbook/multi_tenant_rag.ts. Three documents share one corpus: an HR policy with acl=["group:hr"], an engineering runbook with acl=["group:eng"], and a holidays page with no ACL. One question is asked four ways, and the recipe asserts the answers: the HR employee sees holidays and the HR policy, the engineer sees holidays and the runbook, an anonymous visitor sees only holidays, and a trusted job sees all three. Each recipe reads DATABASE_URL and embedding settings from the environment, runs migrations and writes documents, so point it at a scratch database. Get the packages from PyPI and npm.

The access-control model

Thresholds follow the query's shape

The trigram leg's similarity threshold moves with the query, from 0.45 for a very short query down to 0.15 for a long one, measured in characters for spaceless scripts (Chinese, Japanese, Thai, Khmer, Burmese, Lao, Tibetan) and in words otherwise. A matching minimum cosine similarity for the vector leg exists but is off by default: search.vector_floor="adaptive" turns it on (0.45 for one or two words, 0.40 for a code, an acronym or an exact-match query, down to 0.25 for a sentence). With it on, the vector leg can return fewer candidates and nothing backfills them. Measure on your own embedding model before enabling it. search.trgm_limit pins the trigram threshold to one value.

Keyword search settings

The full-text leg uses Postgres text search over the simple configuration, which keeps every word. By default (search.lexical_match="all") a chunk must contain every word of the query, stopwords included: precise for keyword queries, but a question such as “what is the notice period for contractors” usually matches nothing. "any" matches a chunk containing any of the query’s words after dropping stopwords (search.lexical_stopwords, the built-in English list by default) and leaves the order to the rank function. Quoted phrases and -term keep their meaning in both modes. For "notice period" contractor -draft, "all" matches chunks with the exact phrase AND the word contractor and without the word draft; "any" matches chunks with the phrase OR the word contractor, still without draft. The simple configuration does not stem, so -draft does not exclude “drafts”. The query is split into words, phrases and terms on ASCII whitespace. search.lexical_rank picks ts_rank_cd (words close together, the default), ts_rank (frequency anywhere) or the experimental, opt-in bm25 (Okapi BM25 in SQL, turned on per database with context-engine bm25-enable, not a default; with search.bm25.k1 1.2, search.bm25.b 0.75 and search.bm25.idf_scope "source", "global" or "document"; its statistics include documents the caller may not see, though no content or id is exposed), and search.lexical_normalization passes Postgres’s 0 to 63 length-normalization bitmask through. Query text is always bound as data, and every variant applies the same scope predicate as the other legs. A leg weighted 0 in fusion.weights is switched off and its query is not sent: the vector leg also skips the query embedding unless compression needs it, and the graph leg is never started, billed as hybrid, and reported as graph_weight_zero. The default weights are 1.0 for the full-text, vector and graph legs and 0.4 for the trigram leg, which the retrieval benchmark measured ahead of 0.8 on four of five public datasets. search.trgm_max_query_words skips the trigram leg for queries longer than that many words, and search.lexical_candidates sets how many candidates the full-text and trigram legs hand to fusion; both are off by default, and measured on five public datasets neither improved ranking everywhere. Fusion uses reciprocal rank fusion with fusion.k 20. The TypeScript names are lexicalMatch, lexicalRank, bm25.idfScope, lexicalNormalization, lexicalStopwords, trgmMaxQueryWords and lexicalCandidates.

Why a document failed

A failed document carries error, the raw exception text for operators, and failure_reason, one fixed code, with failure_message as its fixed sentence. Model-facing doors (this tool's get_doc, get_docs and list, and the routers' document read for non-trusted callers) never return error. A later success clears both.

An unrecognised code reads as unknown rather than as no failure.

The ingest usage event

hooks.on_usage is the metering seam; the package never handles billing itself. Every UsageEvent(kind="ingest") carries mode (the mode the document was actually processed at) and mode_reason (why that differs from the request, currently only "tabular documents stay hybrid", else None) in its detail, on every path that emits one. A DOCX is counted by the page count the file reports, with pages_source saying where it came from: app_xml, rendered_breaks, page_breaks or estimate. A declared count is only believed as far as the body could hold it.