# SEM — Mathematical Apparatus (Capability Catalog)

*A non-internal catalog of the operators SEM offers, what each is for,
and which entry points of the `sem_cython12` library back them.*

This document describes WHAT the apparatus does and WHERE to use it.
It does not describe HOW any operator works internally — algorithms,
formulas, lemmas and proofs are intentionally not reproduced here.

---

## Conventions

- "Item" / "world" / "observation": one row of input data.  Items live
  in some payload space (real numbers, vectors, matrices, sampled
  functions, sampled manifolds, distributions, complex amplitudes,
  time-series windows, recursive concept trees) — the apparatus
  treats them uniformly via a small set of structural operators.
- "Concept": a subset of items that share structural meaning.  The
  apparatus can either be told the concepts (labelled mode) or
  discover them from data (unsupervised mode).
- "Witness": an item whose structural position carries information
  beyond merely belonging to one concept.
- "Verdict": the system's qualified output for a new observation -
  one of `confident`, `gap`, `incoherent` (see §4.6).

All of the apparatus is parameter-free and threshold-free: there are
no fitting parameters, no numeric cut-offs, no fidelity knobs.

---

## 1.  Structural similarity primitives

These are the lowest-level building blocks.  Each is exposed directly
in `sem_cython12.wrapper`.

### 1.1  Pairwise similarity

| | |
|---|---|
| Purpose | Score how close a query item is to the most similar member of a reference set. |
| Output | A score in `[0, 1]` per query (1 = at the reference set, 0 = effectively far). |
| Applications | Membership tests, retrieval, anomaly detection, k-nearest-neighbour pre-filtering, similarity-weighted aggregation. |
| Cython entry point | `batch_max_similarity(X_query, X_members, lam)` |

### 1.2  Multi-class similarity matrix

| | |
|---|---|
| Purpose | The same operation applied across `K` independent reference sets in one call, returning a `(Q, K)` score matrix. |
| Applications | Multi-class classification scoring, multi-criterion membership, class-confusion matrices, support-vector inputs to higher-level filters. |
| Cython entry point | `concept_support_matrix(X_query, member_mats, lam)` |

### 1.3  Pairwise distance matrix

| | |
|---|---|
| Purpose | Symmetric `(N, N)` distance matrix between rows of `X`. |
| Applications | Graph construction, clustering, scale estimation, downstream filtering and ranking. |
| Cython entry point | `pairwise_distances(X)` |

### 1.4  Nearest-neighbour distance vector

| | |
|---|---|
| Purpose | For each row, the minimum positive distance to any other row.  Rows with no positive-distance neighbour receive `inf`. |
| Applications | Local-density estimation, intrinsic-scale derivation, duplicate detection, outlier identification. |
| Cython entry point | `nn_distances(X)` |

---

## 2.  Multi-criterion filtering primitives

Given a real-valued matrix `S` of shape `(N, k)` (rows are items,
columns are independent criteria — each in maximisation orientation),
these primitives identify structurally informative subsets of rows.

### 2.1  Best-tradeoff filter

| | |
|---|---|
| Purpose | Mask the rows that survive a multi-objective best-tradeoff filter (i.e. items that are not strictly worse than another item on every criterion). |
| Applications | Multi-objective optimisation frontier, concept-membership trade-off, candidate winnowing before further analysis. |
| Cython entry point | `pareto_core_mask(S)` |

### 2.2  One-sided peak flagging

| | |
|---|---|
| Purpose | Flag row/column pairs where the row is the column-wise winner but contributes nothing on the remaining columns - i.e. items that "peak" on a single criterion alone. |
| Applications | Removing items that are only locally informative; finding cross-criterion contributors; bridge identification. |
| Cython entry point | `one_sided_mask(S)` |

### 2.3  Non-redundant witness identification

| | |
|---|---|
| Purpose | The subset of rows that survive both 2.1 and 2.2 — items that contribute meaningfully across multiple criteria, not just on one. |
| Applications | Bridge-witness selection between concept regions, structurally informative subset extraction, downstream gap analysis. |
| Cython entry point | `non_redundant_witnesses(S)` |

---

## 3.  Incremental aggregation primitive

### 3.1  Fused centroid + radius update

| | |
|---|---|
| Purpose | One-pass bulk update for an incremental aggregation step.  Given `F` reference items - each summarised by a centre vector and a radius (representing the dispersion of `cur_arity` underlying points) - and `A` candidate new contributions, produce all `F * A` updated (centre, radius) pairs that result from appending one candidate to one reference item. |
| Applications | Streaming centroid / radius maintenance, candidate-frontier expansion in multi-stage selection, online aggregation pipelines. |
| Cython entry point | `extend_frontier_kernel(cur_centers, cur_radii, new_emb, cur_arity)` |

---

## 4.  Higher-level apparatus

Built on the primitives in §1–§3.  These are the operators that
distinguish SEM as a reasoning system rather than a computation
library.  Their internal construction is not reproduced here; the
"Cython entry points used" column lists the public primitives the
operator composes.

### 4.1  Intrinsic scale

| | |
|---|---|
| Purpose | Derive the kernel scale from the data's own structural geometry, so that no manual `lam` value is ever required. |
| Applications | Any pipeline that wants the scale property to be a function of the data, not a tuning knob; cross-application portability. |
| Cython entry points used | `nn_distances`, `pairwise_distances` |

### 4.2  Concept discovery

| | |
|---|---|
| Purpose | Group observations into structurally coherent regions without using labels, ML training, or numeric thresholds.  Returns the concepts the data itself supports. |
| Applications | Unsupervised classification, regime identification, exploratory analysis, foundation for downstream operators. |
| Cython entry points used | `pairwise_distances`, `nn_distances`, `pareto_core_mask` |

### 4.3  Relational hypothesis generation

| | |
|---|---|
| Purpose | Enumerate candidate structural relationships between concepts (pair-wise and higher-arity) and rank them by support. |
| Applications | Discovering laws / regularities between groups, cross-concept analysis, scientific structure recovery. |
| Cython entry points used | `concept_support_matrix`, `pareto_core_mask`, `extend_frontier_kernel` |

### 4.4  Semantic gap detection

| | |
|---|---|
| Purpose | Identify positions in structural space where the data should produce a witness bridging two or more concepts but does not. |
| Applications | Detecting missing variables, hidden mediators, unobserved confounders; identifying where additional measurement would resolve ambiguity. |
| Cython entry points used | `concept_support_matrix`, `non_redundant_witnesses` |

### 4.5  Prototype construction

| | |
|---|---|
| Purpose | Predict the structural features of an item that should exist between known concepts but has not yet been observed. |
| Applications | Drug-candidate suggestion, missing-mediator prediction, "what if" scenario generation, hypothesis-driven data acquisition. |
| Cython entry points used | `batch_max_similarity`, `concept_support_matrix` |

### 4.6  Verdict-qualified inference

| | |
|---|---|
| Purpose | Decide which concept best explains a new observation, returning one of three outcomes: `confident` (a single concept dominates), `gap` (multiple concepts are equally admissible), `incoherent` (no concept admits the observation consistently). |
| Applications | Decision-support systems that must abstain when ambiguous, safety-critical classification, regime change detection, automated triage. |
| Cython entry points used | `concept_support_matrix`, `pareto_core_mask`, `batch_max_similarity` |

### 4.7  Lifecycle / dominance verification

| | |
|---|---|
| Purpose | When a real observation arrives, decide whether it confirms, displaces, or co-exists with a previously predicted prototype.  Maintains the prototype's status across its lifetime. |
| Applications | Continuous-learning pipelines, theory revision under new evidence, audit-trail-preserving inference. |
| Cython entry points used | `pareto_core_mask` |

### 4.8  Hierarchical recursion

| | |
|---|---|
| Purpose | Apply every operator above to recursive concept trees — concepts whose members are themselves concepts.  Operators bubble through the hierarchy and remain mathematically consistent at every level. |
| Applications | Taxonomies, organisational hierarchies, multi-scale analysis (chemical → biological → organism, file → folder → project, etc.). |
| Cython entry points used | the operators above, recursively |

### 4.9  Streaming kNN graph maintenance

| | |
|---|---|
| Purpose | Maintain an exact k-nearest-neighbour graph as items are added or removed one at a time, without rebuilding from scratch on each update. |
| Applications | Online time-series ingest, sliding-window analytics, sensor-stream monitoring, real-time anomaly detection. |
| Cython entry points used | `pairwise_distances`, `nn_distances` (on the contiguous buffer); `scipy.spatial.cKDTree` is used internally above 1000 items for exact O(log N) queries — no fidelity knob. |

### 4.10  Time-series streaming model

| | |
|---|---|
| Purpose | A complete reasoning model over sliding windows of a stream: state extraction, transition modelling, intrinsic-scale maintenance, and verdict-qualified prediction on novel windows.  Optionally projects high-dimensional windows to lower dimensions when configured to do so. |
| Applications | Multivariate time-series classification, regime detection, online anomaly identification, signal-quality forecasting. |
| Cython entry points used | `nn_distances` (intrinsic scale), `concept_support_matrix` (verdict), the streaming-kNN apparatus from 4.9 |

---

## 5.  Composition properties

The operators in §1–§4 compose along several axes:

- **Across payload types**: the same operator works for scalars,
  vectors, matrices, tensors, functions, manifolds, complex states,
  distributions, time-series windows.  The caller supplies the
  appropriate distance function or, equivalently, an embedding into
  Euclidean space.
- **Across hierarchy levels**: concepts can themselves be members of
  parent concepts; operators recurse through the tree (§4.8).
- **Under wrapping**: stochastic and temporal extensions can be
  layered over any base payload type.  Triple compositions like
  "hierarchy of stochastic time-series" are admissible and produce
  consistent results at every level.

---

## 6.  What the apparatus does NOT offer

Stated explicitly so users can plan around the limits:

- No probability distributions over outcomes.  Verdicts are
  structural, not Bayesian.
- No reward / objective optimisation.  The apparatus does not learn
  policies; it identifies structural relationships.
- No tuning knobs that trade fidelity for speed.  Where some
  alternatives expose `epsilon`, `top_k`, `temperature`, etc., the
  apparatus uses data-derived structural boundaries instead.
- No approximate-mode kNN (HNSW / IVF / LSH / FAISS lossy modes).
  Every kNN-related operator returns exact results.

---

## 7.  Mapping summary

| Apparatus operator | Cython entry point(s) |
|---|---|
| Pairwise similarity | `batch_max_similarity` |
| Multi-class similarity | `concept_support_matrix` |
| Pairwise distance | `pairwise_distances` |
| Nearest-neighbour distance | `nn_distances` |
| Best-tradeoff filter | `pareto_core_mask` |
| One-sided peak flag | `one_sided_mask` |
| Non-redundant witness | `non_redundant_witnesses` |
| Fused centroid + radius update | `extend_frontier_kernel` |
| Intrinsic scale | composed of `nn_distances`, `pairwise_distances` |
| Concept discovery | composed of `pairwise_distances`, `nn_distances`, `pareto_core_mask` |
| Relational hypothesis generation | composed of `concept_support_matrix`, `pareto_core_mask`, `extend_frontier_kernel` |
| Semantic gap detection | composed of `concept_support_matrix`, `non_redundant_witnesses` |
| Prototype construction | composed of `batch_max_similarity`, `concept_support_matrix` |
| Verdict-qualified inference | composed of `concept_support_matrix`, `pareto_core_mask`, `batch_max_similarity` |
| Lifecycle / dominance verification | composed of `pareto_core_mask` |
| Hierarchical recursion | every operator above, recursively |
| Streaming kNN graph | `pairwise_distances`, `nn_distances` |
| Time-series streaming model | `nn_distances`, `concept_support_matrix`, streaming kNN |

## 8.  Library availability

The Cython entry points in the right column of §7 are all in
`sem_cython12.wrapper`, distributed at
[https://git.sevana.biz/vvs/sem_cython12](https://git.sevana.biz/vvs/sem_cython12).
Higher-level apparatus (composed operators in §4) is built on those
primitives and ships in the SEM foundation package, separate from
this library.