Skip to content

Governed Inference Doctrine

Inference is a model-portable kernel capability in service of personal AI continuity. Providers are implementation surfaces behind role policy; they do not own cognition, routing, authority, approval, execution, memory, identity, or source-of-truth status.

The purpose of model portability is not provider abstraction for its own sake. It keeps Luna's reasoning style, operator relationship, and continuity from being trapped inside one model, vendor, or local runtime.

Canonical Model Registry And Truthful Inventory

The governed inference registry separates a user-facing canonical model ID from provider identity, backend/runtime identity, wire protocol, and the backend-native model identifier. OpenAI-compatible describes a protocol; it does not mean the provider is OpenAI. A local llama.cpp server can use that protocol while remaining a local llama.cpp target. Ollama, llama.cpp, OpenAI, and xAI/Grok therefore retain distinct provider and backend identities.

Registry composition adapts the deployed LUNA_FAST_MODEL, LUNA_MID_MODEL, LUNA_HEAVY_MODEL, and LUNA_MODEL_MODE compatibility settings without changing routing behavior. The current local-tier service path is repository-evidenced as Ollama, so its exact backend identifiers and fast, mid, or heavy assignments are preserved. A reference whose provider binding is unknown remains an explicit unresolved configuration fact rather than being guessed into Ollama or another provider. Provider type existence, outbound-adapter availability, and complete configuration are separate facts. xAI has a configured outbound integration. The provider-neutral authenticated OpenAI-compatible adapter can also represent OpenAI when an outbound API key, base URL, and exact model are configured, but the current deployment has no outbound OpenAI target. LUNA_OPENAI_API_KEY protects AncientOS's inbound API; it is not evidence of outbound OpenAI configuration.

The read-only inventory reports configuration, discovery, reachability, and readiness independently. Registration does not establish discovery. Endpoint reachability does not establish model presence. Discovery does not establish that inference can initialize or complete. Only fresh, bounded readiness evidence can report ready; missing or stale evidence reports unknown. A reachable backend without the configured model is not_discovered and unavailable. A discovered model whose bounded inference attempt fails is degraded.

Qwen 3.8 illustrates the distinction. Its canonical llama.cpp target is qwen38-obliterated, with a local llama.cpp backend using the OpenAI-compatible protocol and a configured context of 4096. Its GGUF may be listed by Ollama under the separately configured runtime target qwen38-obliterated-ollama while failing initialization there, so that observation is discovered and degraded rather than ready. Keeping the runtime targets distinct avoids confusing a backend-native identifier with canonical identity.

luna-inference observes configured targets through two small HTTP adapters. The Ollama adapter uses /api/tags for discovery and /api/chat for a synthetic one-token readiness probe. The provider-neutral OpenAI-compatible adapter uses configurable models and chat paths plus optional bearer authentication, so the same contract serves local llama.cpp and configured frontier APIs without changing provider identity. Both adapters bound connection/read time, response size, failure reasons, and generated tokens. Readiness probes use fixed synthetic content and never send user or session content.

The root Compose deployment maps host.docker.internal to Docker's host-gateway for luna-inference. QWEN38_LLAMA_BASE_URL therefore defaults to http://host.docker.internal:11500; no observed bridge address is persisted in configuration or model identity, and llama.cpp continues to run as the host-side qwen38-llama.service with Vulkan access.

The read-only GET /v1/inference/models/inventory endpoint returns inference-model-inventory-v1. It includes independent configured, discovered, reachable, and readiness state plus evidence observation and expiry timestamps. Observations are cached only until their configured TTL; stale evidence fails closed to unknown. force_refresh=true refreshes local evidence. Paid remote targets are not probed by an ordinary inventory read: they reuse fresh cached evidence, are verified one target at a time during selection, or may be explicitly included by the authenticated diagnostic combination force_refresh=true&include_remote=true. The endpoint uses the service's existing optional API-key guard. The Oracle projection consumes this same bounded payload as observation-only operational state and does not probe, select, approve, or execute.

Provider-filtered questions request evidence only for that provider, so a Grok/OpenAI question does not refresh unrelated local runtimes. General inventory reads retain their truthful live-observation behavior.

Current runtime model status is a separate read-only contract. Rubick supplies only desired posture. The Luna runtime reports its normalized process mode, while luna-inference reports its own process mode and a process-local, allowlisted observation of the most recent successful inference. That observation contains only timestamp, mode, provider, provider path, and backend model identifier; it never contains prompts, responses, headers, endpoints, credentials, or raw environment data. The transport-neutral runtime reconciles the sources and reports mismatch, stale, unreachable, or unknown state without choosing a winning source. Provider/model activity is claimed only from fresh successful-inference evidence. Service restart clears the recent observation.

The authenticated GET /v1/inference/runtime-status endpoint performs no provider probe and uses runtime-model-status-v1. Runtime model-status questions are rendered deterministically, so “most recent successful inference” is kept distinct from the model serving the status answer itself.

Registry inventory and evidence contracts are read-only. They do not select an active model, change defaults, add fallback, introduce health-based routing, or grant Rubick, Lich, Zeus, executor, mutation, or external-contact authority. Unification with the older static provider-role readiness surface remains deferred.

Read-Only Model Questions

The RuntimeKernel exposes one transport-neutral application path for explicit model inventory questions. Discord, Web TUI, terminal, and future text transports submit the same RuntimeMessageEnvelope; transports do not parse model names or own model state. The application retrieves the canonical luna-inference inventory and does not call Ollama, llama.cpp, OpenAI, or xAI directly.

The supported read-only operations are bounded inventory listing, ready/local/ frontier filtering, exact provider-target filtering, individual-target inspection, provider inspection, and current inference posture. Explicit questions about current model mode, local model, or frontier-versus-local inference preempt Rubick snapshot and generic deployment/execution routes and use the canonical runtime-status contract. Conversational output includes at most twelve model rows. Structured results retain canonical rows and explicitly report that the operation is read-only, has no mutation authority, and changed no selection.

Availability language follows evidence:

  • ready means current bounded inference evidence established usability;
  • degraded means the target was discovered but its readiness probe failed;
  • unavailable means current evidence established that the target cannot be used, including a reachable backend where the configured model is absent;
  • unknown means readiness was not observed or the prior evidence is stale;
  • configured or discovered alone is never rendered as available.

Individual inspection uses the registry's exact normalized aliases. Qwen 38, Qwen3.8, Qwen 3.8, and qwen38 resolve to the canonical llama.cpp target qwen38-obliterated. Its separate Ollama observation target remains visible as discovered but degraded when that is the current evidence. Ambiguous aliases fail closed with their canonical candidates; unknown names are not fuzzily substituted. Explicit provider shorthands are limited to Grok/xAI, OpenAI, Ollama, and llama.cpp. A provider descriptor or configuration does not imply a ready canonical model. In the current deployment OpenAI remains a known but unconfigured provider type, while configured xAI readiness is reported only after current per-model evidence exists.

“Which model are you using?” and “What model are you using right now?” report the request or session model actually bound to the turn when runtime provenance exists, including its inventory backend when that row is present. Without that provenance they fail closed instead of guessing from configured routing. Configured xAI/Grok targets remain visible on frontier-inventory questions and are still rendered with evidence-bound readiness, never as usable from configuration alone.

Intent matching is deliberately narrow and passes through the canonical intent boundary. Generic software/data-model questions, hardware model numbers, code, quoted examples, implementation discussions, and switch/use-for-this requests do not select the read-only capability. Provider-role readiness unification remains deferred. Model switching, request overrides, session selection, and previous-selection restoration are implemented through the separate canonical selection service described below. Persistent defaults remain a separate governed configuration mutation rather than an inventory action.

Current operating posture is local-first for coding and implementation lanes, with frontier escalation through governance for interactive conversation and eligible inference roles. Frontier models are infrastructure, not identity. Rubick preserves posture continuity across model and provider changes by making prompt/profile posture, cognitive modes, model mode, provider path, and runtime abstraction inspectable and governable. Meepo protects transition continuity when model/provider changes are part of an approved state transition; it revalidates the transition instead of selecting or judging models itself.

Role Registry

Canonical inference roles are defined in app/inference/roles.py and mirrored for tooling in docs/machine/inference-roles.json.

Current canonical roles:

  • conversational_reply
  • oracle_synthesis
  • operational_narration
  • architecture_summary
  • coordination_analysis
  • route_explanation
  • draft_generation
  • bounded_reasoning
  • meeting_synthesis
  • memory_fact_extraction
  • retrieval_query_embedding
  • architecture_evidence_embedding

Compatibility roles remain available for existing callers: operational_clarification, task_intent_classification, embedding_generation, and memory_fact_embedding.

Each role declares allowed providers, local/frontier eligibility, structured-output eligibility, cost class, fallback policy, replay metadata requirements, forbidden uses, and degradation behavior.

Provider Arbitration

app/inference/arbitration.py selects providers deterministically from role policy, provider readiness, model mode, local-first preference, structured-output requirements, cost policy, fallback eligibility, and degradation state.

The default provider set is:

  • local Ollama through luna-inference
  • OpenAI-compatible frontier provider, disabled unless governed metadata allows it
  • Anthropic-compatible frontier provider, disabled unless governed metadata allows it
  • disabled provider state for fail-closed surfaces

No live credential is required for readiness inspection or tests.

Underlord Escalation

Paid frontier inference is blocked unless explicit Underlord approval metadata or a bounded no-approval budget exists for the role cost class. Silent frontier fallback is forbidden. Open-ended spend is forbidden.

Replay And Observability

Provider arbitration emits deterministic lineage records with role, selected provider, fallback chain, policy decision, cost class, approval requirement, replay hash, degraded state, structured-output flag, advisory-only flag, and model usage lineage.

Lineage records do not store prompts, outputs, credentials, API keys, bearer tokens, or secrets.

Oracle consumes build_inference_readiness_surface() to answer which roles are available, which providers are degraded, whether frontier inference is allowed, which roles require Underlord approval, why providers were selected or denied, and what fallback path exists.

Migrated Runtime Surfaces

app/inference/client.py now performs provider arbitration before any luna-inference completion or embedding gateway call. Failed arbitration returns explicit degraded or fail-closed output and records lineage in the response audit metadata.

app/bot.py migrates the first live chat model-call surfaces by tagging legacy route labels with governed role intent:

  • Oracle questions use oracle_synthesis.
  • Operational readiness, provider, blocked, degraded, and status narration uses operational_narration.
  • Route questions that ask why or how routing happened use route_explanation.
  • Coordination questions use coordination_analysis.
  • Architecture, subsystem, topology, coupling, and component-boundary questions use architecture_summary.
  • Meeting Intelligence uses meeting_synthesis for structured, evidence-grounded advisory cognition over bounded ordered transcript context.

The legacy dictionary contract of run_model_attempt() remains intact while the request payload sent to luna-inference carries the selected governed role. Oracle readiness reports expose these migrated role statuses and lineage hashes without making a model call.

Remaining Legacy Surfaces

The compatibility wrappers call_openwebui(), call_ollama(), and generic conversation fallback still exist for older callers. They route through governed inference but intentionally remain on broad compatibility roles until their callers can provide narrower intent safely.

Non-model HTTP calls to Keeper, luna-tools, Zeus, luna-router, governance interceptor, ops, and Naga publication adapters are outside the governed inference role migration. They remain governed by their existing provider or capability contracts rather than by model inference roles.

Safety Boundaries

Inference output remains advisory-only. It cannot approve execution, mutate state, bypass governance, override deterministic routing, expand provider authority, silently become source of truth, or transfer runtime authority.

Explicit Model Selection (Slice 5A)

The canonical selection application supports two non-persistent scopes. A request override is immutable context for exactly one conversational inference operation. A session selection is stored against AncientOS's normalized conversation key in the existing runtime SQLite store. Effective target precedence is request override, then session selection, then the unchanged fast/mid/heavy and model-mode routing posture.

Selection accepts canonical IDs or exact registered aliases only. Fresh READY evidence is reused. UNKNOWN or stale evidence triggers a bounded observation of only the requested target; current DEGRADED or UNAVAILABLE evidence fails closed. Selection never refreshes the complete inventory. The restoration history is bounded to two prior states and includes the implicit configured posture as an explicit sentinel. An unavailable or removed prior target is not replaced with another model.

Exact selected inference uses the canonical target endpoint and never enters the legacy provider or tier fallback chain. Invocation failure is returned as a bounded failure and does not clear session selection. The selection boundary applies only to the runtime's user conversational reply lane. Classifiers, routing, planning, operational cognition, governed subsystem inference, and embedding lanes continue using their existing governed roles and arbitration.

Selection changes cognition only. It does not modify Rubick capabilities, Lich approval, Zeus evidence, executor or mutation permission, external-contact policy, tool permission, or pending governed actions. Persistent/global default mutation remains deferred to separately governed work. OpenAI-compatible protocol continues to be provider-neutral; the Qwen 3.8 canonical target uses local llama.cpp, not the separately observed and degraded Ollama representation.

Natural-Language Model Selection (Slice 5B)

All transports reach one shared model-selection intent stage after read-only model inquiry and before general conversational routing. The stage recognizes only narrow affirmative directives. switch to, let's switch, from now on, and for this session select session scope. for this, for this one, for this answer, just this time, answer this with, and try this using select request-only scope. Request overrides are passed as immutable context to one conversational inference and are not written to session state.

switch back and equivalent language calls the bounded Slice 5A restoration operation. reset the model and equivalent normal-routing language clears the session selection and resumes configured routing; reset is not restoration. Canonical IDs and exact registered aliases remain the only identity resolver. The intent layer does not perform fuzzy matching, provider discovery, readiness probing, or inventory refresh.

Provider shorthand is resolved by the same canonical selection gateway rather than by the intent layer. Grok/xAI, OpenAI, Ollama, and llama.cpp resolve only when one configured target is unambiguous. When several configured targets exist, exactly one fresh READY target may disambiguate the provider; otherwise selection fails closed. A sole configured target still passes through target-specific freshness and readiness verification before it can be selected. No provider default is invented.

Questions, explanations, comparisons, quoted commands, supplied examples, hypothetical or prospective language, code/config excerpts, and development planning do not select a model. Read-only questions continue through the model inquiry stage. Persistent wording such as always use or make ... default returns a bounded deferred response and is never reinterpreted as session selection.

Selection confirmation and failure results are transport-neutral. Identity and readiness failures remain fail-closed and never invoke a fallback model. Natural-language selection affects only the conversational lane established in Slice 5A and does not change Rubick, Lich, Zeus, executor, external-contact, tool, mutation, or pending-action authority. Persistent default mutation remains a separate future governed capability.

Configured frontier targets use the same request/session selection, restoration, reset, freshness, and no-fallback contracts as local targets. Authenticated readiness uses a one-token synthetic probe. Exact frontier chat uses the registered backend model ID and provider endpoint and never enters the legacy local fallback chain. Remote exact selection is available only under the existing LUNA_MODEL_MODE=frontier conversational posture, whose governed inference policy defines the bounded-low paid-use budget. This does not create new approval or egress authority. A broader per-target egress/cost policy and usage accounting remain future work.

The compatibility settings LUNA_FAST_MODEL, LUNA_MID_MODEL, LUNA_HEAVY_MODEL, LUNA_MODEL_MODE, and luna-local:preferred remain the configured-routing source when no explicit selection exists. Registry entries adapt those values; they do not replace or rewrite them. A persistent default would need an exact governed mapping across these settings plus restart and rollback evidence, so conversational intent does not edit environment files.