ModelOps & PromptOps Studio · Documentation

Architecture

ModelOps & PromptOps Studio's pipeline, its owned data, the events it emits/consumes, and what is out of scope.

ModelOps & PromptOps Studio
catalogdual-modeselectprompt+verevalmeteredpromoteownermetered · LLMdeterministic · $0
Live diagram — the catalog, versions and run log are deterministic; only the eval round-trip is metered.
catalog$0select$0$evalmeteredpromote$0COST LEVER · eval on demand
Live diagram — spend accrues only on a fixture eval; browsing the catalog and promoting a version are free.

Pipeline

Browse the model catalog → select a prompt + version → run eval (metered fixture harness) → promote/rollback the active version (owner). The catalog, versions and run log are deterministic; only the eval round-trip is metered.

Owned data + the invariant

Owns the prompt registry and model catalog (demo_eco_c8_prompt with versions + active pointer, demo_eco_c8_model_card). Reads the derived run log. The data invariant holds: promotion is owner-only — C8 rejects viewer writes — persisted canonical and setWhere-guarded, while a viewer's eval is ephemeral.

Events + metering, dual-mode

Emits prompt.promoted on a promotion; each eval emits cost.logged into the shared ledger. The eval stage is dual-mode (Cloud `claude-haiku-4-5` cost-capped/fail-closed · OSS recorded $0). The catalog itself is honestly dual-mode: one Cloud model and several OSS entries, recorded on local hardware.

Out of scope (simulated + labelled)

Eval scores come from a deterministic fixture rather than a live benchmark, and the eval call is a simulated metered round-trip over recorded behavior. No real model weights, no production prompts, no PII. In Stage-2 the same method surface maps to a real prompt store and eval job with no UI change.

Architecture · ModelOps & PromptOps Studio · Abhishek Saxena