NEXUS operates the serving fabric behind large language and vision models: multi-region batching, attention-level caching, deterministic evaluation and continuous drift control.
▊
A control plane for every model you ship
From weights to wire: routing, quantisation policy, evaluation gates and rollback, driven by one declarative spec.
Adaptive inference routing
Requests are matched to the cheapest healthy replica by latency, cache warmth and quantisation compatibility — with automatic failover inside 40 ms.
scheduler/v4Continuous evaluation
Regression suites run on every checkpoint. Statistical gates block promotion when quality, latency or safety metrics drift outside tolerance.
eval/offline+shadowAttention-level caching
Prefix, KV and graph-level caches are shared across tenants with cryptographic partitioning, cutting repeat-context cost by up to 7×.
cache/kv-shardPolicy & isolation
Per-tenant quotas, content policy hooks and hardware-isolated execution lanes, auditable end to end from request id to token span.
guard/policy-v2Observability
Token-level traces, fan-out timelines and cost attribution exported as open telemetry streams — no proprietary agent required.
otel/streamFederated rollout
Canary cohorts, shadow traffic and instant rollback across regions from a single declarative release manifest.
release/manifestWhat the fabric is doing right now
Aggregated, anonymised counters from the serving mesh. Refreshed continuously from edge collectors.
Built on boring, verifiable primitives
transport
gRPC + HTTP/2 with HTTP/3 for edge ingest, mTLS everywhere, zero plaintext east-west traffic.
compute
Heterogeneous accelerators behind one abstraction; quantisation profiles chosen per workload at admission time.
storage
Content-addressed weight store, immutable release manifests, reproducible builds with signed provenance.
Idea to production, gated at every step
Spec as code
A model spec, dataset pointers and evaluation policy enter version control as one reviewable artifact.
Graph & quantise
Graphs are fused, shapes are specialised and quantisation profiles are benchmarked against the evaluator.
Statistical admission
Quality, latency, cost and policy suites must pass together; failures return a diff, not a stack trace.
Shadow then shift
Traffic is mirrored, compared offline, then shifted by cohort with automated rollback on anomaly.
Continuous drift control
Production distributions are sampled against the reference set; drift opens an evaluation ticket automatically.
Talk to the systems team
Evaluation harness design, throughput economics, private deployments and migration reviews.