Platform Health
Operations evidence behind the AI-SOC layer — scale triggers (§8.1), model & prompt registry, redaction precision/recall and hallucination findings (NIST AI RMF Measure).
Scale triggers
1
watch / trigger
Production models
6
8 total registered
Canary trafficShare of traffic routed to the canary advisor model. Increase only after eval set passes.
10%
altisrc-soc-mistral-7b-v3.3
Override rate (7d)
4.2%
analysts override AI verdict
Grounded citations
96.4%
AI citations that resolve
Hallucinations open
1
awaiting review
Scale triggers — Phase 0 → Phase 1 baselineOne L4 GPU + Redis Streams + vLLM is the baseline. Each metric is tied to a documented trigger from §8.1. Multi-GPU / Kafka introduced only when validated by these signals.
Auto-refresh 30s| Metric | Current | SLO / cap | Trend (8h) | Trigger condition | Status |
|---|---|---|---|---|---|
| Redis queue depth | 12 items | 100 items | Sustained > 100 → add AI worker | Healthy | |
| Case → brief p95 latency | 6.4 s | 8 s | p95 > 8s while GPU > 80% → add 2nd GPU | Watch | |
| L4 GPU utilisation | 64 % | 80 % | Sustained > 80% with rising queue | Healthy | |
| vLLM tokens / sec | 1,820 tok/s | 1,200 tok/s | Floor 1200 — alert if drops below | Healthy | |
| Worker count (active) | 3 pods | 6 pods | Auto-scale to 6 on queue pressure | Healthy | |
| DLQ depth (24h) | 1 msgs | 10 msgs | > 10 in 24h → ops page | Healthy | |
| Kafka adoption trigger | 0 % | 100 % | Replay / multi-consumer demand observed | Healthy |
Model registryEvery component that produces AI output is versioned, owned, and promotable. Canary runs in parallel with production for evaluation before cutover.
Prod Canary Shadow Retired
| Component | Version | Status | Promoted | Owner |
|---|---|---|---|---|
| AI Triage Advisor (LLM) | altisrc-soc-mistral-7b-v3.2 | Production | 2026-05-22 | Nexus AI |
| AI Triage Advisor (LLM) | altisrc-soc-mistral-7b-v3.3 | Canary | 2026-06-04 | Nexus AI |
| Risk Scorer | risk-xgb-v1.7 | Production | 2026-04-18 | AI-SOC Eng |
| Correlation Agent (rules) | corr-rules-v2.4 | Production | 2026-05-30 | AI-SOC Eng |
| MITRE Mapper | mitre-bert-v0.9 | Production | 2026-05-12 | Nexus AI |
| Redaction Engine | redact-regex-v4.1+presidio | Production | 2026-06-01 | Privacy Eng |
| RAG Index | rag-index@2026-06-04 | Production | 2026-06-04 | AI-SOC Eng |
| AI Triage Advisor (LLM) | altisrc-soc-mistral-7b-v3.1 | Retired | 2026-04-22 | Nexus AI |
Prompt registryVersioned system / user prompts. Hash + author let any AI output be traced back to the exact prompt that produced it.
- case-brief.systemv12sha256:7a3f…b22026-06-02 · K. Yamamoto
- kill-chain.userv8sha256:2dc1…812026-05-28 · K. Yamamoto
- fp-likelihood.userv5sha256:9b04…ee2026-05-19 · M. Okafor
- handoff-draft.userv4sha256:0f7a…3c2026-05-10 · M. Okafor
Redaction engine — precision 97.6% · recall 94.8%Privacy guardrail evaluated against a labelled PII test set. Recall measures missed PII (false negatives = privacy leak risk). Precision measures over-redaction.
Eval set: 1,240 · last run 2026-06-07SSN
Precision
98.4%
Recall
96.9%
Loan #
Precision
97.8%
Recall
94.2%
Account #
Precision
97.1%
Recall
93.6%
Email
Precision
96.5%
Recall
92.8%
Name
Precision
94%
Recall
89.4%
Recall < 95% on any PII class blocks promotion of a new redaction engine version.
Hallucination findings (1 open)Cases where AI output diverged from evidence. Detected by citation grounding, analyst override, or eval-set replay.
- CASE-2026-0823MediumAI cited SOP-DLP-004 §2 — section does not exist2026-06-08 08:41 IST · Citation checkOpen
- MITRE T1572 mapped where evidence points to T1567.0022026-06-07 17:12 IST · Analyst overrideAcknowledged
- CASE-2026-0779MediumConfidence 92% but only 1 weak signal — overconfident2026-06-06 20:38 IST · Eval setFixed
- CASE-2026-0744HighSuggested closure as FP while DLP marked QUARANTINED2026-06-05 13:24 IST · GuardrailFixed
Scale-out is trigger-based — Kafka and multi-GPU are introduced only when these metrics prove the need. Every AI output is traceable to a model version + prompt version + RAG index snapshot.