eval team · 2026-07-11
A saved rule is retrieved automatically by a new session and followed without a human repeating it.
First-party dogfooding evidence — see how we verify these sessions ↗
Timeline
- 02:42
A manual re-fire uses the wrong model provider. The team investigates how the drift got into the run.
- 15:49
The corrected procedure is saved to MemNexus as a standing rule for manual re-fires.
- 22:27
A fresh session, 6.5 hours later, retrieves the rule on its own at startup. No human repeats the instruction in this session.
- Next run
The following scored run completes 120 of 120 calls with zero human corrections.
Evidence
- Incident transcript: 365fdadb-237b-4c46-8ce2-d0a997363d60.jsonl (line 1134). The full transcript is retained internally. The incident message above is paraphrased at the product owner's direction and does not appear verbatim anywhere on this page.
- Retrieval transcript: session a26e639e… startup (line 29, timestamp 22:27:35Z). The retrieval was a tool call the agent issued on its own; no human message in this session conveyed the rule.
- Outcome: the following run's scorecard records 120/120 scored calls completed and zero human corrections.
What this does not prove
That the rule alone caused the clean run — the session also had other context. The demonstrated mechanism is: rule saved once → retrieved automatically at next session start → no human repetition → subsequent run clean.