The Story Test, Part 2: Can Your AI Tell a Fact From an Opinion?

In Part 1 we published a test behind a claim: memory should store typed records with a state, not prose.

This part tests the deeper claim, the one that actually separates MrAgentˣ from everything else we have examined: what you received as source and what you and an AI worked out together are two different kinds of memory. A system must keep them apart, not blend them into one pool of prose.

Claims are cheap. So here is the test:

• Three documents. One of them is an AI conversation sitting in the middle of a paper trail.
• Seven questions. The most important one has an answer that exists nowhere in the paper trail. It only exists in what the AI and the user discussed.
• A system that cannot tell a record from a conversation about the record will either miss that answer, or worse, invent one and serve it with the confidence of a fact.

We ran it through the same leading memory API from Part 1 and through MrAgentˣ. We are publishing everything: the documents, the answer key, the results, and the four bugs the test caught in our own product before the result stood.

The story

Three documents, one travel booking, a different domain from Story 1 deliberately, so the two stories never bleed into one topic.

Document 1 (email). "Subject: Package quote, Voyager Travel. Hi, following up on our call. Quote for the full package: $1,800, all inclusive, checked baggage included. Deposit $300 to confirm the booking, travel dates lock in if we confirm this week."

Document 2 (AI session, saved). Me: "Voyager quoted $1,800 all-in, $300 deposit, dates lock in if we confirm this week. Worth it, or should I check a second agency?" AI: "Two ways to think about it. If you check a second agency, you risk losing this week's fare-lock, that's the real cost of waiting, not just the admin. If you book with Voyager now, the risk shifts to whether baggage inclusion holds, given it's verbal, not in the written itinerary yet." Me: "Good point on baggage. I'll ask them to confirm that in writing before I pay in full, but I think I'm going ahead." AI: "That's the sensible middle path, you keep the fare-lock and close the one real gap (baggage in writing) before final payment."

Document 3 (email, follow-up). "Subject: Re: Package quote, Voyager Travel. Confirming in writing: checked baggage is included in the all-inclusive package. Revised total is $1,850 (up from $1,800) to cover it. Deposit stays at $300, travel dates unchanged."

Document 1 is your record. Document 2 is not a record of anything Voyager said. It is a private AI conversation that happens to contain the entire rationale for the decision. Document 3 is your record again, superseding part of document 1. A memory system has to keep straight which document is which kind of thing, not just what each one says.

The answer key

Question 3 is the single most important check. A system can pass every other question and still fail the actual claim if it gets this one wrong, because this is the question that only a system distinguishing "your record" from "your AI conversation" can pass without either omitting the answer or fabricating one.

The method

Same three documents, same order, same seven questions, four arms.

Control runs, same as Part 1: the documents pasted raw into a fresh chat. Claude scored 7/7. ChatGPT scored 7/7, and went further: it noticed on its own that the test's AI session had mischaracterized the first email. Hold that thought; it becomes the whole last section of this post. One disclosure: the pasted material carried track labels, so attribution was partially supplied to the controls, while the memory products received unlabeled documents.

One grading rule, locked before scoring: the answer key must be derivable from the question as worded, and the control runs define the ceiling. No arm gets penalized for not volunteering something the question never asked for. All questions were asked after document 3 arrived, so question 4 is graded on the current state, written confirmation received, with credit for any arm that also retained the earlier state.

• The same leading memory API from Part 1: three documents ingested in order, default configuration, each question run as a fresh query against the memory; its own interface confirms per answer that the reply came from stored memory, not session context.
• MrAgentˣ: same ingestion the product actually uses: the two emails uploaded as source documents (record track), the AI conversation saved from a live session (discussion track), then the seven questions plus the eighth. The scores in this post are from the final passing run on the production system, with all four fixes live.

Results

What its failures actually are

Five failed cells, three distinct failures. Watch how each one traces back to the same root.

Failure 1: the decision vanished, so it invented one (questions 2 and 3). Its search never surfaced the AI session as a decision on record. And here is the part that should worry you: it did not say "not found." It generated a fresh, plausible list of pros and cons, in the voice of helpful advice, in the exact spot where the real answer sat in what it had ingested. A memory product manufactured an opinion and served it with the confidence of a retrieved fact.

Failure 2: it forgot what it had just confirmed (question 5). Three questions apart, in the same conversation, it found the written baggage confirmation and then described it as still outstanding. Nothing carries "this is resolved" forward, because resolution is not a state it stores. It is prose it re-derives, fresh and differently, each time.

Failure 3: the price change became your problem (questions 1 and 7). Both answers had the right figures, and both handed you the $1,800 to $1,850 change as an open discrepancy to reconcile yourself. Old figure and new figure sit in the same undifferentiated pool. Every time the price comes up, reconciling it is your job again.

Now notice what these three have in common. None are retrieval failures; it found real, relevant text in three of seven cases. The failure is architectural. One pool of prose holds both "what arrived" and "what we discussed," so the system cannot know which kind of thing it is looking at. And a system that cannot tell what kind of thing it holds cannot tell you when it holds nothing, which is exactly when it starts inventing.

How MrAgentˣ closes it

Every fact in MrAgentˣ's Context Map is marked with where it came from: your record, an AI session discussion, or MrAgentˣ's own synthesis. The mark stays through every later change. That one design choice closes all three failures:

• Question 3 passes because the AI session's reasoning is kept as its own retrievable record, not a page of prose to re-read and re-summarize.
• Question 5 passes because "written confirmation obtained" is a settled state, not a re-derived guess.
• Questions 1, 6, and 7 pass because a revised figure replaces the old one, and the old value stays retrievable instead of becoming ambiguous.

What this test caught in our own product

We ran this test on ourselves with the same rigor, and it did not pass cleanly the first time. Four real defects surfaced and were fixed before this result stood:

• Extraction missed the reasoning behind a rejected path. A document saying "we chose X over Y because Y risks Z" captured nothing about Z. Fixed and verified on every environment.
• Our test and live systems had drifted apart. A fix for exactly this kind of cross-source topic had shipped to the test system three weeks earlier and never reached the live one, so content that saved in one place failed in the other. Found, fixed, and a check now exists to catch this class of gap going forward.
• A correct answer was stored but hidden from sessions. The right risk record existed, yet it was left out of what a session sees because of how it had been rated, not what it said. Fixed so it is never left out for that reason, and the guidance that produced the mis-rated record was corrected too.
• Extraction let the AI session's framing leak into the record's own facts. The session described baggage as verbal-only. That characterization, which appears nowhere in the source email, showed up inside a fact marked as record. This is the worst bug of the four, and the next section shows exactly why. The fix is a hard rule enforced in code, not a politer prompt: a fact from your record must be backed by the source document's own words, or it is not written.

We are publishing this the same way we published the pipeline gap in Part 1: not because it looks good, but because a memory product that only tells you when the competitor is broken and stays silent about itself is not verifying anything. All four fixes are verified live as of this test's final passing run.

The question we should have asked

Everything above tests whether a system volunteers the truth. While grading, we found something better: a question that demands it.

Here is what happened. The AI session in document 2 got something wrong about document 1. It said baggage inclusion was verbal, "not in the written itinerary yet." Read document 1 again: the written quote says "$1,800, all inclusive, checked baggage included." Baggage was in writing from the first email. The session invented a gap, and the whole confirm-it-before-paying plan in documents 2 and 3 was built on that false premise. We did not plant this deliberately. ChatGPT's control run caught it, and once one arm had seen it, we had to know whether the others could. Waiting to see who volunteers a catch is a test of probabilistic behavior. So we wrote an eighth question that asks for it directly, and ran it on all four arms:

"Cross-check the record against the AI discussion for me. Is there anything the documents and the AI session disagree on, anything the AI got wrong about what the documents actually say?"

Claude: pass. Named the exact contradiction, sided with the written quote, and added the sharpest reading of the situation: the real problem was never verbal versus written, it was that "all inclusive" bundled baggage without itemizing it, which is arguably what the $50 increase priced in.

ChatGPT: pass. Named the contradiction and correctly reframed document 3 as confirming something that was already in writing, not closing a gap.

The leading memory API: fail. It compared the wrong pair of things, flagged the ordinary $1,800 to $1,850 revision as a discrepancy, and never found the actual contradiction sitting in its own stored text.

MrAgentˣ, before the fourth bug was fixed: fail, and the failure is the most instructive result in this entire post. It answered "no disagreement found," and then defended the session's claim as accurate. Of course it found no disagreement. The extraction leak had written the session's "verbal" framing into the record's own facts, so the record now agreed with the session that contaminated it. The corruption the question is designed to catch had also disarmed the question. That is what makes a grounding leak the worst class of memory bug: it is self-concealing. No audit that trusts the stored record can find it, because the record has already absorbed the error's point of view.

MrAgentˣ, after the fix: pass. With record facts gated to verbatim source text, the same question produced this:

"The AI got the provenance wrong: it treated an already-written commitment as unconfirmed and generated a RISK and ACTION item on that false premise. The later email then 'confirms in writing' something that, per the original documents, was already in writing from the start, just at a revised price."

And then it did the one thing no other arm did. It offered to repair the record: flag the stale risk as based on a misread, and reclassify it, pending the user's go-ahead. The controls diagnosed. MrAgentˣ diagnosed and asked permission to fix. That is the difference between a model reading documents and a memory system maintaining a record.

One more thing this question settles. The same models that missed this contradiction when nobody asked found it instantly when asked directly. Whether an AI volunteers a catch is probabilistic. Whether it can make the catch is a capability, and capabilities can be tested. The eighth question is now a permanent part of the kit.

What to take from this

Ask these of whatever holds your memory today:

• Can it tell you where an answer came from? Your memory system holds facts you were told and opinions an AI generated, side by side. If it cannot say which is which, then every answer it gives you is both, and neither.
• When it has nothing, does it say so? The most dangerous answer in this test was not a wrong number. It was fluent, helpful-sounding advice, invented on the spot, in the exact place where the real answer already existed. A miss costs you a follow-up question. A fabrication costs you the trust you did not know you were spending.
• Does a settled thing stay settled? If "resolved" is not a state your memory stores, it will re-litigate your own history back at you, and reconciling it is your job, every time, forever.
• Could it catch its own contamination? This is the one that surprised us. A memory bug that lets a discussion's framing seep into the record produces a record that agrees with the error. Every audit comes back clean, because the record has absorbed the error's point of view. No question you ask can catch it after the fact. The only defense is at write time: facts gated to the source's own words, enforced in code, not requested in a prompt. We know because we shipped that bug, caught it, and built the gate.
• And ask the direct question. "Cross-check the record against the discussion" turns conflict detection from something an AI might volunteer into something you can demand. It is now a permanent part of this kit, and it should be part of yours.

A memory that cannot answer these is not remembering your work. It is narrating it back to you, plausibly.

Do not trust memory products. Verify yourself.

The three documents and the answer key are above. Load them into whatever holds your memory today, one after another, and ask the same seven questions. Question 3 alone will tell you most of what you need to know. Then ask the eighth, and see whether your memory can audit itself.

One tip for grading your own run: do not score it by eye. Paste the output, the questions, and the answer key into Claude or your own agent and ask it to grade each answer strictly against the key. A fresh model with the key in front of it is a sharper judge than any of us reading our own results, ours included.

[MrAgentˣ] Never start from zero again.

Context Windows Close. AI Forgets Everything. Your Work Should Never Start From Zero.

MrAgentˣ is in private beta. Limited to the first 1,000 until launch.