fix(longmemeval): judge abstention questions with the abstention rubric

[thegoodengineer]

Problem

On LongMemEval, every abstention question is graded with the wrong rubric, so a provider that correctly abstains is scored as incorrect.

LongMemEval encodes abstention in the question id, not the question type. The 30 abstention entries in longmemeval_s keep their original question_type and are marked with an _abs suffix on question_id. For those entries the answer field is an explanation of why the question is unanswerable, not an answer to match against:

{ "question_id": "0862e8bf_abs", "question_type": "single-session-user",
  "answer": "You did not mention this information. You mentioned your cat Luna but not your hamster." }
{ "question_id": "88432d0a_abs", "question_type": "multi-session",
  "answer": "The information provided is not enough. You did not mention baking egg tarts." }

Judge prompt selection in getJudgePromptForType keys purely off the question type:

if (type.includes("abstention") || type.includes("adversarial")) {
  return ABSTENTION_JUDGE_PROMPT
}

single-session-user, multi-session, temporal-reasoning and knowledge-update never match, so ABSTENTION_JUDGE_PROMPT is dead code for LongMemEval and the questions fall through to the exact answer rubric. The judge is then asked whether "I don't know" contains the correct answer, where the "correct answer" is a sentence explaining that the information was never mentioned. The answer is no, so the run records incorrect for exactly the behaviour the question is testing for.

This is also what framework.md already says should happen (ABSTENTION <- LoCoMo: Cat 5 | LongMemEval: abstention | ConvoMem: Abstention); the LongMemEval half of that mapping was never wired up.

Upstream switches on the same signal, in src/evaluation/evaluate_qa.py:

prompt = get_anscheck_prompt(qtype, q, ans, hyp, abstention='_abs' in entry['question_id'])

and its abstention template presents the reference as an explanation, not as an answer.

Impact

• 30 of the 500 LongMemEval questions, spread over multi-session (12), single-session-user (6), temporal-reasoning (6) and knowledge-update (6).
• The bias is systematic, not noise. Well behaved providers that say "I don't know" are pushed toward 0 percent on the abstention subset, while a provider that hallucinates a plausible answer can occasionally be scored correct, so hallucination is rewarded relative to abstention.
• Per type accuracy in the report is dragged down for the four types above, and overall accuracy and MemScore quality with it. Sampled runs (-s) hit it too, since the abstention questions sit inside the regular type buckets.

Fix

Carry an isAbstention flag from the benchmark to the judge and let it drive rubric selection, keeping the existing question type check as a fallback.

• UnifiedQuestion.isAbstention and JudgeInput.isAbstention (both optional, no checkpoint format change).
• isAbstentionQuestionId() in the LongMemEval benchmark, set on load from the _abs suffix.
• LoCoMo (adversarial) and ConvoMem (abstention_evidence) set the same flag, so all three benchmarks now express abstention the same way instead of relying on type name matching.
• getJudgePromptForType(questionType, isAbstention?) honours the flag first.
• buildJudgePrompt labels the reference Explanation for abstention questions rather than Ground Truth Answer, matching the upstream template, and one sentence was added to ABSTENTION_JUDGE_PROMPT to say the reference is an explanation.

Prompt actually sent for 0862e8bf_abs before this change:

I will give you a question, a correct answer, and a response from a model. Please answer yes if
the response contains the correct answer. Otherwise, answer no. ...

Question: How many chapters of the hamster care book have I read so far?
Ground Truth Answer: You did not mention this information. You mentioned your cat Luna but not your hamster.
System's Hypothesis: I don't know.

and after:

You are evaluating an abstention question. The correct answer is that the information was NOT in
the conversation, so the system should abstain or say it doesn't know. You are given an explanation
of why the question is unanswerable, not an answer to match against. ...

Question: How many chapters of the hamster care book have I read so far?
Explanation: You did not mention this information. You mentioned your cat Luna but not your hamster.
System's Hypothesis: I don't know.

Compatibility

• LoCoMo and ConvoMem keep resolving abstention through the type name, so their behaviour is unchanged.
• No provider, checkpoint or report schema change. Existing runs can be re-scored with -f evaluate on the same run id, no re-ingest needed.
• Question type buckets in the report are untouched, so abstention questions stay grouped under their native type exactly as upstream groups them.

Testing

• bun test (new src/judges/base.test.ts, 6 tests) covers the flagged path, the answerable question of the same type, the LoCoMo and ConvoMem type based routing, and the temporal and preference rubrics.
• bunx tsc --noEmit clean.
• Verified the dataset shape against xiaowu0162/longmemeval-cleaned (longmemeval_s_cleaned.json): 500 questions, 30 with an _abs id, all of them carrying a regular question_type.