Repository navigation
PrefEval drops a sample when search returns no text memories, so the judge gets an empty file #2526
Description
Pre-submission checklist | 提交前检查
- I have searched existing issues and this hasn't been mentioned before
- I have read the project documentation and confirmed this issue doesn't already exist
- This issue is specific to MemOS and not a general software issue
Bug Description | 问题描述
evaluation/scripts/PrefEval/pref_memos.py search_memory_for_line always reads relevant_memories["text_mem"][0]["memories"]. A successful search payload with "text_mem": [] raises IndexError: list index out of range. The except block prints the error and returns None.
The search, response, and add writers keep a row only when if result: is true. A None result is not written. generate_response_for_line does the same on an API error: it prints the error and returns None, so that row is also omitted.
evaluation/scripts/PrefEval/pref_eval.py then opens the response file, sees no lines, prints Error: Input file is empty., and returns before any judge call. An empty memory list, which the response function itself treats as valid when retrieved_memories_text is "", never reaches the judge. A failed answer call is also dropped instead of being recorded as an error row.
On a two-row PrefEval smoke run, one search returned no text_mem and logged list index out of range. The other row reached answer generation, the chat gateway returned HTTP 503, and that exception also returned None. The response file was 0 bytes, and pref_eval.py reported an empty input. The 503 is an environment failure. The missing judge input is the script discarding the row. The empty-text_mem crash happens with a healthy model as well.
How to Reproduce | 如何重现
- From
evaluation/, importsearch_memory_for_lineinevaluation/scripts/PrefEval/pref_memos.pyon commita7367d07. - Call it with a client whose
searchreturns{"text_mem": [], "pref_string": ""}and a JSONL line{"user_id": "u1", "question": "What should I study?", "metrics": {}}. - Observed:
Error searching memory for line 1 (user_id: u1): list index out of range, and the function returnsNone. - The search writer (
if result: outfile.write(...)) writes 0 lines. - Run
evaluation/scripts/PrefEval/pref_eval.pyon that empty response file. - Observed:
Error: Input file is empty.No judge prompt is sent.
Environment | 环境信息
- Python version: 3.12.15
- Operating System: macOS (darwin)
- MemOS version: MemoryOS 2.0.33 (commit
a7367d07)
Willingness to Implement | 实现意愿
- I'm willing to implement this myself
Activity
Memtensor-AI commentedon Oct 21, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
1e7e18e1535eb444Working branch:
bugfix/autodev-2526-20261021095907885Target branch:
latest dev* branchWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
fix: preserve PrefEval rows when text_mem is empty or answer call fails
Memtensor-AI commentedon Oct 21, 2026
✅ AutoDev task
1e7e18e1535eb444completed.Summary: Fixes #2526.
evaluation/scripts/PrefEval/pref_memos.pyno longer drops PrefEval rows when the MemOS product search returns an emptytext_memlist, nor when the chat gateway raises on the answer call.search_memory_for_linenow defensively extracts the first text_mem bucket's memories (empty list →retrieved_memories_text = "" + pref_string, which is the "no retrieved memories" state the response stage already treats as valid). Bothsearch_memory_for_lineandgenerate_response_for_lineannotate the row with anerrorfield instead of returningNoneon exception, and the response helper also setsresponse = ""sopref_eval.py's judge (data["response"]) still fires — classifying the failed call asUnhelpful Response, the correct accounting for an upstream API failure.Added
tests/evaluation/test_pref_memos.pywith 5 pytest cases: emptytext_memkeeps the row with the rightretrieved_memories_text; search exception keeps the row witherror+ emptyretrieved_memories_text; OpenAI exception keeps the row witherror+response = ""; search-flagged pass-through row gains an emptyresponsekey (no OpenAI call); and a happy-path regression test pinning the pre-existing"- mem_1\n- mem_2\npref-string-X"output format. All 5 tests fail against the unmodified file (4 as the bug and 1 as the pass-through gap) and pass after the fix.ruff formatandruff checkare clean on both touched files.opsp artifacts live under
.ai-tasks/andopenspec/changes/(excluded from the working branch via.git/info/exclude) and have been archived to the siblingmemos-autodev-specsrepo under2026-10-21-2526-prefeval-drops-a-sample-when-search-returns-no-text-memories/and pushed tomain.Base branch:
mainBranch:
bugfix/autodev-2526-20261021095907885Commit:
1602f82PR: #2530
Assigned to: @CarltonXiang
Reviewers: @WeiminLee