An unresolved evaluation question prompted by reading MemGPT. The proposed source-checking test is an original plan and has not been run.
Question
When does an assistant stop searching even though a relevant source note exists?
Why it matters
A fluent draft can hide a missed source. For nonfiction, the visible citation should support the specific claim.
Starting evidence
See the retrieval limitation in Packer 2023: MemGPT, §3.2.1.
Proposed test, not yet run
Prepare known source notes and questions with documented answers. Record which notes the assistant reads, which claims its answer supports, and when it admits uncertainty. Repeat with stale or contradictory notes.
What would settle it
A repeatable evaluation separating retrieval misses from errors made after the correct source was read. Track both; do not collapse them into one success claim.
Method reference: Virtual context management.