Repository navigation
MemoryParsingService classifies error-adjacent preferences and instructions as error #1890
Description
Found while building a migration showcase for #1609. Imported memories are
auto-classified in bulk, which is how the pattern became visible.
Summary
MemoryParsingService classifies a preference or instruction as error
whenever it happens to mention errors, failures, or bugs. The rules match the
topic of errors rather than the reporting of one, so "how I want errors
presented" lands in the same bucket as "an error occurred".
Reproduction
Rule-based path only, so no API key and no network:
from memanto.app.core import MemoryRecord
from memanto.app.services.memory_parsing_service import MemoryParsingService
svc = MemoryParsingService()
for text in [
"Show the failing case first, then the fix.",
"Prefer answers that explain what went wrong before the solution.",
"I want error messages explained in plain English.",
"When reviewing my code, always point out the bug before suggesting a fix.",
"Always run the failing test before changing anything.",
]:
got = svc.parse_memory(MemoryRecord(
title=text[:100], content=text,
agent_id="a", actor_id="a", source="user")).type
print(f"{got:<12} {text}")Output on main (c84429c):
error Show the failing case first, then the fix.
error Prefer answers that explain what went wrong before the solution.
error I want error messages explained in plain English.
instruction When reviewing my code, always point out the bug before suggesting a fix.
instruction Always run the failing test before changing anything.
The first three are a user preference or standing instruction, not an incident.
Root cause
Classification takes the single strongest rule score. The error ruleset scores
5 for a bare word match:
r"\b(?:errors?|fail(?:s|ed|ing|ure)?|bugs?|exceptions?|tracebacks?|crash(?:e[ds]|ing)?|outages?|incidents?)\b"For "i want error messages explained in plain english." that is the only
rule that fires at all, so error wins uncontested. There is no competing
signal for a leading "I want ..." or "Prefer ...".
Note the last two cases classify correctly, so this is not "the error rules are
too broad" in general. It is specifically that a topical mention wins when no
intent phrase competes with it.
Impact
Memory type drives retrieval filtering (#memory_type:preference). A preference
filed under error will not surface when an agent asks for preferences, and it
pollutes error queries with things that are not errors.
This compounds during migration. Imported memories are auto-classified in bulk,
and assistant memory is full of sentences about how the user wants problems
explained, so the misfile rate is meaningfully higher on migrated data than on
hand-written memories.
Possible directions
Deliberately not prescribing a fix, but the options seem to be: require an
error rule to co-occur with a reporting signal (past tense, a stack trace, an
identifier) before scoring 5; lower the bare-keyword score below
MIN_RULE_SCORE on its own; or add intent-phrase rules for "I want", "Prefer",
"I like" so preference has something to compete with.
Happy to open a PR if a direction is preferred.
Activity
Suraj-kumar00 commentedon Aug 23, 2026
Hello @mjfekri @Xenogents, could you please review on this?
Xenogents commentedon Aug 24, 2026
Hi @Suraj-kumar00! I'll discuss this with the team but I think devising a solution to this will have to be a little bit more involved as there are probably a numerous amount of edge cases to consider along with the examples you've provided and it's probably impossible to perfectly resolve all of them. There are just a lot of limitations trying to do intent parsing using basic text matching. Thanks for bringing this up!
Xenogents commentedon Aug 25, 2026
Ok I brought up the issue to the team, it is more or less a calculated loss as we still prioritize latency the most. Not that the auto-parser can't be improved but we're willing to accept the inaccuracies in its logic. We're still open to ideas on improving the auto-parser like changing around the values, switching up the fallback logic, or maybe changing how text is weighted entirely, just not super high priority.
You're welcome to suggest any changes and I'll leave this open so maybe others could come up with some unique ideas!
Hariharanpugazh commentedon Oct 3, 2026
I reproduced this on main the three preference/instruction cases land as
errorbecause the bare error-keyword rule scores 5 and nothing competes when the sentence has no intent verb.Of the directions you listed, I'd lean toward adding intent-phrase rules ("I want", "I'd like", "Prefer that...") so preference/instruction have a
competing signal, rather than lowering the error score additive, so it won't disturb the cases that already classify right. Happy to open a PR on that approach if it sounds good to you.
Xenogents commentedon Oct 14, 2026
@Hariharanpugazh Thanks for the suggestion! I think that sounds fine but make sure those changes don't compete with #1997, that is a PR I believe we plan on merging eventually. This is a bit of a low priority issue so been slow to get around to it.