Repository navigation
Best way to limit the number of matches + negative examples #265
Replies: 3 comments
|
Hi Jonathan, Were you able to solve this? I think i have a similar use case. I have a prompt and example combination that works for a small pdf. When the same prompt and examples are used for a larger pdf (and not incredibly big one either) the output generates a lot of classes for which no example where provided. I am running the model with a temperature of 0.0. My current approach was similar to yours. I added these lines to the prompt: But the execution with (or without) those lines generates very similar outputs (same extraction classes that were never shown in the examples provided, different attributes). About negative examples, I had posted this (#238) a while back. So far, we have managed to work arrund it by improving our examples. |
To reduce duplicates out of the box, set max_workers=1 and use a large max_char_buffer (large token window). The main problem I’m facing is performance. Maybe adding shared memory between workers, or adding a post-hook for deduplication that we can override, would help us meet our goals. |
|
Do you know how many matches your document have?
In our experiments we found that the generation of classes that were not
seen are a result of the prompt and the underlying model.
A very direct prompt makes this less likely, I went to the extreme of
adding a line like: "IF you provide extractions with classes you have not
seen before YOU HAVE FAILED!", Yet at the same time, we have observed small
models, and quantized models tend to get creative with classifications.
REgarding negative examples, If don't think the current json schema
accept this. I have been trying a few variations but have not been able to
consistently get results.
…On Mon, 8 Dec 2025 at 13:21, liadlevy-pando ***@***.***> wrote:
I have a document that has 6 entities that I want to extract, and I've
provided 16 examples of things to match, but the tool is identifying over
40 entities.
Is there an optimal way to instruct the tool to be parsimonious in what it
returns? I could put it in the prompt, but I was wondering if this is the
ideal, or if there's some parameter to adjust.
I was also wondering if there's a mechanism to instruct the tool of things
I don't want to match. Like if I wanted to extract vehicle information
about public transit vehicles (bus, train, airplane), but not personal
vehicles. Currently I'm providing 8 examples of the public transit
vehicles, and 8 examples of personal vehicles, but I feel like that's
making the task too complex.
To reduce duplicates out of the box, set max_workers=1 and use a small
max_char_buffer (small token window). The main problem I’m facing is
performance.
Maybe adding shared memory between workers, or adding a post-hook for
deduplication that we can override, would help us meet our goals.
—
Reply to this email directly, view it on GitHub
<#265 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AISMMARUGYTSJK6LQ4SWP234AV3L5AVCNFSM6AAAAACJJIUJLKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKMJZGYZDSOI>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Uh oh!
There was an error while loading. Please reload this page.
I have a document that has 6 entities that I want to extract, and I've provided 16 examples of things to match, but the tool is identifying over 40 entities.
Is there an optimal way to instruct the tool to be parsimonious in what it returns? I could put it in the prompt, but I was wondering if this is the ideal, or if there's some parameter to adjust.
I was also wondering if there's a mechanism to instruct the tool of things I don't want to match. Like if I wanted to extract vehicle information about public transit vehicles (bus, train, airplane), but not personal vehicles. Currently I'm providing 8 examples of the public transit vehicles, and 8 examples of personal vehicles, but I feel like that's making the task too complex.
All reactions