AI Snips
Chapters
Transcript
Episode notes
More Context Isn't Always Better
- Increasing input context length (more documents or history) does not guarantee better outputs.
- As context length grows, model accuracy steadily declines, especially for middle-positioned information.
Return Fewer, More Relevant Documents
- When using retrieval for QA, prioritize returning fewer documents and ensure the single relevant document appears near the start or end.
- Limit retrieved documents to reduce noise and raise the chance the model finds the correct answer.
Surprised By Open-Book Regression
- Sally-Ann recounts being surprised that GPT-3.5 performed worse with documents than in closed-book.
- The paper changed how she redesigned production systems to give less, more targeted context.