Custom software developmentPay for results
Improve speaker attribution and claim quality in an LLM extraction pipeline
Project brief
Improve an existing Python extractor for an editorial product that turns conversational transcripts into claims for human review.
This is one bounded evaluation-and-prompt milestone: one pipeline version, one agreed model and up to 100 labeled excerpts. Diagnose errors, revise prompt/context handling, and compare the baseline with one candidate on a held-out split. Retain useful uncertain statements while measuring attribution mistakes and queue noise.
Deliver reproducible Python evaluation code, versioned prompts and an evidence-based handoff. No guaranteed metric uplift, production integration or ongoing development. Agree authorized transcripts, labels, repository access, split and API allowance before starting.
Deliverables & acceptance
What you'll deliver
- Evaluation scripts, revised prompts and a concise comparison report.
What the result must meet
- Report precision/recall, attribution errors and representative regressions on an untouched held-out split; distinguish label issues from model/prompt failures.
- Reproduce baseline and candidate results from documented commands and disclose uncertainty and remaining failures.