Stop Asking Whether the Model Is Accurate
Accuracy without a defined task, distribution and consequence is not a useful production measure.
“How accurate is it?” sounds rigorous while hiding the decisions that matter. Accurate at which task, on whose data, under which conditions and with what cost of failure?
Define representative cases and score dimensions separately: factual support, completeness, instruction adherence, action validity and appropriate refusal. Weight severe errors differently from stylistic imperfections.
Compare with the current human or system baseline. Measure the complete workflow, including review and recovery. Segment results so good average performance cannot conceal failure for rare or vulnerable groups.
The production question is whether the system performs a declared job within an acceptable risk envelope. A single accuracy percentage cannot answer it.
Model selection is often secondary to system design. Better retrieval, clearer task boundaries, deterministic validation and a designed refusal can improve outcomes more than moving to a larger model. Evaluate those interventions separately so “the model” does not become the default explanation for failures created elsewhere.