30-SECOND SUMMARY
What to take away
- Create 10–20 representative prompts with known evidence.
- Change one variable at a time and repeat each prompt.
- Track critical failures and operational limits, not only average quality.
A repeatable model-selection loop
Change one variable per comparison.
- 01PROMPTS
Fixed cases
- 02REPEAT
Run multiple times
- 03SCORE
Quality and operations
- 04DECIDE
Apply thresholds
Define the job
Replace ‘find the smartest model’ with a concrete objective such as ‘summarize English support tickets without inventing refund promises.’
Specify the device, acceptable latency, memory ceiling, language, and disqualifying failures.
Build a small representative set
Include common requests, rare exceptions, formatting, answer-not-found, conflicting instructions, and safety cases. Attach expected answer points and source evidence.
Use public or anonymized data so the evaluation set does not become a new sensitive repository.
Fix the conditions
Record prompt, temperature, context, runtime version, quantization, and hardware state. Change only one variable per comparison.
Separate first-run loading time from repeated generation and run each prompt at least three times to observe variance.
Score quality and operations separately
Quality can include factual accuracy, instruction following, evidence alignment, format validity, and language quality. Operations can include time to first output, total generation time, peak memory, and failure rate.
A critical error may disqualify a model even if its average score is high.
| Test | What to record | Critical failure example |
|---|---|---|
| Fact extraction | Correct names and numbers | Invented value |
| Summary | Coverage and additions | Unsupported claim |
| Structured output | Schema validity | Parsing failure |
| No answer | Appropriate uncertainty | Confident fabrication |
| Operations | Latency, memory, errors | Crash or timeout |
Switch models only when the gain matters
Define a threshold that a new model must exceed before replacing the current one. This prevents constant churn whenever a new release appears.
Rerun the same set after runtime or model updates and keep a rollback path for regressions.
Frequently asked questions
Are public benchmarks useless?
No. They help shortlist candidates, but they do not replace tests on your language, documents, hardware, and failure conditions.
How many prompts should I start with?
A carefully reviewed set of 10–20 is practical. Add real failures as permanent regression cases.
What speed metric matters most?
Measure time to first output and sustained generation separately, and distinguish first from repeated runs.
Primary sources
Check the original documentation for version-specific details.
Ollama Chat API Metrics