Custom Model Evaluation
Evaluate custom-model request logs with an LLM judge and track hallucination, accuracy, relevance, and other quality metrics.
Studio Evaluation samples a custom model's request logs and sends them to another Model Zoo model acting as an LLM judge. Each run stores aggregate results, per-item scores, and the judge's rationale.
Evaluation requires request logs, an available judge model, and a judge prompt. Custom-model owners and co-editors can use the station; read-only users cannot open it.
Open Evaluation
The legacy /customize/[id]/evaluation URL redirects to /customize/[id]/evaluate. Knowledge Base Test on the same page tests retrieval and is separate from the LLM-as-judge evaluation described here.
Available metrics
| Metric | What it evaluates |
|---|---|
| Hallucination | Unsupported or ungrounded claims in the response |
| Accuracy | Whether the response is correct |
| Helpfulness | Whether the response resolves the user's need |
| Answer relevance | Whether the answer addresses the question |
| Groundedness | Whether the answer is supported by supplied context or evidence |
| Faithfulness | Whether the answer represents its context without distorting it |
| Relevance | Overall relevance to the request |
Scores are normalized to 0–1. Higher is better for the ordinary quality metrics; a lower Hallucination score is better. A record without enough evaluable data is counted as Insufficient data, not as a zero score.
Configure the judge
Use Settings in the upper-right corner to configure:
- Judge model: an available, connected base model from Model Zoo.
- Judge prompt: the criteria and output instructions for the evaluation.
- Metrics: at least one metric to calculate for each sampled request.
- Sample limit: 1–500 records per run; the default is 50.
- Automatic evaluation: whether Studio creates runs on a schedule.
- Cron schedule: the cron expression that controls run times.
- Automatic range: how far each scheduled run looks back for request logs.
- Notes: optional maintenance or purpose information.
Until a custom model saves its own configuration, the page can inherit Eval Judge defaults configured by an administrator in system-model settings. Saved custom-model values take precedence.
A run cannot start without a judge model, judge prompt, and at least one metric. Judge calls also consume model usage and incur the corresponding cost, so validate a schedule with a small sample first.
Run an evaluation
The recent-runs table shows creation time, data range, log count, primary scores, judge model, status, and Job ID. A pending or running job cannot be deleted.
Review results
Open a completed run to review:
- Aggregate Hallucination, Accuracy, and other available scores.
- Counts of evaluated and insufficient-data items.
- Per-request scores for enabled metrics.
- The judge's rationale.
- The evaluated range and judge model.
Results describe only the selected period and sampled records. After changing the system prompt, RAG, base model, or behavior settings, collect new request logs and run a new evaluation instead of treating scores from different configurations as the same baseline.
Recommended workflow
- Start with 20–50 samples and verify that the judge prompt and score direction match your intent.
- Review insufficient-data items and confirm that logs contain the question, answer, and required context.
- Use repeated low-score rationales to guide system-prompt, knowledge-base, or model changes.
- Generate new representative requests and evaluate again.
- Once the rubric is stable, enable cron scheduling and keep ranges and sample sizes comparable.