Metric Templates

Using and creating reusable metric templates

Metric Templates

Metric templates allow you to save and reuse metric configurations across datasets. TurnWise includes built-in templates and lets you create custom ones.

What Are Templates?

Templates are pre-configured metrics you can reuse:

• System Templates: Built-in templates provided by TurnWise
• User Templates: Custom templates you create
• Reusable: Use the same metric across multiple datasets
• Consistent: Ensures consistent evaluation criteria

Built-In Templates

TurnWise includes system templates for common evaluation needs:

Conversation-Level Templates

• Goal Achievement: Did the conversation achieve its goal?
• Overall Quality: Overall conversation quality score
• Customer Satisfaction: Customer satisfaction rating
• Conversation Efficiency: How efficiently was the goal achieved?

Message-Level Templates

• Response Helpfulness: Is the response helpful?
• Response Accuracy: Is the information accurate?
• Tone Appropriateness: Is the tone appropriate?
• Response Completeness: Is the response complete?

Step-Level Templates

• Tool Selection: Was the correct tool selected?
• Parameter Accuracy: Were tool parameters correct?
• Reasoning Quality: Is the reasoning sound?
• Error Detection: Did the agent detect errors?

System templates are always available - You can use them in any dataset without setup.

Using Templates

Via UI

1. Click “Add Column”
2. Select “Use Template” (if available)
3. Choose Template
   • Browse system templates
   • Browse your custom templates
4. Review Configuration
   • Template provides pre-filled values
   • Edit if needed
5. Create
   • Click “Create” to add the metric

Via API

# List available templates
GET /metric-templates

# Use template to create metric
POST /evaluation-pipeline-nodes
{
  "template_id": 5,
  "dataset_id": 1,
  ...
}

Creating Custom Templates

Step 1: Create a Metric

Create a metric as usual (see Creating Metrics).

Step 2: Save as Template

After creating the metric:

1. Open Metric Settings
2. Click “Save as Template”
3. Fill Template Info
   • Name: Template name
   • Description: What this template evaluates
   • Category: Optional category
4. Save

Step 3: Use Template

Your template is now available:

• In the template selector
• Across all your datasets
• For future use

Template Structure

Templates store:

{
  "name": "Response Helpfulness",
  "description": "Evaluates if assistant responses are helpful",
  "evaluation_level": "message",
  "prompt": "Is @CURRENT_MESSAGE.output helpful? Answer yes or no.",
  "output_type": "checkbox",
  "model_name": "openai/gpt-5-nano",
  "is_advanced": false,
  "template_variables": null
}

Advanced templates include:

{
  "name": "Context-Aware Helpfulness",
  "evaluation_level": "message",
  "prompt": "Evaluate @CURRENT_MESSAGE.output for helpfulness given @PREVIOUS_USER_MSG",
  "is_advanced": true,
  "template_variables": ["@CURRENT_MESSAGE.output", "@PREVIOUS_USER_MSG"]
}

Best Practices

Naming Templates

Use descriptive names:

• ✅ “Response Helpfulness - Customer Support”
• ✅ “Tool Selection Accuracy - E-commerce”
• ❌ “Metric 1”
• ❌ “Test”

Organizing Templates

Group by use case:

• Customer Support Templates
• E-commerce Templates
• Technical Support Templates
• Research Assistant Templates

Documenting Templates

Include clear descriptions:

• What the template evaluates
• When to use it
• What output to expect

Sharing Templates

Currently, templates are user-specific. To share:

1. Export Template Configuration
   • Copy the JSON configuration
2. Share with Team
   • Share via documentation
   • Include in team wiki
3. Import Template
   • Create metric from configuration
   • Save as template

Team sharing coming soon - Future versions will support shared team templates.

Example Templates

Template 1: Simple Helpfulness

{
  "name": "Simple Helpfulness",
  "description": "Basic helpfulness check",
  "evaluation_level": "message",
  "prompt": "Is this response helpful? Answer yes or no.",
  "output_type": "checkbox",
  "model_name": "openai/gpt-5-nano"
}

Template 2: Quality Score

{
  "name": "Response Quality Score",
  "description": "Comprehensive quality rating",
  "evaluation_level": "message",
  "prompt": "Rate response quality from 0-1 considering accuracy, completeness, and helpfulness.",
  "output_type": "progress",
  "model_name": "openai/gpt-5-nano"
}

Template 3: Tool Evaluation

{
  "name": "Tool Selection Correctness",
  "description": "Evaluates if correct tool was selected",
  "evaluation_level": "step",
  "prompt": "Given @PREVIOUS_STEP.tool_result, was @CURRENT_STEP.tool_call the correct tool? Answer yes or no.",
  "output_type": "checkbox",
  "model_name": "openai/gpt-5-nano",
  "is_advanced": true,
  "template_variables": ["@PREVIOUS_STEP.tool_result", "@CURRENT_STEP.tool_call"]
}

Managing Templates

Viewing Templates

• System Templates: Always visible
• Your Templates: Listed in template selector
• All Templates: View via API

Editing Templates

1. Open Template
2. Edit Configuration
3. Save Changes

Editing affects future use - Changes apply to new metrics created from template, not existing ones.

Deleting Templates

1. Open Template Settings
2. Click “Delete”
3. Confirm

Deletion is permanent - Deleted templates cannot be recovered.

Next Steps

Creating Metrics

Learn how to create metrics

Advanced Metrics

Explore advanced metric features