Skip to main content

​
Python SDK

The TurnWise Python SDK allows you to run evaluations programmatically using your own LLM API keys, while automatically syncing results to the TurnWise platform.

​
Why Use the SDK?

Use Your Own API Keys

Run evaluations locally with your OpenRouter or other LLM provider keys

Programmatic Control

Integrate evaluations into your CI/CD pipelines and automation workflows

Cost Control

Pay only for LLM API calls, not for TurnWise compute resources

Automatic Sync

Results automatically sync to TurnWise web UI for visualization

​
Key Features

  • Local Evaluation: Run evaluations on your machine using your API keys
  • Template Variables: Use @VARIABLE syntax for context-aware prompts
  • Multiple Levels: Evaluate at conversation, message, or step level
  • Structured Outputs: Support for text, number, checkbox, progress, and custom JSON schemas
  • Batch Processing: Evaluate multiple conversations concurrently
  • Goal Extraction: Automatically extracts user goals from conversations
  • Progress Tracking: Real-time progress callbacks during evaluation

​
Installation

pip install turnwise-sdk

​
Quick Start

import asyncio
from turnwise import TurnWiseClient, Metric, EvaluationLevel, OutputType

async def main():
    # Initialize client
    client = TurnWiseClient(
        turnwise_api_key="tw_your_api_key",      # From TurnWise platform
        openrouter_api_key="sk-or-your_key"      # From OpenRouter
    )

    # Verify connection
    await client.verify()

    # Define a metric
    metric = Metric(
        name="Response Helpfulness",
        prompt="""Evaluate how helpful this response is.

USER GOAL: @GOAL
RESPONSE: @CURRENT_MESSAGE.output

Score from 0.0 to 1.0.""",
        evaluation_level=EvaluationLevel.MESSAGE,
        output_type=OutputType.PROGRESS,
    )

    # Run evaluation
    results = await client.evaluate(
        dataset_id=1,
        metric=metric,
        progress_callback=lambda p: print(f"Progress: {p.completed}/{p.total}")
    )

    print(f"Evaluated {len(results)} items")
    await client.close()

asyncio.run(main())

​
Architecture

The SDK consists of several components:
  • TurnWiseClient: Main client for API communication and high-level evaluation
  • EvaluationOrchestrator: Lower-level orchestrator for manual control
  • OpenRouterProvider: LLM provider for making API calls
  • GoalAgent: Extracts user goals from conversations
  • EvaluationAgent: Executes individual evaluations

​
Next Steps

Installation & Setup

Get started with installation and configuration

Basic Usage

Learn how to run your first evaluation

Advanced Usage

Explore advanced features and customization

API Reference

Complete API documentation