Python SDK Overview

Use TurnWise programmatically with Python

Python SDK

The TurnWise Python SDK allows you to run evaluations programmatically using your own LLM API keys, while automatically syncing results to the TurnWise platform.

Why Use the SDK?

Use Your Own API Keys

Run evaluations locally with your OpenRouter or other LLM provider keys

Programmatic Control

Integrate evaluations into your CI/CD pipelines and automation workflows

Cost Control

Pay only for LLM API calls, not for TurnWise compute resources

Automatic Sync

Results automatically sync to TurnWise web UI for visualization

Key Features

• Local Evaluation: Run evaluations on your machine using your API keys
• Template Variables: Use @VARIABLE syntax for context-aware prompts
• Multiple Levels: Evaluate at conversation, message, or step level
• Structured Outputs: Support for text, number, checkbox, progress, and custom JSON schemas
• Batch Processing: Evaluate multiple conversations concurrently
• Goal Extraction: Automatically extracts user goals from conversations
• Progress Tracking: Real-time progress callbacks during evaluation

Installation

pip install turnwise-sdk

Quick Start

import asyncio
from turnwise import TurnWiseClient, Metric, EvaluationLevel, OutputType

async def main():
    # Initialize client
    client = TurnWiseClient(
        turnwise_api_key="tw_your_api_key",      # From TurnWise platform
        openrouter_api_key="sk-or-your_key"      # From OpenRouter
    )

    # Verify connection
    await client.verify()

    # Define a metric
    metric = Metric(
        name="Response Helpfulness",
        prompt="""Evaluate how helpful this response is.

USER GOAL: @GOAL
RESPONSE: @CURRENT_MESSAGE.output

Score from 0.0 to 1.0.""",
        evaluation_level=EvaluationLevel.MESSAGE,
        output_type=OutputType.PROGRESS,
    )

    # Run evaluation
    results = await client.evaluate(
        dataset_id=1,
        metric=metric,
        progress_callback=lambda p: print(f"Progress: {p.completed}/{p.total}")
    )

    print(f"Evaluated {len(results)} items")
    await client.close()

asyncio.run(main())

Architecture

The SDK consists of several components:

• TurnWiseClient: Main client for API communication and high-level evaluation
• EvaluationOrchestrator: Lower-level orchestrator for manual control
• OpenRouterProvider: LLM provider for making API calls
• GoalAgent: Extracts user goals from conversations
• EvaluationAgent: Executes individual evaluations

Next Steps

Installation & Setup

Get started with installation and configuration

Basic Usage

Learn how to run your first evaluation

Advanced Usage

Explore advanced features and customization

API Reference

Complete API documentation