With Tool Usage

Conversation with tool calls and results

Conversation with Tool Usage

This example shows how to capture tool calls and their results.

The Scenario

A customer is requesting a refund for a delayed order. The agent needs to look up the order and process the refund.

JSON

{
  "conversations": [
    {
      "name": "Refund Request",
      "description": "Customer requesting refund for delayed order",
      "agents": [
        {
          "name": "Support Agent",
          "description": "Customer support agent handling refund requests",
          "tools": {
            "lookup_order": {
              "name": "lookup_order",
              "description": "Look up order details by order ID",
              "parameters": {
                "order_id": "string"
              }
            },
            "process_refund": {
              "name": "process_refund",
              "description": "Process refund for an order",
              "parameters": {
                "order_id": "string",
                "amount": "number"
              }
            }
          }
        }
      ],
      "messages": [
        {
          "role": "user",
          "sequence_number": 0,
          "content": "Hi, I ordered a product 2 weeks ago but it hasn't arrived yet. Can I get a refund?"
        },
        {
          "role": "assistant",
          "sequence_number": 1,
          "content": "I understand your concern about the delayed order. To help you with a refund, I'll need your order ID. Can you please provide it?",
          "steps": [
            {
              "model_name": "gpt-4",
              "step_order": 0,
              "thinking": "Customer wants a refund for a delayed order. I need the order ID to look it up and process the refund.",
              "output_content": "I understand your concern about the delayed order. To help you with a refund, I'll need your order ID. Can you please provide it?"
            }
          ]
        },
        {
          "role": "user",
          "sequence_number": 2,
          "content": "My order ID is ORD-12345"
        },
        {
          "role": "assistant",
          "sequence_number": 3,
          "content": "Thank you for providing the order ID. I've looked up your order ORD-12345 and can see it was placed 14 days ago and is still pending. I've processed your refund of $49.99. Your refund reference number is REF-789, and it should appear in your account within 5-7 business days.",
          "steps": [
            {
              "model_name": "gpt-4",
              "step_order": 0,
              "thinking": "Customer provided order ID. Let me look up the order details.",
              "tool_call": {
                "name": "lookup_order",
                "arguments": {
                  "order_id": "ORD-12345"
                }
              },
              "tool_result": {
                "order_id": "ORD-12345",
                "status": "pending",
                "order_date": "2024-01-15",
                "amount": 49.99
              }
            },
            {
              "model_name": "gpt-4",
              "step_order": 1,
              "thinking": "Order found. It's been 14 days and still pending. This qualifies for a refund. Processing now.",
              "tool_call": {
                "name": "process_refund",
                "arguments": {
                  "order_id": "ORD-12345",
                  "amount": 49.99
                }
              },
              "tool_result": {
                "refund_id": "REF-789",
                "status": "processed",
                "amount": 49.99
              }
            },
            {
              "model_name": "gpt-4",
              "step_order": 2,
              "thinking": "Refund processed successfully. Let me inform the customer with all the details.",
              "output_content": "Thank you for providing the order ID. I've looked up your order ORD-12345 and can see it was placed 14 days ago and is still pending. I've processed your refund of $49.99. Your refund reference number is REF-789, and it should appear in your account within 5-7 business days."
            }
          ]
        },
        {
          "role": "user",
          "sequence_number": 4,
          "content": "Great, thank you so much!"
        },
        {
          "role": "assistant",
          "sequence_number": 5,
          "content": "You're very welcome! I'm glad I could help. If you have any other questions, feel free to ask. Have a great day!",
          "steps": [
            {
              "model_name": "gpt-4",
              "step_order": 0,
              "thinking": "Customer is satisfied with the resolution. I should acknowledge and offer further assistance.",
              "output_content": "You're very welcome! I'm glad I could help. If you have any other questions, feel free to ask. Have a great day!"
            }
          ]
        }
      ]
    }
  ]
}

Key Points

Agent Definition

The agent is defined with its available tools, documenting what capabilities it had.

Multiple Steps

Message at sequence_number 3 has three steps:

1. Step 0: Calls lookup_order tool
2. Step 1: Calls process_refund tool
3. Step 2: Generates the final response

Tool Call Structure

"tool_call": {
  "name": "lookup_order",
  "arguments": {
    "order_id": "ORD-12345"
  }
}

Tool Result Structure

"tool_result": {
  "order_id": "ORD-12345",
  "status": "pending",
  "order_date": "2024-01-15",
  "amount": 49.99
}

What This Enables

With this structured data, you can evaluate:

• Tool Selection: Did the agent use the right tools?
• Argument Accuracy: Were the tool arguments correct?
• Result Handling: Did the agent correctly interpret tool results?
• Response Quality: Was the final response accurate and helpful?