Playbook evaluations

This guide explains how to use the built-in evaluations feature in the Dialogflow CX console to verify your agent's capabilities and prevent any regressions after updates. Dialogflow CX provides out-of-the-box metrics to help evaluate your agent's performance.

All metrics except for latency require at least one test case, a "golden response" that Dialogflow CX compares the agent's performance against to measure results. Each test case can be measured in the context of an environment, which lets you specify different versions of playbooks, flows, and tools to use in the agent's performance evaluation.

(Optional) Create an environment

Creating an environment is optional. If you don't create one, the default value is Draft.

1. Go to Environments and click Create.
2. Choose the versions of the playbooks, flows, and tools that you want to use to measure agent performance.
3. Click Save to save the environment.

Create a test case

You can create a test case from an existing conversation in your conversation history, create a new conversation to save as a test case, or import test cases into Dialogflow CX.

Create a test case in the console

1. Go to Conversation history in the navigation menu.
2. To create a new conversation, activate your agent, such as by calling the agent's phone number. When you have a conversation that you want to use as a test case, select it.
3. View the conversation and verify the agent responses, tools invoked, and how each response sounds. When you're satisfied, select Create test case.
4. Provide a display name for the test case and specify your expectations of events that must happen at the conversation level. This can include tools, playbooks, and flows that you expect to be called within the conversation. Select +Add expectation to add more expectations. To have the expectations evaluated in sequential order as listed from top to bottom, toggle Sequential validation.
5. Click Save to save the test case.

Upload test cases

1. Test cases must be in the following CSV format.
2. To upload test cases to the system, click Import at the top of the test cases menu.
3. In the menu that appears, either select the locally stored file or enter the path to its Cloud Storage bucket.
4. Your test cases should now appear in the test cases menu.

Run a test case

1. Go to Test cases in the navigation menu and select the test cases that you want to compare your agent against. You can select a single test case or multiple test cases.
2. Click Run selected test cases.

Test results

1. Access results: The latest test run execution results appear for each test case in the Test Case view after completion:
   1. Semantic similarity: Measures how similar the agent's conversations were to the "golden response" (responses in the Test Case). Golden responses are required to receive this metric. Values can be 0 (inconsistent), 0.5 (somewhat consistent), or 1 (very consistent).
   2. Tool call accuracy: Reflects how faithfully the conversation includes the tools that are expected to be invoked during the conversation. Values range from 0 to 1. If no tools are used in the conversation, the accuracy appears as -- (not applicable).
   3. Latency: The total time that the agent takes to process a user request and respond to the user, measured as the difference between the end of the user utterance and the beginning of the agent response in seconds.
2. Update golden test case: If the latest run reflects expected changes due to an agent update, select Save as golden to overwrite the original Test Case.
3. Filter and sort results: You can filter and sort evaluation results by any of the generated metrics or by a specific Environment to track changes in performance after each update.

Batch import test case formatting

This section describes how to format a CSV file for importing batch test cases for your agent. The system reads this file to create a structured set of test cases, each containing one or more conversational turns.

A single test case can span multiple rows in the CSV file. The first row of a test case defines its overall properties, for example, its name and language. Each subsequent row for that test case defines a single back-and-forth turn in the conversation where the user says something and the agent replies.

Header

The CSV file must have a header row as the very first line. This header defines the data in each column.

Required headers

The two required headers must appear in the order shown. Both are required for the first row of a new test case. You can start a new test case by providing new DisplayName and LanguageCode values.

• DisplayName: The name of your test case. This is only filled in for the first row of a new test case.
• LanguageCode: The language code for the test (for example, en, en-US, es).

Optional headers

You can include the following optional headers to provide more detail for your test cases. They can be in any order after the first two required columns.

Test case metadata:

• Tags: Space-separated tags for organizing tests (for example, "payments onboarding").
• Note: Free-text notes or a description of the test case purpose.
• TestCaseConfigV2.StartResource: Specify the flow or playbook to start the test with.

User input:

• UserInput.Input.Text: The text the user enters for a given turn.
• UserInput.InjectedParameters: Parameters to inject into the conversation at the start of a turn, formatted as a JSON string.

Agent output:

• AgentOutput.QueryResult.ResponseMessages.Text: The exact text that the agent replied with.
• AgentOutput.QueryResult.Parameters: The parameters that were extracted by the agent, formatted as a JSON string.

Ordered expectations:

• OrderedExpectations.ExpectedFlow: The flow you expect to be active after the turn.
• OrderedExpectations.ExpectedIntent: The intent you expect to be matched for the turn.
• OrderedExpectations.ExpectedAgentReply: The text you expect the agent to reply with, which can be a substring of the full reply.
• OrderedExpectations.ExpectedOutputParameter: The parameters you expect to be set at the end of the turn, formatted as a JSON string.

Audio metadata:

• AudioTurnMetadata: Metadata for audio-based tests, formatted as a JSON string.

Build a test case

Test cases are organized by data rows.

1. To start a new test case, fill out its metadata row:
   • Rule: This row must have a value in the DisplayName column.
   • Action: Enter values for DisplayName and LanguageCode. Include tags, notes, or a TestCaseConfigV2.StartResource in this row as needed. Leave conversation-turn columns (such as UserInput.Input.Text) empty in this row. If you use tags, separate each tag with a space (for example, tag1 tag2 tag3). If you use TestCaseConfigV2.StartResource, prefix the resource name with start_flow: or start_playbook: (for example, start_flow:projects/p/locations/l/agents/a/flows/f).
2. Add a conversational turn to the test case you just started by adding a new row immediately below it:
   • Rule: The DisplayName column must be empty. This tells the parser that it's a turn belonging to the previous test case.
   • Action: Fill in the columns that describe the user's action and the expected agent response for this turn, such as UserInput.Input.Text and OrderedExpectations.ExpectedAgentReply. For columns requiring JSON, you must provide a valid JSON object as a string (for example, {"param_name": "param_value", "number_param": 123}).