1.1 Composing AI Agent Tests Visually on the Step Canvas

How to build multi-turn conversational agent test cases visually on the Step Builder canvas without writing code or YAML.

Composing a Multi-Turn Conversation Without Writing YAML by Hand

Testing a conversational AI agent means orchestrating multi-turn dialogue, intercepting tool calls deterministically, and validating semantic responses – three different concerns that, hand-authored as a script or a raw JSON/YAML file, quickly turn into something only the person who wrote it can safely edit. CertiQA’s Step Canvas keeps each of those concerns as its own modular card instead: a user turn, a tool mock, a tool-call expectation, and a judge rubric are each configured through a purpose-built form rather than typed as free-form structured text, so a case stays readable and editable by someone who didn’t write the original version.

Getting to the Step Canvas

• 1. Register your agent under Project Settings -> Agent Under Test (invoke URL, protocol, credentials).
• 2. Open Test Suites and click “+ New Suite” → choose “AI Agent Suite” – this is what tells CertiQA to use the AI Agent step vocabulary (SEND_USER_MESSAGE, MOCK_TOOL_RESPONSE, etc.) instead of the Web UI one.
• 3. Click on any Test Case to open the Visual Step Builder Canvas.
• 4. Click “+ Add Step” to visually add user messages, tool mocks, and LLM judge rubrics – each card exposes only the fields relevant to that step type.
• 5. Click “▶ Run Case” to execute live against your agent endpoint.

Prerequisites

• Target AI agent invoke URL (OpenAI-compatible, Anthropic Claude, or Webhook format).
• Agent authentication credentials (API key or Bearer token) stored in Vault.

Building the Turn Sequence, Step by Step

1. {“title”:“Step 1: Add SEND_USER_MESSAGE”,“text”:“Select SEND_USER_MESSAGE. Type the user prompt (e.g. "Where is order {{ orderId }}?"). Set Time-To-First-Token (TTFT) ceiling (e.g. 400ms) and maximum turn latency – these turn a vague "the agent felt slow" complaint into a concrete, repeatable assertion.”}
2. {“title”:“Step 2: Add MOCK_TOOL_RESPONSE”,“text”:“Select MOCK_TOOL_RESPONSE. Enter the tool name (e.g. lookup_order) and paste the mock JSON payload you want returned. Add simulated latency (e.g. 150ms) to test timeouts – see 1.2 Local Tool Interception for the three matching modes available here.”}
3. {“title”:“Step 3: Add EXPECT_TOOL_CALL”,“text”:“Select EXPECT_TOOL_CALL. Pick the tool name and choose argument matching mode: EXACT, PARTIAL (key-value check), or SCHEMA (Ajv JSON Schema validation) – the same three modes a mock uses, so the vocabulary for "what was sent" and "what to send back" stays consistent.”}
4. {“title”:“Step 4: Add EXPECT_RESPONSE (LLM-as-a-Judge)”,“text”:“Select EXPECT_RESPONSE. Choose rubric type (GROUNDEDNESS, GUARDRAIL, or RESPONSE_QUALITY) and write evaluation criteria in plain English – see 1.3 LLM-as-a-Judge for what each rubric type actually checks and how to write one that grades consistently.”}

Two Things That Bite People Early

• Agent Tool Routing: ensure your agent is configured to send tool requests to the runner interceptor port (4790) or proxy URL during test runs – a mock that never intercepts anything usually means routing, not the mock configuration, is the actual problem.
• Session Isolation: use RESET_SESSION between independent test cases so conversational context from previous runs does not pollute subsequent tests – an agent that “remembers” something from a prior case is a common, confusing source of flaky-looking failures that aren’t actually flaky.