Customer Troubleshooting Playbook

CertiQA Enterprise Support & Triage Playbook

An authoritative diagnostic and troubleshooting manual for Support Engineers, QA Leads, and Solutions Architects triaging issues on the CertiQA platform.

🧭 Triage Decision Matrix

                                  [TEST FAILURE / ISSUE REPORTED]
                                                 │
                   ┌─────────────────────────────┼─────────────────────────────┐
                   │                             │                             │
          [🤖 AI AGENT TESTING]         [🌐 WEB UI & CERTIHEAL]       [🏢 ENTERPRISE & AUTH]
                   │                             │                             │
         • Tool Interception Mismatch   • Broken Selectors / Healing   • CMEK Key Decryption Error
         • LLM Judge False Verdict      • Iframe & Timeout             • BYOK Quota Exceeded
         • TTFT / Latency SLA Breach    • Flake Quarantine Rule        • Scoped API Token 403

1. AI Agent Testing & Red-Teaming Triage

1.1 Issue: `Expected tool call to <tool_name> was not observed`

Symptom: Step EXPECT_TOOL_CALL fails with AssertionError: Expected tool call was not observed.
Root Cause & Diagnosis:
1. Agent Did Not Invoke Tool: The model chose to answer directly rather than call the tool. Check the reasoning trace (<thinking>) in the Transcript Drawer.
2. Schema / Argument Mismatch: If argsMatch: EXACT or SCHEMA was used, inspect whether the agent passed extra parameters or slight spelling differences (e.g. order_id vs orderId).
3. Interceptor Unreachable: If the agent is remote (hosted in cloud/K8s), verify the agent’s toolCallbackUrl can reach the runner’s interceptor port (default: 4790).
Resolution:
- In YAML / Step Builder, switch argsMatchMode from EXACT to PARTIAL.
- For remote agents, ensure reverse proxy or ngrok tunnel routes traffic to http://runner-host:4790/tool-call.

1.2 Issue: LLM Judge Verdict Misclassification (False Positive/Negative)

Symptom: Agent gives a correct answer, but EXPECT_RESPONSE marks it as FAIL, or vice versa.
Root Cause & Diagnosis:
1. Ambiguous inlineRubric text without explicit constraints.
2. For RAG evaluations, missing referenceContext (ground truth doc snippet).
3. Judge model parse failure (judge returned conversational text instead of valid JSON { "pass": boolean, "score": number, "reasoning": string }).
Resolution:
- In certiqa-api, the JudgeServiceImpl automatically retries JSON formatting up to 2 attempts. If it persists, specify rubricType: GROUNDEDNESS and supply referenceContext.
- Tighten rubric language: Replace "Ensure agent is polite" with "Assert that response directly addresses delivery delay without asking for passwords".

1.3 Issue: High Stochastic Flakiness in Agent Tests

Symptom: Test case passes intermittently between consecutive test runs.
Root Cause: LLM non-determinism, temperature $> 0$, or floating-point token sampling.

Resolution:

Add statistical repeats to the terminal assertion step:

- action: EXPECT_RESPONSE
  inlineRubric: "Ensure response contains confirmation code"
  repeatCount: 5
  passThreshold: 0.8 # Passes if at least 4 out of 5 runs succeed

1.4 Issue: Anthropic Messages API or MCP Protocol Failures

Symptom: AxiosError: 401 Unauthorized or Invalid JSON-RPC 2.0 request.
Diagnosis:
- For Anthropic Claude endpoints, check whether API key was supplied with x-api-key or Authorization: Bearer.
- For Model Context Protocol (MCP), ensure the client sends { "jsonrpc": "2.0", "method": "tools/call", "params": { ... } }.
Resolution:
- AnthropicMessagesAdapter handles both Bearer tokens and raw API keys automatically. Pass --protocol anthropic in CLI.

2. Web UI Automation & CertiHeal™ Triage

2.1 Issue: Step Failed Due to Changed DOM Selector

Symptom: Button or input moved, class name changed, or ID randomized (e.g. React/Vue hashed classes btn_x9a8f).
Diagnosis:
1. Check if CertiHeal™ is enabled on the project settings.
2. Open the Run Report and check the Triage Proposal Drawer. CertiHeal computes a similarity score across:
  - data-testid / aria-label
  - Text content and semantic proximity
  - DOM hierarchy coordinates
  - AI visual locator matcher
Resolution:
- If CertiHeal healed the element during execution, the test will display as HEALED (Auto-Recovered).
- Click “Apply Remediation Patch” in the UI to permanently update the primary selector in the test case without re-recording.

2.2 Issue: Quarantine & Flaky Test Triaging

Symptom: Unstable test blocking CI/CD deployment pipelines.
Diagnosis: Open Test Health & Flakiness Center (/health).
Resolution:
- Click “Quarantine Case”. Quarantined test cases run in observational mode and generate logs, but do NOT block pipeline exit codes or fail Quality Gate policies.
- Review automated root cause tags: TIMING_RACE_CONDITION, DOM_SELECTOR_DRIFT, or BACKEND_UNAVAILABLE.

3. Enterprise Security & Administration Triage

3.1 Issue: `CMEK_DECRYPTION_ERROR: Unable to unwrap ciphertext`

Symptom: Vault secrets or database records fail to load with encryption errors.
Diagnosis:
1. Check if the Customer-Managed Key in AWS KMS or HashiCorp Vault was disabled or rotated without re-encrypting data keys.
2. Check IAM role permissions for kms:Decrypt on the CertiQA backend ARN.
Resolution:
- Use CertiQA’s Zero-Downtime Key Rotation API (POST /api/v1/admin/enterprise/keys/rotate) to re-wrap all tenant secrets under the new master key version.

3.2 Issue: `BYOK_QUOTA_EXCEEDED: Credit limit reached`

Symptom: AI generation or LLM judge calls return HTTP 429 with quota error.
Diagnosis: Check the Usage Meter on the top navigation bar.
Resolution:
- Navigate to Settings → BYOK Routing and verify external API key credentials (OpenAI, Anthropic, Gemini, Grok).
- When BYOK is enabled, LLM evaluation costs route directly through customer enterprise accounts with zero platform surcharge.

3.3 Issue: CI/CD Quality Gate Rejection

Symptom: Pipeline fails with exit code 1 during certiqa-agent-runner run or GitHub Actions workflow.
Diagnosis:
- Check Quality Gate Policy configured in CertiQA (e.g. Fail if Pass Rate < 95% or Fail on any Critical Security Vulnerability).
Resolution:
- Inspect the CTRF JSON summary report generated in the artifacts directory to identify the exact failing test cases or security fuzzer findings.