Testing and Deployment

AI applications fail in surprising ways. Unlike traditional software with predictable bugs, AI can produce plausible-but-wrong outputs that look fine until they cause problems. Rigorous testing and careful deployment protect:

• Your users from AI mistakes
• Your reputation from embarrassing failures
• Your organization from liability
• Your resources from costly fixes

Ship fast, but ship responsibly.

AI-Specific Testing Challenges

Testing Strategies

1. Golden Dataset Testing Create a set of inputs with known-good outputs:

2. Persona Testing Test with different user types:

• Friendly user with clear request
• Confused user with vague request
• Hostile user testing boundaries
• Technical user with complex needs
• Non-native English speaker

3. Failure Mode Testing Deliberately try to break it:

• Extremely long inputs
• Empty or minimal inputs
• Conflicting instructions
• Off-topic requests
• Prompt injection attempts

4. Consistency Testing Run same input multiple times:

• Are outputs meaningfully similar?
• Does quality vary unacceptably?
• Are there occasional failures?

Agent Evaluation Frameworks

When testing AI agents (not just prompts), you need specialized evaluation approaches. Agents have additional dimensions of complexity that simple input/output testing doesn't capture.

The Agent Evaluation Matrix

Industry Benchmarks Reference

Leading AI labs evaluate agents against standardized benchmarks. Understanding these helps you design your own evaluations:

Building Your Own Evaluation

Don't copy industry benchmarks—design evaluations for YOUR use case:

Step 1: Define success dimensions What does "good" mean for your agent? Rank these by importance:

• Accuracy of final output
• Efficiency (time, tokens, API calls)
• User experience (response quality, helpfulness)
• Safety (staying in bounds, escalating correctly)

Step 2: Create golden examples For each critical task, define:

GOLDEN EXAMPLE
├── Input: [What the agent receives]
├── Expected tool calls: [Which tools, in what order]
├── Expected reasoning: [Key decision points]
├── Expected output: [What success looks like]
└── Failure modes: [What would be wrong]

Step 3: Test systematically

Happy path      → Does it work when everything is normal?
Edge cases      → How does it handle unusual inputs?
Error recovery  → What happens when tools fail?
Adversarial     → Can users manipulate or confuse it?
Consistency     → Same input 5 times = same quality?

Step 4: Measure and iterate Track metrics over time. Improvements to prompts, tools, or orchestration should show up in your evaluation scores.

Evaluation Anti-Patterns

LLM-as-Judge Pattern

Use AI to evaluate AI—with careful design:

EVALUATOR PROMPT STRUCTURE
├── Clear criteria: Exactly what to evaluate
├── Rubric: Specific scoring guidelines (1-5 scale)
├── Examples: What each score looks like
├── Output format: Structured (JSON) for parsing
└── Reasoning: Ask for explanation before score

Example rubric prompt:

Rate this customer support response on a 1-5 scale:

5: Fully addresses query, accurate, helpful, appropriate tone 4: Addresses query with minor gaps, mostly accurate 3: Partially addresses query, some inaccuracies 2: Misses key points, notable errors 1: Incorrect, unhelpful, or inappropriate

First explain your reasoning, then provide score as JSON: {"score": N}

Caution: LLM evaluators have biases (prefer verbose answers, struggle with domain expertise). Calibrate against human judgment.

Pre-Launch Checklist

Deployment Approaches

Approach 1: Shadow Mode

User request → Current system responds
            ↘ AI system also runs (not shown to user)
            → Compare outputs, evaluate AI readiness

Approach 2: Limited Rollout

Week 1: 5% of requests → AI
Week 2: 25% of requests → AI (if Week 1 OK)
Week 3: 100% of requests → AI (if Week 2 OK)

Approach 3: Human-in-the-Loop

AI generates → Human reviews → Approved outputs go live
            → Rejected outputs inform improvements

Approach 4: Internal First

Internal users test → Feedback incorporated → External users

Monitoring in Production

Iterative Improvement

After launch, establish feedback loops:

Production usage
      ↓
Collect feedback (automated + manual)
      ↓
Identify improvement opportunities
      ↓
Update prompts / configuration
      ↓
Test changes
      ↓
Deploy updates
      ↓
[Repeat]

Exercise: Create a Test Plan

Testing Template

TEST PLAN: [Application Name]

1. GOLDEN DATASET
| ID | Input | Expected Output | Pass Criteria |
|----|-------|-----------------|---------------|
| G1 | ... | ... | ... |
| G2 | ... | ... | ... |

2. PERSONA TESTS
| Persona | Scenario | Expected Behavior |
|---------|----------|-------------------|
| P1 | ... | ... |
| P2 | ... | ... |

3. FAILURE MODE TESTS
| Mode | Input | Expected Handling |
|------|-------|-------------------|
| F1 | ... | ... |
| F2 | ... | ... |

4. SUCCESS METRICS
| Metric | Target | Measurement |
|--------|--------|-------------|
| ... | ... | ... |

5. DEPLOYMENT STAGES
| Stage | Users | Duration | Success Gate |
|-------|-------|----------|--------------|
| ... | ... | ... | ... |

Common Launch Failures

Self-Check

You've built your Implementation skills. Before moving to Phase 4, complete:

Lab 5: Build an AI Assistant — Create a functional AI assistant using no-code tools

Phase 3 Deliverable: Working AI Application — Build and deploy a functional AI-powered tool that solves a real business problem