Plano Prompt Engineers (ChatGPT, OpenAI, Langchain)

🧪 Evals: One Does Not Simply Ship an Agent

Building an AI agent is easier than ever.
Knowing whether it actually works is much harder.

An agent might handle your initial test cases, but how does it perform when requirements change, edge cases appear, or responses become more open-ended?

Without a structured evaluation framework, it's difficult to measure progress or know whether your changes are actually making the agent better - or how it performs in the first place.

In this hands-on workshop, we'll stand up and explore a working eval framework using a Claude-powered shopping assistant.

Along the way, you'll learn how evaluation tasks are defined, how deterministic and LLM-as-judge graders are implemented, how results are interpreted, and how evals can guide iterative improvements.

You'll leave with a practical reference implementation and a clearer understanding of how modern AI agent evals are built.

What You'll Learn:

• Explain why evals are essential for reliable AI agent development
• Demonstrate a working eval harness using a shopping assistant example
• Teach how to create task definitions, graders, and result summaries
• Show how to improve agent performance in a repeatable way
• Add an advanced LLM-as-judge grader for open-ended evaluation

Hands On Lab: If you'd like to follow along during the workshop, please review the repository setup instructions ahead of time and clone the project with your preferred Claude environment.