Spec-driven development (SDD) vs evaluation-driven development (EDD) in AI software engineering
Two engineering disciplines have moved from niche terminology to the centre of AI software delivery: Spec-Driven Development (SDD) and Evaluation-Driven Development (EDD). Teams frequently ask how they differ, how they relate to traditional Test-Driven Development (TDD), and which skills engineers need to practice both in production.
The short answer is that they solve two different problems created by generative AI:
- Spec-Driven Development (SDD) governs how software is built with AI coding assistants and agents: instead of prompting an agent with vague natural-language wishes ("vibe coding"), engineers write explicit, machine-checkable specifications, contracts, and invariants first, then treat generated code as a disposable implementation artefact.
- Evaluation-Driven Development (EDD) governs how stochastic AI systems themselves are tested and improved: because an LLM, RAG pipeline, or agent does not return a single hard-coded string, engineers define benchmark datasets, deterministic assertions, and calibrated scoring rubrics before changing prompts, retrieval logic, or models.
Spec-Driven Development (SDD) constrains the deterministic boundaries of the system; Evaluation-Driven Development (EDD) measures the probabilistic quality and failure rates inside those boundaries.
What is Spec-Driven Development (SDD)? Contracts before code generation
In classical software development, code was expensive to type and specifications often rotted in static wiki pages. Coding agents and AI-assisted IDEs inverted that economics: generating five hundred lines of code takes seconds, while reviewing, debugging, and maintaining ungrounded code takes days.
Spec-Driven Development (SDD) (closely linked to specification engineering) makes the structured specification—not the ad-hoc chat prompt—the primary source of truth. Before an engineer or coding agent writes implementation code, the specification defines four explicit elements:
- Data schemas and interface contracts. Strict input/output types (OpenAPI schemas, JSON Schema, protocol buffers, database constraints) that reject malformed payloads deterministically.
- State invariants and forbidden transitions. Rules that must hold before and after every operation—for example, idempotency keys on payment mutations, permission checks, or AgentSpec runtime enforcement predicates.
- Behavioral acceptance criteria. Testable preconditions and postconditions covering normal execution, boundary limits, timeouts, and error recovery.
- Non-goals and scope boundaries. Explicit statements of what the module or agent must not modify, import, or expose.
When an AI coding agent generates or refactors code under SDD, verification is not "eyeballing the diff". The generated implementation must compile against the type contracts, satisfy property-based or unit tests derived from the specification, and respect architectural boundaries. In LLM product behaviour, the same philosophy appears in behavioural specifications such as the OpenAI Model Spec, which separates non-negotiable root rules from lower-level defaults.
What is EDD in software engineering and AI? Evaluation-Driven Development explained
When developers search for what is EDD in software engineering or AI EDD, they are referring to Evaluation-Driven Development (EDD): the practice of driving every change to a prompt, retrieval pipeline, model router, or agent workflow through repeatable, quantitative evaluation suites before and after deployment.
Traditional unit tests check binary equivalence (assert output == expected). That works for deterministic functions, but fails when an AI system summarises a clinical note, extracts fields from a complex PDF, or plans a multi-step tool trajectory. Without EDD, teams fall into "vibe-based iteration": tweaking a system prompt to fix one visible bug while silently breaking twelve other query patterns.
As emphasised in the NIST AI 600-1 Generative AI Profile, rigorous pre-deployment and continuous measurement is a foundational control against confabulation, harmful outputs, and reliability drift.
A production EDD loop requires four technical building blocks:
- Representative golden and adversarial datasets. Curated test cases drawn from real production failure modes, edge cases, multi-turn conversations, and adversarial red-teaming inputs—not synthetic happy-path examples alone.
- Tiered scorers (deterministic first, model-graded second). Fast, deterministic checks (JSON schema validity, regex constraints, citation link existence, latency, and token cost) run first. Calibrated LLM-as-a-judge rubrics or domain-specific metrics (such as RAG faithfulness evaluation) run second, always validated against human expert labels.
- Automated CI/CD regression gates. Using LLM evaluation frameworks and tools such as DeepEval or custom LLM testing harnesses, every pull request that modifies prompts, chunking, or model versions triggers an automated benchmark comparison against the baseline commit.
- Production trace feedback. Real-world failures captured via OpenTelemetry GenAI traces are triaged, anonymised, and promoted into the golden evaluation suite so the system never regresses on the same failure twice.
SDD vs EDD vs TDD: side-by-side comparison
Rather than competing methodologies, TDD, SDD, and EDD operate at complementary layers of a modern AI application stack.
| Dimension | Test-Driven Development (TDD) | Spec-Driven Development (SDD) | Evaluation-Driven Development (EDD) |
|---|---|---|---|
| Primary artefact written first | Unit or integration test case (test_calculate_invoice()). | Structured specification: schemas, state invariants, architecture & behavioral contracts. | Golden/adversarial dataset + scoring rubric + release threshold policy. |
| Target system behaviour | Deterministic functions, business logic, and state transitions. | AI-generated code, service interfaces, tool contracts, and runtime safety boundaries. | Stochastic LLM outputs, RAG retrieval quality, and multi-step agent trajectories. |
| Verification mechanism | Binary pass/fail assertion (true / false). | Type checking, contract validation, invariant testing, and runtime predicate enforcement. | Distribution of scores across test cohorts (precision, recall, faithfulness, latency, pass@k). |
| Primary failure prevented | Logic regressions in deterministic code paths. | Architectural drift, hallucinated APIs, unsafe tool side effects, "vibe-coded" tech debt. | Silent prompt regressions, retrieval degradation, hallucination spikes, cost/latency blowups. |
| Core Atlas skills required | Software testing, API design, CI/CD automation. | Specification engineering, schema validation, AI guardrails. | LLM evaluation design, DeepEval, agent evaluation, LLM observability. |
How SDD and EDD work together in one production workflow
To see why mature AI engineering teams require both disciplines, consider an enterprise team building an automated claims-triage agent:
- Define the specification (SDD). The team writes the JSON Schema for the triage decision object, defines the exact tool contracts the agent is allowed to call, and encodes AgentSpec runtime predicates guaranteeing that claims above a monetary threshold can never be auto-approved without human sign-off.
- Generate and verify the scaffolding (SDD + TDD). Engineers use coding assistants to generate the API handlers, schema validators, and retry state machines, verifying 100% compliance with deterministic unit and contract tests.
- Build the evaluation harness (EDD). Before tuning the model prompt or retrieval index, the team assembles 300 historical claims—including ambiguous edge cases and prompt-injection attempts—and defines scoring metrics in DeepEval for classification accuracy, citation grounding, and abstention on insufficient evidence.
- Iterate on prompts, models, and retrieval (EDD). When an engineer tests switching from a frontier model to a smaller distilled model to reduce latency, the EDD harness immediately reveals a 9% drop in abstention accuracy on incomplete documents—blocking an unsafe release before it reaches production.
- Close the loop in production (ISO/IEC 42001 & AI auditability). Every specification version and evaluation report is archived alongside the release manifest, giving governance and operations teams a complete, inspectable audit trail.
This workflow directly reflects what DORA software delivery research has consistently shown: speed without fast, automated feedback loops does not produce high performance—it produces instability.
What this shift means for engineering skills and job role redesign
As we examine in skills intelligence in IT and job role redesign for AI, the rise of SDD and EDD changes what distinguishes a senior engineer from a junior one. When code generation is commoditised, engineering value shifts upstream into specification clarity (SDD) and downstream into empirical evaluation and failure analysis (EDD).
When assessing your engineering workforce or hiring for AI product teams, check that your skill profiles and learning paths cover this prerequisite chain:
- Schema & contract design → prerequisite for reliable tool calling and SDD.
- Statistical thinking & experimental design → prerequisite for LLM evaluation design and judge calibration.
- Evaluation frameworks (DeepEval, LLM testing, RAG evaluation) → prerequisite for automated EDD release gates.
- Distributed tracing (OpenTelemetry) → prerequisite for turning production incidents into new evaluation test cases.
Continue exploring
- Read the Spec-Driven Development (SDD) guide in the AI Glossary
- Read the Evaluation-Driven Development (EDD) guide in the AI Glossary
- Explore LLM Evaluation Design and DeepEval skills in the Atlas
- See how AgentSpec and OpenTelemetry GenAI conventions enforce and trace agents
- Learn how evaluation evidence supports ISO/IEC 42001 and AI auditability
- Apply SDD and EDD to IT workforce planning and job role redesign