Evaluating Structured LLM Outputs and Detecting Regressions with LLM Eval Toolkit

As generative AI transitions from experimental chat interfaces to core production application backends, software systems increasingly rely on Large Language Models to return structured data. Whether generating JSON payloads for automated booking systems, extracting invoice line items, or routing customer intents, backend services depend on strict data contracts to function correctly.

However, LLMs are fundamentally probabilistic token generators. Unlike traditional deterministic software where a unit test proves consistent input-output behavior, language model outputs are subject to subtle drift. A minor modification to a system prompt intended to improve conversational tone can unintentionally cause the model to omit a required JSON field, alter an integer ID to a string, or output Markdown formatting blocks that break JSON.parse().

The Challenge: Nondeterminism vs. API Contracts

When an LLM outputs malformed data in production, the failure cascades downstream. A frontend application waiting on an expected array crashes with a TypeError: items.map is not a function. A database insert fails because a required UUID is absent. Worse, subtle hallucinations can introduce prohibited terminology or leak sensitive instructions.

To solve this, developers need a lightweight, deterministic testing framework that can evaluate hundreds of test cases against explicit schemas, compare results against committed baselines, and fail CI/CD builds when regressions occur.

Architecture of LLM Eval Toolkit

To address these requirements, I authored and open-sourced LLM Eval Toolkit, a developer-focused, offline-first evaluation and regression testing framework written in TypeScript.

The toolkit is designed around a modular evaluator registry. Each evaluator is an isolated pure function that accepts test inputs, model completions, and assertion criteria, returning structured pass/fail results with descriptive failure rationales. The framework operates 100% offline with zero external network dependencies, ensuring tests run rapidly, deterministically, and securely without leaking proprietary prompt datasets.

Supported Evaluator Modules

The framework includes a comprehensive suite of specialized evaluators:

• JSON Schema Validation: Uses Ajv (with full draft-07 and ajv-formats support) to enforce strict schema adherence, checking required properties, nested objects, and regex formats.
• Exact Match: Performs whitespace-normalized or strict string equality verification for deterministic classification tasks.
• Required Content: Verifies that essential keywords, phrases, or identifiers are present in the response.
• Prohibited Content: Scans for forbidden phrases, confidential terms, or hallucinated patterns that must never appear in production output.
• Field Types: Asserts that individual fields extracted via dotted property paths match expected primitive or complex types (string, number, boolean, array, object).
• Regression Comparison: Computes mathematical score differences between the current test run and a version-controlled baseline snapshot, flagging score decreases.

Practical CLI Workflow

The toolkit provides an intuitive command-line interface designed for both interactive developer exploration and automated continuous integration:

# Run evaluations against a test dataset and compare with committed baseline
llm-eval run --dataset tests/fixtures/prompts.json --baseline tests/baselines/v1.json

# Create or update a baseline snapshot after verifying prompt improvements
llm-eval baseline create --dataset tests/fixtures/prompts.json --output tests/baselines/v2.json

# Output JSON summary for automated CI pipeline gating
llm-eval run --dataset tests/fixtures/prompts.json --reporter json --output results.json

Verification and Automated Test Evidence

The toolkit is verified by a rigorous Vitest test suite consisting of 23 test files and 122 passed tests. The automated tests validate evaluator accuracy, schema edge cases, baseline comparison math, security path sanitization (guarding against directory traversal), and CLI exit code compliance across Node.js 18.x, 20.x, and 22.x.

Scope and Limitations

LLM Eval Toolkit evaluates structural contracts, schema compliance, and deterministic assertions. It does not attempt to evaluate subjective qualities like humor, creativity, or emotional resonance without explicit user-defined criteria. By focusing strictly on contractual correctness and regression prevention, it provides engineering teams with a dependable, measurable testing harness.

Getting Started and Project Resources

The project is published under the MIT License. You can clone the repository, run tests locally, and adapt it to your team's workflow from the LLM Eval Toolkit GitHub Repository.

To view this tool alongside other verified engineering utilities, visit the Open-Source Projects Resource Hub. If your team is deploying LLM-backed systems and requires guidance on prompt testing, fallbacks, and architecture, explore my AI Engineering Consulting services.