7 · Evaluation

“It seemed to work” is not an engineering claim. packages/api/evals/ turns prompt and model changes into measurable ones.

poetry run python -m evals run --model gemini-3.5-flash --provider gemini
poetry run python -m evals compare --models gemini-3.5-flash,gpt-4o-mini

The parts

A case

- id: top_5_products_by_revenue
  question: "What are the top 5 products by revenue?"
  difficulty: medium
  tags: [join, group_by, top_n]
  sql: |
    SELECT p.name, SUM(oi.line_total) AS revenue
    FROM order_items oi
    JOIN products p ON p.id = oi.product_id
    GROUP BY p.name
    ORDER BY revenue DESC
    LIMIT 5

Tags are what make a score actionable. “0/9 on join” tells you what to fix; “26% accuracy” does not.

It runs the real pipeline

from src.pipeline.stages.schema_select import select_schema
from src.pipeline.stages.sql_gen import generate_sql
from src.pipeline.stages.sql_validate import validate_and_set

The same schema_select, sql_gen, sql_validate and execute that serve production traffic. An eval that exercises a parallel code path measures the wrong thing.

Two stages are deliberately skipped:

• Cache — a hit would score a model on a previous model’s output.
• Chart — deterministic, no AI involved.

The metrics

Why execution accuracy and not string match

SELECT SUM(x) / COUNT(y) FROM t
SELECT AVG(...)          FROM t
 
FROM a JOIN b ON a.id = b.a_id
FROM b JOIN a ON b.a_id = a.id

Comparison normalises floats onto a tolerance grid, so 10.0 and 10.000001 do not count as a miss:

if isinstance(value, int | float | Decimal):
    as_float = float(value)
    tol = tolerance if tolerance is not None else DEFAULT_TOLERANCE
    return round(as_float / tol) * tol      # quantise onto the grid

Numeric-looking strings are coerced too, because drivers disagree about whether a NUMERIC column comes back as Decimal, float or str.

LLM-as-judge

Execution accuracy scores the SQL. It says nothing about whether the prose answer is faithful to the rows — a model can return correct data and then describe it wrongly.

So an optional judge model scores the answer 1–5 given the question, the rows and the answer. Use it knowing the caveats:

• It is another model, with its own biases and failure modes.
• It is non-deterministic — use a low temperature and treat small differences as noise.
• It costs money and quota on every case.

Which is why it is opt-in, and why execution accuracy remains the headline. Being able to articulate when LLM-as-judge is appropriate — for subjective qualities with no programmatic ground truth — and when it is not, is the sophisticated answer.

A worked example: when the number is the harness’s fault

A run reported 26.1% execution accuracy, with every join case failing. That reads like a model that cannot write joins.

It was not:

"validation_error": "generation failed: 429 RESOURCE_EXHAUSTED ...
  quotaId: 'GenerateRequestsPerDayPerProjectPerModel-FreeTier', quotaValue: '20'"

Gemini’s free tier allows 20 requests per day. Six cases ran; the remaining seventeen never reached the model. Their SQL was never generated, let alone wrong. Each also spent ~180 seconds politely backing off before failing, so the run took three hours to produce a meaningless number.

Three fixes, each generalisable:

1. Classify failures by cause, not symptom. A daily quota is a 429 with a RetryInfo, exactly like a per-minute limit. Only the quotaId distinguishes them.
2. A skipped case is not a failed case. Cases that never reached the model are excluded from every rate and from the latency percentiles — counting them as misses reports a model-quality problem that does not exist.
3. Make partial runs impossible to misread. The first daily-quota failure aborts the run, and both reports carry a prominent warning.

@property
def scored(self) -> list[CaseResult]:
    """Cases the model actually attempted."""
    return [c for c in self.cases if not c.skipped]
 
def _rate(self, predicate) -> float:
    scored = self.scored
    if not scored:
        return 0.0
    return sum(1 for c in scored if predicate(c)) / len(scored)

Client-side pacing

class RateLimiter:
    """
    Provider free tiers are strict (Gemini allows 5 requests/minute/model). Even
    with retries, hammering the limit turns a run into mostly-backoff and the
    numbers stop reflecting model quality.
    """

Pacing requests up front keeps a run honest and finishes sooner than fighting 429s.

Why evals are not in CI

The test suite runs on every push; the eval suite does not. Deliberately:

• It costs money and quota on every run.
• It is non-deterministic — a model can produce different SQL for identical input, so a threshold-based gate would flake.
• It is slow — minutes, not seconds.

Evals are run when a prompt, model or retrieval strategy changes, and the report is read by a human. Unit tests gate merges; evals inform decisions. Knowing the difference is the point.

What a good report tells you

 METRICS
   execution accuracy  ████████████████░░░░░░░░  67.0%   <- headline
   valid SQL rate      ██████████████████████░░  91.0%

 BY DIFFICULTY
   easy     ███████████████░  95.0%  (19/20)
   medium   ██████████░░░░░░  62.0%
   hard     █████░░░░░░░░░░░  31.0%

 WEAKEST CAPABILITIES
   window_function     20.0%  (1/5)
   anti_join           25.0%  (1/4)

That last block is the useful one: it converts a score into a work item.

How you would extend this

Good answers to “what would you do next”:

1. More cases, more schemas. 23 on one schema catches regressions but cannot support cross-model claims. Cross-schema generalisation is where text-to-SQL systems actually break.
2. Regression tracking. Store each run and diff against the previous, so an accuracy drop is attributable to a specific commit.
3. Per-case cost and latency budgets, not just aggregates.
4. Ambiguity cases — questions with no single right answer, to measure how gracefully the system asks for clarification.
5. Adversarial cases — prompt injection in sample values, to prove the validator holds under pressure.