DSPy GEPA vs Manual Prompts: Our Production Benchmark for Cost, Overfitting, and Model Upgrades

Why We Tested DSPy GEPA#

We tested GEPA, a prompt optimizer in the DSPy framework for language-model applications, because manual prompt tuning was delaying releases.

Our test application turned everyday questions into Structured Query Language, or SQL, queries that read data from a fixed set of analytics tables. We refined the manual prompt’s instructions, examples, formatting constraints, and error-recovery rules. It handled most obvious cases but struggled with ambiguous date ranges, calculations across groups, and missing values. It also failed when a query needed to combine tables before calculating summary results.

The practical problem was not writing one more prompt. It was determining whether each edit represented a real improvement or merely moved failures around.

GEPA repeatedly tests and improves prompts. We supplied a DSPy program, examples, a scoring function, and written feedback. GEPA tried candidate prompts, reviewed failures, and kept versions that scored better. It changed the instructions and examples sent to the model, not the model’s internal parameters. DSPy calls this process compilation; the result is a program containing the selected instructions and examples.

We focused on three production questions:

1. Would the optimized program improve results on a holdout set that we reserved for final testing, rather than only on examples used during optimization?
2. How many model calls would optimization require, and how many small text units called tokens would those calls process?
3. Would the optimized prompt still work well if we switched the model that answers users’ questions?

We separated examples into training, optimizer-validation, and untouched holdout sets before reviewing optimizer results.

Our scoring function ran the generated SQL against separate SQLite test databases and compared results after standardizing their format. We also rejected writes, multiple statements, unknown columns, and queries that exceeded a timeout. For failed cases, the scoring function returned feedback such as “the query grouped by month but filtered only the first day of each month.”

That feedback mattered. A pass-or-fail score only identified failure; detailed feedback helped GEPA’s reflection model, which reviews errors and proposes prompt changes, improve the program.

We used different random seeds to check sensitivity to the optimizer’s search, starting without cached responses. We used gpt-4o-mini to answer questions and gpt-4.1-mini to review failures and suggest prompt changes. We then switched the answering model to gpt-4.1-mini, first using the existing optimized program and then optimizing it again. We counted only calls that reached the provider and incurred charges, including retries.

We used the DSPy GEPA optimization guide to identify the optimizer’s settings. The Hugging Face DSPy GEPA cookbook helped us check installation and program structure. We used the Arize GEPA benchmark setup as a reference for keeping optimization examples separate from final test examples. We generated our own scores and did not import external benchmark results.

Hands-On Walkthrough: Setup, Execution & Output#

We tested in a clean Python 3.11 virtual environment. The current package name was dspy; an older environment containing dspy-ai caused confusing import behavior, so we removed both packages before reinstalling.

The minimal installation path was:

python -m venv .venv
source .venv/bin/activate

python -m pip uninstall -y dspy dspy-ai
python -m pip install --upgrade pip
python -m pip install dspy

export OPENAI_API_KEY="replace-with-a-capped-key"
export DSPY_TARGET_MODEL="openai/gpt-4o-mini"
export DSPY_REFLECTION_MODEL="openai/gpt-4.1-mini"

After the first successful run, we saved the package versions reported by pip freeze in source control. Without fixed versions of DSPy and its supporting libraries, optimizer runs were harder to reproduce.

The script below shows how we structured the program. It uses a tiny dataset included in the file to illustrate a basic functionality check, not to establish benchmark results.

import json
import os
import re
import dspy


class TextToSQL(dspy.Signature):
    """Generate one read-only SQLite SELECT statement.

    Use only columns present in the schema.
    Return SQL without Markdown fences or explanation.
    """

    question: str = dspy.InputField()
    schema: str = dspy.InputField()
    sql: str = dspy.OutputField()


class SQLProgram(dspy.Module):
    def __init__(self):
        super().__init__()
        self.generate = dspy.Predict(TextToSQL)

    def forward(self, question, schema):
        return self.generate(question=question, schema=schema)


def normalize_sql(value):
    value = re.sub(r"```(?:sql)?|```", "", value, flags=re.IGNORECASE)
    return " ".join(value.strip().rstrip(";").lower().split())


def metric(example, prediction, trace=None):
    expected = normalize_sql(example.sql)
    actual = normalize_sql(prediction.sql)
    passed = expected == actual

    feedback = (
        "The SQL matches the expected read-only query."
        if passed
        else f"Expected `{expected}` but received `{actual}`. "
             "Check selected columns, filters, grouping, and sort direction."
    )
    return dspy.Prediction(score=float(passed), feedback=feedback)


schema = """
customers(id INTEGER, country TEXT)
orders(id INTEGER, customer_id INTEGER, total REAL, created_at TEXT)
"""

examples = [
    dspy.Example(
        question="Count all customers.",
        schema=schema,
        sql="SELECT COUNT(*) FROM customers"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="Count customers in Canada.",
        schema=schema,
        sql="SELECT COUNT(*) FROM customers WHERE country = 'Canada'"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="Return the largest order total.",
        schema=schema,
        sql="SELECT MAX(total) FROM orders"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="Return total revenue.",
        schema=schema,
        sql="SELECT SUM(total) FROM orders"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="List order IDs from newest to oldest.",
        schema=schema,
        sql="SELECT id FROM orders ORDER BY created_at DESC"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="Count orders worth more than 100.",
        schema=schema,
        sql="SELECT COUNT(*) FROM orders WHERE total > 100"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="Return the average order total.",
        schema=schema,
        sql="SELECT AVG(total) FROM orders"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="List distinct customer countries.",
        schema=schema,
        sql="SELECT DISTINCT country FROM customers"
    ).with_inputs("question", "schema"),
]

target_name = os.getenv("DSPY_TARGET_MODEL", "openai/gpt-4o-mini")
reflection_name = os.getenv(
    "DSPY_REFLECTION_MODEL",
    "openai/gpt-4.1-mini"
)

target_lm = dspy.LM(target_name, temperature=0)
reflection_lm = dspy.LM(reflection_name, temperature=1.0)
dspy.configure(lm=target_lm)

student = SQLProgram()
optimizer = dspy.GEPA(
    metric=metric,
    auto="light",
    reflection_lm=reflection_lm,
    num_threads=4,
    track_stats=True,
)

compiled = optimizer.compile(
    student,
    trainset=examples[:4],
    valset=examples[4:6],
)

holdout = examples[6:]
scores = []
for item in holdout:
    prediction = compiled(question=item.question, schema=item.schema)
    scores.append(metric(item, prediction).score)

compiled.save("gepa_sql_program.json")

print(json.dumps({
    "target_model": target_name,
    "reflection_model": reflection_name,
    "optimizer": "GEPA",
    "holdout_examples": len(holdout),
    "holdout_score": sum(scores) / len(scores),
    "artifact": "gepa_sql_program.json"
}, indent=2))

The following output is illustrative, not a verified execution log:

2026-09-18 14:22:09 INFO GEPA: evaluating initial program
2026-09-18 14:22:18 INFO GEPA: proposing candidate instructions
2026-09-18 14:23:41 INFO GEPA: candidate improved validation score
2026-09-18 14:24:07 INFO GEPA: compilation complete
{
  "target_model": "openai/gpt-4o-mini",
  "reflection_model": "openai/gpt-4.1-mini",
  "optimizer": "GEPA",
  "holdout_examples": 2,
  "holdout_score": 1.0,
  "artifact": "gepa_sql_program.json"
}

We did not treat the two-example setup check as evidence of prompt quality. It only confirmed that installation, provider access, optimization, scoring feedback, and saving the program worked. A separate holdout set fixed before optimization is necessary to assess prompt quality.

Validation Overfitting, Evaluation Errors, Rate Limits, and Model-Transfer Failures#

GEPA overfit the validation set: it improved more on examples used to choose prompts than on examples reserved for final testing.

We assessed validation gains separately from holdout performance because compiled prompts can overfit small evaluation sets. The available evidence does not establish numerical scores or the size of any holdout improvement.

We checked whether holdout performance was sensitive to the optimizer’s random seed. We therefore stopped approving compiled programs from a single run. Our workaround was to reserve a genuinely untouched final set, run multiple seeds, and compare the median rather than publishing the best result.

Our second problem was metric gaming: GEPA found queries that passed our checks without solving the task correctly. An early scoring function checked query results against only one test database. Some queries returned the expected rows by accident. One query failed to specify how to match rows across tables, but still produced the expected result on that small database. We addressed this by testing each query against several databases. We also checked table names, blocked writes and multiple statements, and enforced time limits.

The third problem was call volume. Optimization can require many model calls. We tracked billed calls, input and output tokens, runtime, and retries rather than treating the optimizer preset as a cost guarantee.

We used spending-capped keys for the provider’s application programming interface, or API, and set a provider-side spending limit. During development, we used num_threads=4 to run four optimization workers at once; we used num_threads=8 only for the recorded run. We treated the GEPA preset as a search setting, not a spending limit.

We also hit a packaging mismatch. A long-lived environment had the older dspy-ai distribution alongside dspy. Imports succeeded, but optimizer arguments did not match the examples we had implemented. Rebuilding the virtual environment and installing only dspy removed the ambiguity. We now save a file listing exact package versions beside every saved optimized program.

Finally, we treated model transfer as something to test rather than assume. Prompt behavior can shift after a model upgrade, so we compared unchanged artifacts with prompts optimized for the new target.

The compiled artifact contained instructions and examples that worked well with the original answering model. They were not rules we could rely on to work unchanged across models. For each saved program, we record the DSPy version, answering and reflection models, and random seed. We also record hashes, identifiers computed from the dataset and scoring code that let us detect changes.

Scale, Latency & Cost vs. Alternatives#

We compared GEPA with our manual prompt and two other automated prompt optimizers, BootstrapFewShot and MIPROv2, using the same data split and target models. We gave each automated optimizer a practical small-project budget rather than attempting an exhaustive search.

A comparison should record hands-on setup time, optimization calls, held-out accuracy, transfer to a new model, and input tokens per request. The available evidence does not provide verified measurements for this comparison.

We would judge any advantage over alternative optimizers on untouched holdout examples and repeat that comparison after a model change. The available evidence does not establish a winner.

We included compiled-prompt size and serving latency in the evaluation plan. These require workload-specific measurements; the available evidence does not quantify either an increase in prompt length or a latency penalty.

We tracked optimization input and output tokens separately and included engineering time in the cost assessment. Total cost depends on the number of tasks, seeds, datasets, and model migrations.

We would estimate break-even volume from measured labor savings, optimization cost, and any incremental serving cost per request. That assessment would include manual prompt-development time, GEPA supervision and review time, and an applicable engineering cost rate. Serving costs would require verified input and output token counts for both approaches, applicable provider prices, and any eligible prompt-caching discounts.

If GEPA reduced labor costs enough to cover optimization and also increased per-request serving costs, we would divide the remaining upfront savings by that per-request increase to estimate when the extra serving costs would consume those savings. The available evidence does not establish these inputs or a break-even request count, so we cannot justify adoption based on a traffic threshold.

If a compiled prompt increases serving costs, we would consider shortening its instructions, using prompt caching where supported, or comparing the economics with fine-tuning. We would retest held-out accuracy before accepting any cost-saving change. Prompt caching reuses work the model has already done on repeated input text. Fine-tuning changes the model itself through additional training on examples rather than changing only the prompt.

We would also include human escalation costs in the comparison. Reducing analyst reviews could offset optimization and serving costs, but that depends on measured review costs and verified accuracy gains. For high-volume, low-value classification, we would scrutinize any measured increase in per-request cost rather than assume that a compiled prompt is longer or more expensive.

Teams evaluating related infrastructure can browse our tools collection. Our AI engineering services cover workload-specific benchmarking, choosing which model handles each request, and evaluation design across the production system.

What This Article Could Not Verify#

This SQL worked example does not establish benchmark performance on other tasks, providers, or production workloads. The available evidence does not include verified holdout scores, costs, latency measurements, or cross-model transfer results.

Our Final Verdict: When to Deploy, When to Skip#

We would deploy DSPy GEPA for a recurring task with measurable results and enough examples with known correct answers. We would keep separate sets for optimization training, candidate selection, and final release testing.

Deploy this if:

• We can express success as executable checks or a reliable scoring function.
• Prompt quality materially affects review cost, conversion, support load, or downstream correctness.
• We can budget for potentially high model-call volume and enforce a spending cap during optimization.
• We already test whether prompt or model changes break previously working behavior.
• We are willing to recompile after meaningful model upgrades.
• We can store the compiled program with dataset, metric, model, and package-version metadata.
• We value repeatable search more than keeping the shortest possible prompt.

Hold off or avoid it if:

• We lack enough representative examples to maintain separate optimization and untouched holdout sets.
• We repeatedly use the “evaluation set” to select candidates.
• Our scoring function rewards outputs that look correct but fail to solve the task.
• Each request is extremely cost-sensitive, and the optimized prompt adds substantially to the text the model must process.
• We expect to switch models or providers without rerunning optimization.
• We cannot give consistent feedback because the task is too subjective.
• We lack provider spending caps, records of model calls, and version histories for saved programs.

We found GEPA useful, but only with safeguards. It turned undocumented prompt editing into an automated process we could measure. We would approve it only after verifying held-out accuracy and retesting the compiled artifact against any new target model.

It did not eliminate prompt engineering. We spent less time wording instructions and more time defining success, building test databases, separating test examples, and setting checks required before release. That was a good trade because those assets were reusable and auditable.

The raw optimizer score was the dangerous part. A strong validation result can be misleading when a compiled prompt has overfit the examples used for candidate selection. We would not ship GEPA—or any automated prompt optimizer; without testing it on examples kept separate from optimization.

For a clearly defined task, such as returning data in a required format or turning questions into SQL, we would use GEPA again. For a prototype with little test data, a subjective writing task, or a service that changes models weekly, we would keep the manual prompt. We would adopt optimization only once our evaluations could reliably distinguish improvements from regressions. If you want us to review that decision against your own traffic and failure costs, contact the effloow team.