Data-driven prompt optimizer (Vertex AI)

TL;DR

The Vertex AI Data-Driven Optimizer is a batch, task-level iterative prompt optimizer that improves system instructions and few-shot demonstrations for a target Gemini model by evaluating candidate prompts on a labeled dataset against user-chosen metrics (model-based, computation-based, or custom). It runs as a Vertex AI custom training job, takes minutes-to-hours, and is the heavyweight counterpart to the Zero-Shot Optimizer. The headline use case is prompt translation/migration: take a prompt that works on model A (e.g. an older Gemini, a non-Google model) and produce an equivalent prompt that performs well on model B (e.g. Gemini Flash). Underlying algorithm is Google Research’s automatic prompt optimization (APO) work (NeurIPS 2024), where an optimizer LLM and an evaluator LLM iterate together to propose and score candidate prompts.

Key claims

Two optimization targets are supported: instructions only, demonstrations only, or both, set via optimization_mode [§Configuration].
The optimizer needs a labeled prompt dataset (5–100 examples, controlled by DATA_LIMIT); runtime scales linearly in dataset size [§Configuration].
Prompt templates use named variables that are filled from the dataset at evaluation time; non-optimized context can be pinned via PLACEHOLDER_TO_CONTENT so it isn’t rewritten [§Prompt templates].
Supports model-based, computation-based, and custom evaluation metrics in a single eval_metrics_types list [§Configuration, blog announcement].
Only generally-available Gemini models are supported as target models; for off-Vertex source models, the optimizer can run the source model itself to generate ground-truth responses [§Supported models, notebook docs].
The output is a new system-instruction string plus (optionally) a selected demonstration set, written to a Cloud Storage bucket under instruction/ or demonstration/ subfolders [§Inspect the results].

Method

The user supplies: (1) a baseline system instruction, (2) a prompt template with variables, (3) a labeled dataset (question, ctx, target-style fields), (4) a target model, (5) a list of evaluation metrics, and (6) optimization-loop hyperparameters (num_steps, num_template_eval_per_step, num_demo_set_candidates, demo_set_size). The optimizer LLM proposes candidate instructions / demonstrations; the target model runs them on the dataset; the evaluator LLM scores the outputs against the metrics; the loop keeps the best candidates and iterates. The job runs as a Vertex AI CustomJob in a Google-managed container (us-docker.pkg.dev/vertex-ai-restricted/builtin-algorithm/...). Entry points are a Colab Enterprise notebook, the Vertex AI Studio UI (with “Prompt optimizer” and “Optimize your prompt further” buttons), and the Python SDK (vertexai.Client.prompt_optimizer.optimize_prompt).

Results

No quantitative results in the doc page itself. The launch blog cites AdVon Commerce reporting “10% incremental improvements” on problematic product-attribute prompts after optimization for Gemini Flash, and Augmedix using the optimizer for ambient medical documentation. The underlying APO research (referenced as NeurIPS 2024) is where benchmark numbers live.

Why it’s interesting

This is one of Google’s productized takes on automatic prompt engineering, and it operationalizes a hypothesis the Slack note made explicit: that different Gemini tiers (Flash vs Pro) have systematically different prompt sweet spots, so prompt translation is a real engineering problem worth automating. For Luma — where prompt assembly for image/video generation, captioning pipelines, and agent loops is a recurring source of brittleness — the interesting bits are (a) the framing of prompt-optimization-as-translation-between-models (relevant when swapping VLM backbones in dense-captioning pipelines), and (b) the demonstration-set optimizer, which formalizes few-shot exemplar selection as a sub-problem. The Zero-Shot Optimizer (sibling page) is the low-latency counterpart for runtime prompt rewriting; together they suggest the right architecture is a router + zero-shot rewrite at request time, with periodic data-driven recompilation of system prompts offline.