Performing optimization runs | LaunchDarkly | Documentation

Optimize from LaunchDarkly

This topic explains how to perform an optimization run by pulling its configuration from LaunchDarkly. To do this, you must create an optimization config in LaunchDarkly and then use the optimize_from_config method to perform the run. optimize_from_config pulls an optimization configuration from LaunchDarkly and uses the parameters you specify to run the optimization.

Consider doing this if:

Create an optimization config

You can create an optimization config from the LaunchDarkly UI. Here’s how:

  1. In the left sidebar, click Agents. The AgentControl menu appears.
  2. Click Agent optimization, then Create optimization. The “New optimization” page opens.
  3. Choose an agent to optimize from the Agents dropdown.
  4. Choose a mode to use to improve the agent.
  5. After you choose a mode, configure your inputs for the optimization. Each mode has different input options. To learn more, read Exploratory and expected output modes.
  6. Upload your own CSV or JSONL, or click Download sample to download a sample CV you can fill out to re-upload. LaunchDarkly inserts the data into the UI up to a max of 25 rows.

In the Evaluation section, you can set up your optimization criteria and a threshold for passing or failing. We don’t recommend using a very low threshold, as it can cause you to spend tokens on optimization runs that don’t reach the result you want. High thresholds ensure more “passes” against the input and generally lead to a better result.

There are two different forms of criteria you can use. You can use one or both. We recommend that you always use an acceptance statement.

Acceptance statements

Acceptance statements are the strongest way to create a change in the prompts. To create an acceptance statement:

  1. Click to select the Acceptance statement checkbox. A text box appears.
  2. Use natural language to describe the output you want to get.
  3. Enter a number less than one or use the slider to adjust the passing threshold. A lower number is more lenient and a higher number is more strict.

Write an acceptance statement that will guide your agent to the final output you expect. If you need to adjust the output format of your agent, you could include instructions about the expected output format. For example, if you want the response to appear in bullet points, use “The response should always be in bullet points.”

When an optimization run occurs, acceptance statements have access to things like the duration of the LLM calls, token usage, and the rationale from previous evaluation executions. Because acceptance statements have this ability, you can include statements like “make it faster” and the acceptance statement can evaluate based on that criteria.

Judges

Judges act as defensive mechanisms for your optimization. You can use them with acceptance statements.

  1. Click to select the Judge checkbox. A dropdown appears.
  2. Choose a judge to use, or create a new judge. To learn more, read Judges.
  3. Enter a number less than one or use the slider to adjust the passing threshold. A lower number is more lenient and a higher number is more strict.

Judges help ensure that your prompts continue to align with expected outputs for things like accuracy, toxicity, relevance to source material, or other criteria.

Whether you use an acceptance statement, a judge, or both, select the model you’d like to use to run these criteria. This model is for both the judges and the acceptance statements. It’s best to choose a reasoning model that can evaluate the previous results. If your agent calls tools, you’ll also want to ensure you choose a model capable of running tools.

Now, in the Models section, specify the model choices for the optimization process to use. The selections you make here are the models that the optimization process is allowed to choose and try. We recommend adding a few models here. If you’re optimizing for speed, include different sized models, because the optimization loop attempts to select models that fit the acceptance criteria.

Finally, click Save to create the optimization config.

With all of this configured, you can use optimize_from_config to perform an optimization run referencing this config.

Example optimization run with optimize_from_config

You can use optimize_from_config to perform an optimization run.

Here’s an example:

from ldai_optimization import (
    OptimizationClient,
    OptimizationFromConfigOptions,
    OptimizationResponse,
    LLMCallConfig,
    LLMCallContext,
)
from ldai import LDAIClient
from ldai.tracker import TokenUsage
from claude_agent_sdk import query, ClaudeAgentOptions
from claude_agent_sdk.types import ResultMessage

default_fallback_model = "claude-opus-4-5-20251101"

# (1)
async def handle_agent_call(
    key: str,
    config: LLMCallConfig,
    context: LLMCallContext,
    is_evaluation: bool = False,
) -> OptimizationResponse:
    model = config.model.name if config.model else default_fallback_model
    final_message = None
    async for message in query(
        prompt=context.user_input or "",
        options=ClaudeAgentOptions(
            system_prompt=config.instructions or "",
            model=model,
        ),
    ):
        final_message = message

if not isinstance(final_message, ResultMessage):
        raise ValueError(f"Unexpected final message type: {type(final_message)}")

u = final_message.usage or {}
    input_tokens = u.get("input_tokens", 0)
    output_tokens = u.get("output_tokens", 0)

return OptimizationResponse(
        output=final_message.result or "",
        usage=TokenUsage(
            total=input_tokens + output_tokens,
            input=input_tokens,
            output=output_tokens,
        ),
    )

# (2)
options = OptimizationFromConfigOptions(
    project_key="default",
    context_choices=[context_builder("user-123")],
    handle_agent_call=handle_agent_call,
    handle_judge_call=handle_agent_call,
)

# (3)
client = OptimizationClient(ld_ai_client)
result = await client.optimize_from_config("test-optimization", options)

Handlers

Section (1) sets up the agent call for the provider to handle the actual LLM invocations. These methods are intended to be provider-agnostic, so you can use your own models as long as your agent orchestrator or framework can reach them.

There are two different handlers. They are:

Configuration

Section (2) initializes options. Unlike optimize_from_options, most parameters, including judges, model choices, and evaluation settings, are pulled automatically from your LaunchDarkly configuration object.

The options you provide here are:

Output

Section (3) initializes the client and makes the optimization call. The result is an OptimizationContext containing scores, the winning model, usage data, and iteration history. To learn more about what’s included, read the Optimization quickstart.