Align Evals: Calibrating LLM Evaluation to Human Preference in LangSmith

In the evolving landscape of large language model (LLM) application development, accurate and reliable evaluation is paramount. Developers frequently iterate on their applications, refining prompts, updating logic, or altering architecture. Evaluations serve as a critical tool to score outputs and gauge the impact of these changes. However, a persistent challenge highlighted by development teams is a notable discrepancy between automated evaluation scores and human judgment. This misalignment can lead to unreliable comparisons and misdirected development efforts.

To address this issue, LangSmith has introduced Align Evals, a new feature designed to calibrate LLM-as-a-judge evaluators to better reflect human preferences. This innovation draws inspiration from insights into building effective LLM-based evaluation systems. Align Evals is currently available to all LangSmith Cloud users, with a self-hosted version slated for release later this week.

Traditionally, refining LLM-as-a-judge evaluators has often involved a degree of guesswork. Identifying patterns or inconsistencies in an evaluator’s behavior, and understanding precisely why scores shift after prompt modifications, has been a complex task. The new LLM-as-a-Judge Alignment feature aims to streamline this process by providing developers with enhanced tools for iteration and analysis.

Key functionalities of Align Evals include:

• Interactive Prompt Iteration: A playground-like interface allows developers to refine their evaluator prompts and instantly view an “alignment score,” indicating how closely the LLM’s assessments match human benchmarks.

• Side-by-Side Comparison: The feature enables a direct comparison between human-graded data and LLM-generated scores. This view can be sorted to quickly identify “unaligned” cases where the LLM’s judgment diverges significantly from human expectations.

• Baseline Tracking: Developers can save a baseline alignment score, facilitating a clear comparison between their latest prompt changes and previous versions.

The alignment process within Align Evals follows a structured four-step workflow:

1. Define Evaluation Criteria: The initial step involves establishing precise evaluation criteria that reflect the application’s desired performance. For instance, in a chat application, criteria might include correctness and conciseness, recognizing that a technically accurate but overly verbose response can still be unsatisfactory to users.

2. Curate Human Review Data: Developers select a representative set of examples from their application’s outputs for human review. This dataset should encompass a range of scenarios, including both high-quality and suboptimal responses, to adequately cover the spectrum of outputs the application might generate.

3. Establish Golden Set Scores: For each defined evaluation criterion, human reviewers manually assign scores to the curated examples. These human-assigned scores form a “golden set,” serving as the benchmark against which the LLM evaluator’s performance will be measured.

4. Iterate and Align Evaluator Prompt: An initial prompt is crafted for the LLM evaluator. This prompt is then tested against the human-graded examples. The alignment results provide feedback, guiding an iterative refinement process. For example, if the LLM consistently over-scores certain responses, the prompt can be adjusted to include clearer negative criteria. This iterative approach is crucial for improving the evaluator’s alignment score.

Looking ahead, LangSmith plans to further enhance evaluation capabilities. Future developments are expected to include analytics tools to track evaluator performance over time, providing deeper insights into their evolution. Additionally, the platform aims to introduce automatic prompt optimization, where the system can generate prompt variations to further improve alignment.