原标题：Fix evaluation bias to unlock better models on Amazon Bedrock
Property Finder 在处理房地产图像质量评估时发现，96.8% 的标签来源于现有模型的预测，导致评估严重偏向。他们构建了公正的真实数据，并采用类强化学习的进化提示优化循环，在 Amazon Bedrock 上选中 Llama 4 Maverick 17B Instruct 模型。最终，模型准确率（Micro F1）从 0.55 提升至 0.80，成本降低了 86%。
该案例展示了如何识别模型评估偏见并利用进化提示优化实现模型选型和成本降低。
本页的摘要、推荐理由与解读出自 AIsa Studio 的 AI 风向智能体，原文版权归原作者所有。
房产数字平台 Property Finder 在将视觉 AI 工作负载迁移到更经济的方案时发现，其评估数据集中 96.8% 的标签直接继承自现有模型的预测，而非经人类专家验证。真实值系统性地偏向在用模型，导致任何替代模型在对比中都显得更差。最终，他们通过重建无偏评估集和迭代式提示词优化，在 Amazon Bedrock 上选用 Llama 4 Maverick 17B Instruct，实现了更高准确率，同时成本降低 86%。
这一案例的核心教训是：最难的不是找到更好的模型，而是发现评估方法本身是障碍。「当答案由你正想替换的模型书写时，模型对比就失去了效力。」很多团队用生产模型的输出当作 ground truth，形成循环评估——替代模型被 judged 的不是实际正确性，而是与在用模型行为的接近程度。
结果上，切换到 Llama 4 Maverick 17B Instruct 后每千张图片的估计成本为 2.35 美元，比此前设置降低 86%，同时推理延迟降低 40–60%。这表明在评估框架可信的前提下，开放权重模型加针对性提示词优化可以同时做到更准和更便宜。
团队也总结了失败的尝试：把水印属性与其他属性拆成两次 API 调用的「两桶法」没有带来准确率提升，反而使模型把水印叠加误认为图片边框；通用提示词在房产这种领域特异性强的任务上同样行不通。
以下翻译只覆盖原文的前一部分，其余内容请看下方原文。
本文与 Property Finder 的 Khaled Thabit、Meriem Boughrara 和 Mohamed Desouky 合作撰写。
数字房地产应用 Property Finder 的工程团队发现，其评估标签中有 96.8% 直接继承自单一模型的预测结果，而非由人类专家验证。他们的“真实值”（ground truth）系统性地偏向于现有模型，使得任何替代方案看起来都表现不佳。虽然这些标签也存在错误，但正是这种偏差阻碍了客观的模型比较。他们在将视觉 AI 工作负载迁移到 Amazon Bedrock 上更具成本效益的选项时发现了这一问题。
Property Finder 每天在中东和北非（MENA）地区处理数万张房产图片。他们的系统对每张图片评估 11 个质量属性。这些属性涵盖水印检测、房间分类、家具状态、照片质量和建筑指标。这意味着每天有数千个新的房产列表上线，平均每个列表包含 15 张图片。所有图片在上线前都需要进行质量评估。
Property Finder 需要更具成本效益的解决方案。但是，以那些有偏差的标签为衡量标准，任何替代方案都处于劣势。
最困难的部分不是找到更好的模型，而是发现他们自己的评估方法论本身就是障碍。
在本文中，我们分享了 Property Finder 如何识别评估偏差、构建无偏见的真实值，并应用进化式提示优化（evolutionary prompt optimization）在 Amazon Bedrock 上选择正确的模型，从而以更低的成本实现了更高的准确率（成本降低 86%）。我们将详细介绍方法论和迭代示例，并提供您可以应用于自己工作负载的经验教训。
下图展示了 Property Finder 和 AWS 生成式 AI 创新中心团队用于在 Amazon Bedrock 上迭代提高分类准确率的提示优化循环。
图 1：提示优化循环架构
该图表展示了迭代提示优化周期。房产图片由 Amazon Bedrock 处理，输出 JSON 格式的预测属性。在优化过程中，这些预测结果与 256 张图片的专家标注开发集进行 Micro F1 评分。随后，误差分析检查失败的图片并识别视觉模式，从而指导下一个版本的提示词。该循环从 v0 运行至 v27。AI 助手帮助精炼提示词，后续版本使用了自动化优化。
许多团队都会遇到这种情况。你想比较视觉模型，因此构建了一个评估数据集。获取标签的最常见方法是运行你当前的模型，并将其输出视为真实值。该模型已在生产中运行且无人投诉，因此标签必须足够好。
但这创造了一种循环评估，其中任何替代方案都是根据其是否与现有模型的行为相符来评判的，而不是根据实际的正确性。
Property Finder 的评估数据集正好存在这个问题。在他们标记的样本中，96.8% 直接来自现有视觉语言模型（VLM）的预测，而非独立的人工验证。在任何图片被处理之前，任何替代模型在结构上就已经处于劣势。
隐藏的错误证实了严重性。水印文本提取存在字符级别的错误，却被标记为正确，包括幻觉出的品牌名称和缺失的字符。照片分类将包含多种房间类型的图片强制归类为单一标签。
在剩余 3.2% 的人工标记样本中，标注员引入了原始规范之外的类别，例如在提示词中不存在 Office（办公室）标签的地方出现了 Office。当答案键是由你试图替换的模型编写时，模型比较就失去了有效性。
要打破这种循环，你需要拥有真正领域知识的人类专家进行独立的新标注。不参考之前的模型输出，不访问现有模型的预测结果。
Property Finder 的团队手工标注了 356 张阿联酋图片和一组独立的埃及图片。团队从代表行业细分领域的未标记生产列表中选择了这些图片。本文报告的是阿联酋图片的结果。埃及图片作为跨区域检查，涉及 230 张图片，此处未报告。首批被标注的 256 张阿联酋图片构成了用于微调的开发集。最终模型比较的评估集使用全部 356 张图片，增加了开发运行后标注的 100 张图片。356 张图片的数量大致足以检测模型之间的中等差异。单个属性中的稀有类别样本较少，因此这些属性上的较小差异可能无法被检测到。
一位战略和分析专家标注了所有图片。他们带来了关于房地产列表质量标准的专业知识。AWS 生成式 AI 创新中心团队通过专门的澄清会议进行了系统的质量审查。这些审查解决了模棱两可的案例，并保持整个数据集标注的一致性。
房地产图片经常通过门道显示多个房间，或展示重叠的质量特征。从单标签到多标签分类的转变减轻了人为歧义的主要来源。
一张通过门道显示卧室的走廊照片，本质上既是走廊也是卧室。允许使用多个标签后，你测量的是模型做对了什么，而不是因为它合理的次要预测而惩罚它。
混合方法可以补充这种方法论：不同家族的模型提供候选标签和分歧点，人类专家裁决每一个案例，这样就不会直接将任何模型输出用作标签。这将减少未来数据集扩展的标注负担。
Micro F1 作为主要指标，因为它对每个预测赋予相同的权重，这与整体标签正确性的业务目标相匹配。Macro F1 与其一起报告，因为 Micro F1 受频繁类别的主导，可能会掩盖稀有属性上的弱表现。每标签的精确率和召回率有助于识别具体的故障模式。每样本的交并比（IoU）衡量每张图片的预测标签与专家标签的重叠程度。九个单标签属性（如旋转或水印检测）每张图片只有一个正确答案。AWS 团队还使用精确匹配准确率（exact-match accuracy）对这些属性进行评分，即答案与专家标签匹配的图片比例。
有了无偏见的评估，你就可以开始改进了。如果你不能正确衡量，就无法改进。
优化遵循一个类似于强化学习的循环。可以将每个提示版本视为一个智能体，Micro F1 分数作为奖励信号，误差分析作为改进的梯度。这个想法类似于 GEPA（遗传-帕累托反思式提示演化），其中你变异提示组件，评估适应性，选择幸存者，并进行迭代。在本次合作中，AWS 团队通过误差分析精炼了大多数版本，AI 助手审查失败图片并提出更改建议。
MIPROv2 是 DSPy 库中的一个自动化提示优化框架，在 v27 的照片分类任务中补充了这一手动流程。
第一个提示（v0）使用通用的分类指令，为说明起见简化如下：
“将此图片分类为以下类别之一：客厅、厨房、卧室、浴室……”
专有模型 A 是 Property Finder 的生产基线，在 256 张图片的开发集上 Micro F1 得分为 0.55。它使用的是 v22，这是为其模型家族调优的提示。v21 之前的分数使用的是早期的单标签设置，不可直接比较。
AWS 团队在几个候选模型上测试了早期版本，并添加了结构化输出格式和特定属性的定义。
最具启发性的修复来自图片级别的误差分析。在早期的单标签版本中，模型始终将通过门道可见的房间分类为次要房间类型。AWS 生成式 AI 创新中心团队添加了明确的指令，要求基于占据画面大部分空间的房间进行分类。后来切换到多标签输出，将主房间作为一个标签，并将通过门道可见的房间作为额外标签。
区域定制带来了进一步的提升。在某些地区，列表照片带有装饰性的阿拉伯文文本叠加层，并非水印。针对这些文本的规则消除了一整类误报。
v21 提示将照片分类和房间类别切换为多标签输出，并成为每个模型的共享起始提示。使用它，Meta 的 Llama 4 Maverick 17B Instruct 在开发集上取得了 0.75 的 Micro F1 得分。此后，AWS 团队分别调优了每个模型的提示。对于此模型，v24 针对房间尺寸高估、多标签处理和输出格式进行了调整。然后 v25 是一个自动优化的版本，将关键任务前置。在 v27 中，MIPROv2 重构了照片分类任务，该任务的模型 Micro F1 达到 0.76。
最终提示 v27_PF 是在 v27 的基础上增加了 Property Finder 为渲染图、带边框或画框的图片以及走廊添加的规则。与 v27 一样，它保留了阿拉伯文文本规则和输出格式指令。每次模型调用后的标准化步骤处理剩余的格式差异。
使用 v27_PF，Llama 4 Maverick 17B Instruct 在 356 张图片的评估集上获得了 0.80 的 Micro F1 得分，而专有模型 A 在使用 v22 时得分为 0.56。评估集包括 256 张开发图片，两个模型的提示都在这些图片上调优。因此，这些分数是乐观估计。
从专有模型 A 切换到 Llama 4 Maverick 17B Instruct（各自使用其调优后的提示），开发集上的 Micro F1 从 0.55 提升到 0.81。这是一个 47% 的相对增益。仅在 Llama 4 Maverick 17B Instruct 上进行提示调优，使其从使用共享 v21 提示时的 0.75 提升到使用 v27 时的 0.81。
并非一切都很成功。2-bucket（双桶）方法将水印属性与其他属性放在两个单独的 API 调用中。它没有带来有意义的准确性提升。带框图片检测也急剧下降。如果没有在同一提示中包含水印属性，模型会将水印叠加层误认为是图片边框。
无论你在哪个模型上运行，通用提示也会失败。房地产图像具有太多的领域特异性，不适合一刀切的指令。
AWS 生成式 AI 创新中心团队在 Amazon Bedrock 上使用无偏见评估框架评估了多个模型。所有模型都从共享的 v21 提示开始，然后在开发集上针对其模型家族进行调优。专有模型 A 获得了一次调优，即 v22。Llama 4 Maverick 17B Instruct 获得了多次调优，从 v24 到 v27，加上 Property Finder 的边缘情况规则，因此它具有
This post was co-authored with Khaled Thabit, Meriem Boughrara, and Mohamed Desouky from Property Finder.
The engineering team at Property Finder, a digital real estate application, discovered that 96.8 percent of their evaluation labels had been inherited directly from a single model’s predictions rather than verified by human experts. Their ground truth was systematically skewed toward the incumbent model, making every alternative appear to underperform. The labels also had errors, but the bias was what blocked objective model comparison. They uncovered this when migrating their vision AI workload to a more cost-effective option on Amazon Bedrock.
Property Finder processes tens of thousands of property images every day across the Middle East and North Africa (MENA). Their system evaluates 11 quality attributes per image. These cover watermark detection, room classification, furnishing status, photo quality, and construction indicators. That’s thousands of new property listings with an average of 15 images each. All need quality assessment before going live.
Property Finder needed a more cost-effective option. But measured against those labels, any alternative was at a disadvantage.
The hardest part was not finding a better model. It was discovering that their evaluation methodology itself was the blocker.
In this post, we share how Property Finder identified evaluation bias, built unbiased ground truth, and applied evolutionary prompt optimization to select the right model on Amazon Bedrock, achieving higher accuracy at 86 percent lower cost. We walk through the methodology and iteration examples, with lessons you can apply to your own workloads.
The following figure illustrates the prompt optimization loop that Property Finder and the AWS Generative AI Innovation Center team used to iteratively improve classification accuracy on Amazon Bedrock.
Figure 1: Prompt optimization loop architecture
The diagram shows the iterative prompt optimization cycle. A property image is processed by Amazon Bedrock, which outputs the predicted attributes in JSON format. During optimization, these predictions are scored with Micro F1 against the 256-image expert-annotated development set. Error analysis then inspects failed images and identifies visual patterns, which inform the next prompt version. The loop ran from v0 to v27. AI assistants helped refine the prompts, and later versions used automated optimization.
Many teams encounter this scenario. You want to compare vision models, so you build an evaluation dataset. The most common way to get labels is to run your current model and treat its outputs as ground truth. The model has been running in production and nobody complained, so the labels must be good enough.
But this creates a circular evaluation where any alternative gets judged by how closely it matches the incumbent’s behavior, not by actual correctness.
Property Finder’s evaluation dataset had exactly this problem. Of their labeled samples, 96.8 percent came directly from the predictions of the existing vision language model (VLM) rather than independent human verification. Any alternative model was structurally disadvantaged before a single image was even processed.
Hidden errors confirmed the severity. Watermark text extraction had character-level mistakes marked as correct, including hallucinated brand names and missing characters. Photo classification forced single labels onto images containing multiple room types.
In the remaining 3.2 percent of human-labeled samples, annotators had introduced categories outside the original specification, like Office appearing where no such label existed in the prompt. A model comparison loses validity when the answer key was written by the model you are trying to replace.
To break the cycle, you need human experts with real domain knowledge doing fresh annotations independently. No reference to previous model outputs. No access to what the incumbent predicted.
Property Finder’s team hand-annotated 356 UAE images and a separate set of Egypt images. The team selected these from unlabeled production listings across representative industry segments. This post reports results on the UAE images. The Egypt images served as a cross-regional check on 230 images and are not reported here. The first 256 UAE images to be annotated formed the development set for tuning. The evaluation set for the final model comparison uses all 356, adding 100 images annotated after the development runs. A set of 356 images is roughly sufficient to detect moderate differences between models. Rare classes within individual attributes have fewer examples, so smaller differences on those attributes might not be detectable.
A strategy and analytics expert annotated all images. They brought specialized knowledge of real estate listing quality standards. The AWS Generative AI Innovation Center team conducted systematic quality reviews through dedicated clarification sessions. Those reviews settled ambiguous cases and kept the labeling consistent across the dataset.
Real estate images frequently show multiple rooms through doorways or display overlapping quality characteristics. The switch from single-label to multi-label classification alleviated a major source of artificial ambiguity.
A hallway photo showing a bedroom through a doorway is genuinely both. After you allow multiple labels, you measure what the model gets right rather than penalizing it for reasonable secondary predictions.
A hybrid approach could complement this methodology: models from different families surface candidate labels and disagreements, and human experts adjudicate every case, so no model output is used directly as a label. This would reduce the annotation burden for future dataset expansion.
Micro F1 served as the primary metric because it weights each prediction equally, which matches the business goal of overall label correctness. Macro F1 was reported alongside it because Micro F1 is dominated by frequent classes and can hide weak performance on rare attributes. Per-label precision and recall helped identify specific failure modes. Per-sample intersection over union (IoU) measured how closely each image’s predicted labels overlapped the expert’s labels. The nine single-label attributes, such as rotation or watermark detection, have one right answer per image. The AWS team also scored them with exact-match accuracy, the share of images where the answer matches the expert label.
With unbiased evaluation in place, you can start improving. You cannot improve what you cannot measure correctly.
The optimization follows a loop analogous to reinforcement learning. Think of each prompt version as an agent, the Micro F1 score as the reward signal, and error analysis as the gradient for improvement. The idea resembles GEPA (Genetic-Pareto reflective prompt evolution), where you mutate prompt components, evaluate fitness, select survivors, and iterate. In this engagement, the AWS team refined most versions from error analysis, with AI assistants reviewing failed images and proposing changes.
MIPROv2, an automated prompt optimization framework from the DSPy library, complemented this manual process for the photo classification task in v27.
The first prompt (v0) used generic classification instructions, simplified here for illustration:
“Classify this image into one of the following categories: Living Room, Kitchen, Bedroom, Bathroom…”
Proprietary model A, Property Finder’s production baseline, scored a Micro F1 of 0.55 on the 256-image development set. It used v22, a prompt tuned for its model family. Scores before v21 used the earlier single-label setup and aren’t directly comparable.
The AWS team tested early versions on several candidate models and added structured output formatting and attribute-specific definitions.
The most instructive fix came from image-level error analysis. In the early single-label versions, the model consistently classified rooms visible through doorways as the secondary room type. The AWS Generative AI Innovation Center team added explicit instructions to classify based on the room occupying the majority of the frame. The later switch to multi-label output kept the main room as a label and allowed rooms visible through doorways as additional labels.
Regional customization produced the next gain. In some regions, listing photos carry decorative Arabic text overlays that are not watermarks. Rules for that text removed a whole category of false positives.
The v21 prompt switched photo classification and room category to multi-label output and became the shared starting prompt for every model. With it, Llama 4 Maverick 17B Instruct from Meta scored a Micro F1 of 0.75 on the development set. From there, the AWS team tuned each model’s prompt separately. For this model, v24 targeted room size overestimation, multi-label handling, and output format. Then v25, an automatically optimized version, moved the critical tasks first. In v27, MIPROv2 restructured the photo classification task, and the model’s Micro F1 on that task reached 0.76.
The final prompt, v27_PF, is v27 plus rules that Property Finder added for renders, images with borders or frames, and corridors. Like v27, it keeps the Arabic text rules and output format instructions. A normalization step after each model call handles remaining format differences.
With v27_PF, Llama 4 Maverick 17B Instruct scored a Micro F1 of 0.80 on the 356-image evaluation set, where proprietary model A scored 0.56 with v22. The evaluation set includes the 256 development images, and both models’ prompts were tuned on those images. These scores are therefore optimistic.
The switch from proprietary model A to Llama 4 Maverick 17B Instruct, each on its own tuned prompt, lifted Micro F1 from 0.55 to 0.81 on the development set. That is a 47 percent relative gain. Prompt tuning alone, on Llama 4 Maverick 17B Instruct, raised it from 0.75 with the shared v21 prompt to 0.81 with v27.
Not everything succeeded. The 2-bucket approach put the watermark attributes and the other attributes in two separate API calls. It brought no meaningful accuracy gain. Boxed-image detection also dropped sharply. Without the watermark attributes in the same prompt, the models mistook watermark overlays for image borders.
Generic prompts also failed regardless of which model you ran them on. Real estate imagery has too much domain specificity for a one-size-fits-all instruction.
The AWS Generative AI Innovation Center team evaluated multiple models on Amazon Bedrock with the unbiased evaluation framework. All models started from the shared v21 prompt and were then tuned for their model family on the development set. Proprietary model A got one tuning round, v22. Llama 4 Maverick 17B Instruct got several, v24 to v27, plus Property Finder’s edge-case rules, so it had more tuning effort behind it. The final comparison against proprietary model A used the 356-image evaluation set. Property Finder selected Llama 4 Maverick 17B Instruct for their production workload based on accuracy across all 11 attributes and cost efficiency. For model availability by Region, refer to Supported models by AWS Region in Amazon Bedrock.
The following table shows the exact-match accuracy that Llama 4 Maverick 17B Instruct (v27_PF) reached on each single-label attribute, scored on the 356-image evaluation set. It scored 90 percent or higher on six of the nine, and matched or exceeded proprietary model A on all 11 attributes. On photo classification, which can have several labels per image, the two scored the same.
Attribute
Exact-match accuracy
Rotation detection
100%
People detection
97%
Boxed-image detection
97%
Watermark detection
94%
Render detection
93%
Furnished status
91%
Room under construction
81%
Room size
65%
Watermark text extraction
55%
Property Finder scores tens of thousands of images a day, so per-image cost matters. Llama 4 Maverick 17B Instruct is a cost-effective open-weight model, and the pipeline gets all attributes for an image in a single call.
Result
Value
Estimated cost per 1,000 images
$2.35
Cost reduction compared with Property Finder’s previous setup
86%
These figures are estimates as of December 2025 in the US East (Ohio) Region. The Llama 4 Maverick 17B Instruct figure assumes 9,000 input tokens and 200 output tokens per image, from the original two-call cost analysis. A single call with the production prompt carries about 7,500–8,000 input tokens, so this is an upper estimate. The 86 percent compares it with the cost analysis estimate for Property Finder’s previous setup, which used the v21 prompt. The accuracy comparison used v22, which was tuned from v21, so its cost might differ. For current pricing, refer to Amazon Bedrock pricing.
Property Finder runs this optimized pipeline in production, processing tens of thousands of images daily with Llama 4 Maverick 17B Instruct on Amazon Bedrock. In their own monitoring, Property Finder observed a 40–60 percent reduction in inference latency compared with their previous setup.
Start by auditing your evaluation labels. If they came from model predictions rather than human experts, you have the same bias problem Property Finder found.
Build domain-expert ground truth for a few hundred representative samples, and keep a separate test split that you never use for tuning. Adopt multi-label metrics where your classification task involves overlapping attributes. Then iterate on prompts using image error analysis, treating each iteration as a generation in an evolutionary optimization loop.
When you deploy a pipeline like this, add responsible AI controls. For example, you can use Amazon Bedrock Guardrails to enforce content policies and filter inappropriate model outputs. The attribute predictions in this pipeline don’t come with a calibrated confidence score, so use a practical signal to choose images for human review. Model disagreement is one such signal. When two models give different answers for an attribute, the image is often ambiguous. A second model on every image adds cost, so you can start with a sample. Check how often the signal catches real errors on your development set before you rely on it.
In this post, we walked through how Property Finder identified evaluation bias, built unbiased ground truth, and applied evolutionary prompt optimization to achieve better accuracy at lower cost through model selection on Amazon Bedrock.
Three elements needed to come together for this result: unbiased evaluation to measure honestly, the flexibility to evaluate and switch between model families without re-platforming, and carefully evolved prompts tailored to the domain. Amazon Bedrock provided the foundation. It gave Property Finder access to multiple model families through a single API, inherited AWS governance and security controls, and the ability to adopt new models as they become available without changing their infrastructure.
To get started with model evaluation and selection on Amazon Bedrock, visit the Amazon Bedrock documentation. If you process vision workloads at scale and want help building an unbiased evaluation framework, reach out to the AWS Generative AI Innovation Center.
Share your experiences with model evaluation and prompt optimization in the comments.
Additional contributors from AWS include Yousuf Albannai, Amine Sehili, Thomas Ounnas, Aris Tsakpinis, Rafik Soliman, Nikhil Ulhas Pandit, and Sneh Sharma.
原文版权归 AWS Machine Learning 及原作者所有，本站转载仅供阅读。如需删除，请联系 support@aisa.one。
Studio