AIsaAIsaStudio English 打开 AIsa Studio

Airbnb 如何用 cost per task 指标在 Amazon Bedrock 上运营编码智能体

原标题:How Airbnb uses cost per task to operate coding agents on Amazon Bedrock

Airbnb 工程师与 AWS 撰文介绍 Airbnb 如何用 cost per task 指标在 Amazon Bedrock 上选择和运营编码智能体,模型发布当天即可发起评测,次日早上得到全部任务套件的得分和每任务成本。上线后生产数据显示 model A 每用户周均开销是前一模型的数倍;将其重度用户切换到更新、得分更高的模型后,该群体开销降至约七分之一,请求数基本不变。文中还介绍 CloudWatch 运行时指标如 OTPS 与 EstimatedTPMQuotaUsage、配额 burndown 机制、花费归因和向等效模型再平衡的做法。

为什么值得看

Airbnb 给出 cost per task 的完整做法,包括评测搭建、生产监控与限额调优,方法可迁移到自己的编码智能体运营。

阅读原文 ↗11月19日 AI 日报

本页的摘要、推荐理由与解读出自 AIsa Studio 的 AI 风向智能体,原文版权归原作者所有。

深度解读

发生了什么

AWS Machine Learning 发布了一篇文章,介绍 Airbnb 如何在 Amazon Bedrock 上运营其内部编码智能体。Airbnb 每周有数千名工程师及工程相关岗位人员使用内部工具 AirChat 运行编码智能体,智能体请求经 Airbnb 的 LLM 网关发往 Amazon Bedrock,Bedrock 在 AWS 内部处理其代码、提示词和模型响应。

文章重点说明了 Airbnb 如何用 cost per task(单任务成本)指标配合评测分数来选型和监控模型。「八周内出了八个新模型」——原文表述为约五周内出现八个新模型,到十月中旬,工程师经编码智能体发送的请求中约三分之二流向了这些新模型,这推动团队建立了可复用的评测与监控流程。

关键细节

  • cost per task 定义为智能体完成一个任务所消耗全部 token 的标价成本。评测为每个模型和每个允许的推理努力(reasoning effort)档位同时记录分数和单任务成本,用这些结果设置默认模型和努力上限(effort ceiling),由网关强制执行。
  • 在测试的一个模型上,从低档调到高档推理努力,分数提高 0.8 分,而单任务成本变为原来的 2.5 倍。
  • Airbnb 有七个任务套件,任务来自真实代码变更、评审意见以及工单中未明确写出的决策。每个任务是一个文件夹,包含指令、时限与资源配置、沙箱 Dockerfile、测试和评分器。发布当天一个请求即可启动评测活动,跨数百个任务并行运行;首批结果数小时内到达,次日凌晨即可拿到全部套件的分数与单任务成本,足以做全组织范围的部署决策。
  • 长任务分多轮运行,轮次之间有模拟评审者提出质疑;模型评分器按八个加权维度打分(交付正确性 25、验证质量 15、自主性 15、设计与工作量匹配 10、判断力 10、证据与汇报 10、可读性 10、运维纪律 5)。若评审者反复「救场」,总分封顶为 25/100。
  • 生产监控使用 CloudWatch AWS/Bedrock 命名空间中的指标:延迟(InvocationLatency、TimeToFirstToken、OutputTokenCount、推导出的 OTPS)、限流(InvocationThrottles、EstimatedTPMQuotaUsage)以及按未缓存输入、缓存读、缓存写、输出四类计价的成本。
  • 一次延迟事故中,延迟上升 40% 而 TTFT 和 OTPS 保持平稳;原因是有人修改了系统提示词要求更详尽的回答,单请求输出 token 上升 50%。结论是「把提示词变更和模型升级当作部署对待」,在仪表盘上标记。
  • 模型 A 在评测中比此前最优模型高出三分以上、单任务成本相近,上线数周后人均周支出却是前一个模型的数倍:单请求成本约为三倍,人均请求数也约为三倍。输出 token 只占请求成本的十分之一,大部分成本来自每轮请求都重新读取的缓存上下文——因为该模型能长时间自主工作,用户开始与之进行长会话。「评测衡量的是模型在任务上的表现,生产衡量的是人们选择如何与它协作。」
  • 之后一个更新的模型在同一套任务和评分器上分数更高,单任务成本约为模型 A 的六分之一。通过一次配置变更将模型 A 的重度用户迁移过去,第一个完整周内该组请求数大致相同,支出降至约七分之一。
  • 限流方面:Bedrock 在请求准入时预留未缓存输入、缓存写和 max_tokens;缓存读不计入配额;输出按 burndown 率结算,多数模型为 1:1,部分模型为数倍。一个例子中仪表盘显示配额用量约 30%,按 5:1 burndown 率计算实际消耗约 77%。将 max_tokens 从 4096 降到 1024 后,初始预留缩小约 65%,为准入腾出空间。
  • 为应对容量问题,Airbnb 为每个主力模型保留至少一个在自家评测中表现相当、可负载均衡的等价模型(可来自其他模型家族)。Legacy 模型在闲置 15 天后老客户可能失去访问权;Bedrock 不会自动迁移流量。
  • 成本归因方面:用打了标签的 IAM 角色区分团队/用户,每个应用和模型建一个应用推理配置文件(application inference profile),用 requestMetadata 区分功能或提示词;网关为每个用户代入带缓存凭据的角色,避免触发 AWS STS 速率配额。

为什么重要

文章的核心观点是:模型发布时附带的基准图表「并不能说明模型如何处理我们工程师的工作」,所以模型选型必须基于自有任务的评测,并配上任务级成本指标。单请求成本对智能体是误导性的——一个任务包含多次请求,模型每次都重读上下文;对整个任务计价(并把低分尝试计入平均值)才能给出可用于决策的数字。

模型 A 的案例最有参考价值:评测分数好、单任务成本也可接受,仍无法预测生产支出,因为模型具备长时间自主工作能力后,用户的使用方式发生了变化。把评测分数、单任务成本与生产聚合信号(单请求成本、人均周请求数)放在一起看,几周内就能发现这种偏差,且修复只是一次配置变更,而非迁移项目。

运维层面,文章记录了一个关键陷阱:按计费 token 统计的仪表盘可能显示还有余量,但请求已在被限流,因为配额预留基于未缓存输入、缓存写和 max_tokens,而非实际计费 token。推导出的 OTPS 则是区分「模型变慢」与「回答变长」的关键信号。

对谁有影响

  • 在 Amazon Bedrock 上把编码智能体投入生产的工程团队,他们面对同样密集的模型发布节奏,需要可复用的上线、监控与回退流程。
  • 负责 LLM 网关、成本归因、配额规划与跨模型容量调配的平台团队。
  • 构建智能体评测基础设施的团队,可借鉴任务文件夹+评分器、多档推理努力扫测、加权多维度评分表的做法;文中提到开源沙箱评测框架 Harbor 可作为起步选项之一。
  • 需要按团队、应用、功能做支出归因的 FinOps 与成本规划职能。
全文翻译

以下翻译只覆盖原文的前一部分,其余内容请看下方原文。

Amazon Bedrock 通过通用 API 提供对来自多个提供商的基础模型的访问,并由 AWS 管理模型基础设施。它与 AWS 访问控制和监控集成,并在 AWS 内处理提示、代码和模型响应。

随着编码代理进入生产环境,模型和使用情况的变化可能会影响延迟、容量和成本。Amazon CloudWatch 中的 Amazon Bedrock 运行时指标帮助团队调查延迟增加、限流以及令牌消耗的变化。评估(evals)通过评估代理在固定的一组代表性任务上的工作质量,补充了这些运营指标。

在这篇文章中,我们展示了 Airbnb 如何为编码代理评估模型并监控其在 Amazon Bedrock 上的使用情况。我们解释了每个任务的成本如何指导模型选择,以及生产数据如何帮助我们随着使用情况的变化重新审视这些选择。我们还描述了用于调查延迟、限流和支出的运行时信号。

在 Airbnb,每周有数千名工程师及工程相关人员使用 AirChat——这是我们运行编码代理的内部工具。这些代理通过我们的语言模型(LLM)网关发送其模型请求,该网关运行在 Amazon Bedrock 上。Bedrock 在 AWS 内处理 Airbnb 的代码、提示和模型响应。大约五周内发布了八个新模型。到十月中旬,工程师通过编码代理发送的请求中约有三分之二发往这些新模型。

每次发布都可能同时改变输出令牌、成本、延迟和容量,而随发布发布的基准图表并未显示模型如何处理工程师的工作。因此,我们并行运行两个系统。在部署之前,我们在自己的任务上运行评估,以测量每个模型的得分和每个任务的成本。每个任务的成本是代理完成一个任务所使用的所有令牌的列表价格成本。

部署后,可观测性向我们展示生产中发生了什么变化。当生产使用情况与评估结果不符时,我们找出原因。

解决方案概述

您可以使用 CloudWatch 中 AWS/Bedrock 命名空间下的几个信号来诊断大多数生产问题:

  • 延迟:InvocationLatency(调用延迟)、TimeToFirstToken(首个令牌时间)、OutputTokenCount(输出令牌数),以及您推导出的每秒输出令牌数(OTPS)。
  • 限流:InvocationThrottles(调用限流次数),加上每个请求针对您的配额保留的令牌数(由您自行发布)。Bedrock 还发布 EstimatedTPMQuotaUsage(估计每分钟令牌配额使用量),该指标衡量您在所有推理 API(Converse、InvokeModel、ConverseStream 和 InvokeModelWithResponseStream)上的估计每分钟令牌(TPM)配额消耗,包括缓存写入令牌和输出消耗乘数。您可以使用此指标在达到配额限制之前设置主动警报,跨模型跟踪配额消耗,并在用量被速率限制之前申请进一步的配额增加。
  • 成本:InputTokenCount(未缓存输入)、CacheReadInputTokenCount(缓存读取输入令牌数)、CacheWriteInputTokenCount(缓存写入输入令牌数)和 OutputTokenCount(输出令牌数),每项均按其各自的价格计费。缓存读取的成本低于未缓存输入,而缓存写入的成本可能更高,具体取决于模型。较低的缓存命中率会增加每次请求的成本。

在这些指标之上,我们添加了每个任务的成本。我们的评估为允许推理的每种模型和每种推理力度测量得分和每个任务的成本。推理力度是一个控制模型在响应前进行多少推理的设置。我们使用这些结果来选择每个默认值(即除非工程师选择其他选项,否则编码工具使用的模型和力度),以及每个力度上限(即为某个模型允许的最高力度)。

架构

工程师在 AirDev 工作区上运行 AirChat,AirDev 工作区是云中的个人开发机器。他们还可以将任务交给 AirChat Remote,后者在工程师做其他事情时在单独的云工作区中运行云代理。两条路径都通过我们的 LLM 网关进入 Amazon Bedrock。

图 1 显示了 AirChat 和 AirChat Remote 如何通过我们的网关到达 Amazon Bedrock。它还显示了评估和生产监控如何指导模型默认值和力度上限。

AirChat 和 AirChat Remote 通过 Airbnb 的网关到达 Amazon Bedrock,CloudWatch 指标和评估结果设定模型默认值和力度上限

图 1:编码代理和云代理通过我们的 LLM 网关调用 Amazon Bedrock。我们的使用记录和 CloudWatch 指标显示实际使用情况,我们使用评估结果来设定默认值和力度上限

网关记录每个请求的模型、客户端、团队和用户。我们仅在聚合层面使用这些记录,用于成本和容量规划。CloudWatch 提供令牌计数、延迟、错误和限流信息。我们将这些生产信号与我们的评估得分和每个任务的成本进行比较,以决定何时需要更改默认值或力度上限。

前提条件

在 AWS 方面,您需要:

  • 对您计划运行和评估的 Amazon Bedrock 中的模型具有访问权限。有关按 AWS 区域划分的模型可用性,请参阅 Amazon Bedrock 用户指南中的 Regional availability by models。
  • 通过 ConverseStream 或 InvokeModelWithResponseStream 进行流式调用。Bedrock 仅针对流式 API 发布 TimeToFirstToken。
  • 模型调用日志记录,在您调用的每个 AWS 区域中开启。默认情况下它是关闭的。日志可以包含完整的提示和响应以及调用者的身份,因此请限制谁可以读取它们并设置保留期。

为您的调用者标记 IAM 角色,并激活成本分配标签。在开始收集之前激活它们,因为激活不会回填之前的成本。

您还需要自己准备三样东西:

  • 一套来自您自身工程工作的任务。为每个任务提供一个起始仓库状态、工程师会编写的提示和一个您信任的评分器。评分器是对代理工作进行评分的程序或模型。我们从真实的代码变更、对这些变更的审查意见以及原始工单未明确说明的决策中构建我们的任务。第 2 步展示了一个示例。
  • Bedrock 前面的网关或代理,用于记录每个请求的模型、客户端、团队、用户和令牌计数。
  • 您运行的每个模型的公开列表价格表,每条记录都有日期,以便您可以基于相同的基础对评估令牌和生产令牌进行定价。

在部署之前,定义代理可以访问的内容以及哪些操作需要人工审查。在隔离的工作区中运行代码,并在合并之前测试生成的更改。Amazon Bedrock Guardrails 可以帮助过滤提示和响应中的有害内容和敏感信息。对于基于提供的参考材料的响应,contextual grounding checks 可以标记不受支持的主张。在您自己的任务上测试这些控制措施,并将它们的延迟和成本纳入您的测量中。

逐步指南

首先测量完成一个任务的总成本。然后利用这些结果来选择模型默认值、检查生产行为、管理容量和归因支出。

步骤 1:完成一个任务的成本是多少?

一个编码代理需要多次请求才能完成一个任务,并且在每次请求中,模型都会重新阅读到目前为止的上下文。因此,我们对整个任务进行定价。

Bedrock 为每个请求报告四种类型的令牌:未缓存输入、缓存读取、缓存写入和输出。在评估运行中,我们为每个请求的所有四种类型定价,并累加任务中的请求。本文中的所有成本都使用该运行日期的公开列表价格计算。我们为测试的每个模型、推理力度和代理客户端记录平均成本,旁边是其得分,平均值包括低分尝试。

推理力度既改变得分也改变成本。在我们测试的一个模型上,从低力度切换到高力度使得分提高了 0.8 分。每个任务的成本则高出 2.5 倍。我们使用这些结果来设定力度上限,网关会强制执行这些上限。

步骤 2:评估新模型的速度有多快?

在新模型发布的当天下午,向我们的代理发送单个请求便启动了一项评估活动,该活动让新模型通过我们所有的任务套件运行。套件是一组固定类型的任务,我们有七个。代理将数百个任务分散到各个套件和几个云工作区中,每个结果一旦评分器评分就会立即出现。第一批结果在几小时内到达。到第二天早上,每个套件都有了得分和每个任务的成本,我们拥有了整个组织部署决策所需的一切。

这种速度依赖于已经写好的任务。每个任务都是一个文件夹,包含一次运行所需的一切。以下是我们机器学习工作中一个精简的任务示例:

  instruction.md     告诉代理的内容
  task.toml          时间限制、CPU 和内存
  environment/       沙箱的 Dockerfile
  tests/test.sh      运行评分器并写入奖励
  tests/grader.py    检查项
  solution/          必须通过评分器的参考解决方案

instruction.md (摘录):
  我们有两个预测任务的候选模型,我们需要
  知道应该发布哪一个。
  ...
  然后回答我们必须实际决定的问题:任何一个
  候选模型值得覆盖当前在生产系统中的模型吗?

评分器在每次运行中都从冻结的数据重新计算正确答案,只有当所有检查都通过时,代理才能获得 1 的奖励。要在每个力度下测试一个模型,请在作业配置中将每个力度列为独立的代理:

job_name: model-a-effort-sweep
n_attempts: 1                # 每个力度下每个任务一次尝试
datasets:
  - path: tasks/ml           # 每个任务自带其评分器
agents:
  - name: model-a-low
    model_name: vendor/model-a
    kwargs: {reasoning_effort: low}
  - name: model-a-medium
    model_name: vendor/model-a
    kwargs: {reasoning_effort: medium}
  - name: model-a-high
    model_name: vendor/model-a
    kwargs: {reasoning_effort: high}

这两个示例使用了 Harbor 的格式,这是一个用于在沙箱中运行代理评估的开源框架。如果您刚开始起步,这是一个不错的选择。

运行评估可以进行优化。编写评分标准(评分器遵循的评分指南)是最困难的部分。我们的长任务跨越几个回合,模拟评审员在回合之间提出反对意见。模型评分器根据八个加权维度对每次运行进行评分。每个维度的得分为 0 到 3。

交付正确性             25     判断力                 10
验证质量               15     证据和报告             10
自主性                 15     可读性                 10
设计和工作量匹配度     10     运营纪律                5

上限:如果评审员不得不反复救援代理
(自主性为 0 或 0.5),总分限制为 100 分中的 25 分。

由于存在这样的上限,需要救援的代理仅靠打磨无法获得高分。

步骤 3:部署后您关注什么?

当延迟上升时,在打开支持案例之前,绘制三个信号的图表:TimeToFirstToken(生成开始前的等待时间)、OutputTokenCount(响应长度)和 OTPS(生成速度)。Bedrock 没有原生的 OTPS 指标。您可以使用指标数学推导它,正如 Amazon Bedrock User Guide 中所描述的那样:

OTPS = OutputTokenCount / (InvocationLatency - TimeToFirstToken) * 1000

在我们会话的示例中,延迟上升了 40%,而首个令牌时间和 OTPS 保持不变。有人更改了系统提示,要求更多 th

查看原文

Amazon Bedrock provides access to foundation models from multiple providers through a common API, with AWS managing the model infrastructure. It integrates with AWS access controls and monitoring, and processes prompts, code, and model responses within AWS.

As coding agents move into production, changes in models and usage can affect latency, capacity, and cost. Amazon Bedrock runtime metrics in Amazon CloudWatch help teams investigate latency increases, throttling, and changes in token consumption. Evaluations, or evals, complement these operational metrics by assessing the quality of an agent’s work on a fixed set of representative tasks.

In this post, we show how Airbnb evaluates models for coding agents and monitors their use on Amazon Bedrock. We explain how cost per task guides model selection and how production data helps us revisit those choices as usage changes. We also describe the runtime signals used to investigate latency, throttling, and spend.

At Airbnb, thousands of engineers and engineering-adjacent people use AirChat, our internal tool for running coding agents, every week. The agents send their model requests through our large language model (LLM) gateway, which runs on Amazon Bedrock. Bedrock processes Airbnb’s code, prompts, and model responses within AWS. Eight new models came out in about five weeks. By mid-October, about two thirds of the requests our engineers sent through coding agents went to those new models.

Each release can change output tokens, cost, latency and capacity at once, and the benchmark charts published with a launch don’t show how a model handles our engineers’ work. So we run two systems side by side. Before a rollout, we run evals on our own tasks to measure each model’s score and cost per task. Cost per task is the list-price cost of all the tokens an agent uses to finish one task.

After a rollout, observability shows us what changed in production. When production usage doesn’t match the eval results, we find out why.

Solution overview

You can diagnose most production problems with a few signals in the AWS/Bedrock namespace in CloudWatch:

  • Latency: InvocationLatency, TimeToFirstToken, OutputTokenCount, and output tokens per second (OTPS), which you derive.
  • Throttling: InvocationThrottles, plus the tokens each request reserves against your quota, which you publish yourself. Bedrock also publishes EstimatedTPMQuotaUsage, which measures your estimated tokens per minute (TPM) quota consumption, including cache write tokens and output burndown multipliers, across all inference APIs (Converse, InvokeModel, ConverseStream, and InvokeModelWithResponseStream). You can use this metric to set proactive alarms before reaching your quota limit, track your quota consumption across your models, and request further quota increases before usage is rate limited.
  • Cost: InputTokenCount (uncached input), CacheReadInputTokenCount, CacheWriteInputTokenCount and OutputTokenCount, each priced at its own rate. Cache reads cost less than uncached input, and cache writes can cost more depending on the model. A lower cache hit rate can increase cost per request.

On top of these metrics, we add cost per task. Our evals measure a score and a cost per task for every model and every reasoning effort we allow. Reasoning effort is a setting that controls how much a model reasons before it responds. We use the results to choose each default, which is the model and effort that our coding tools use unless an engineer picks another, and each effort ceiling, which is the highest effort we allow for a model.

Architecture

Engineers run AirChat on their AirDev workspaces, which are personal development machines in the cloud. They can also hand a task to AirChat Remote, which runs cloud agents in separate cloud workspaces while the engineer does something else. Both paths go through our LLM gateway to Amazon Bedrock.

Figure 1 shows how AirChat and AirChat Remote reach Amazon Bedrock through our gateway. It also shows how evals and production monitoring guide model defaults and effort ceilings.

AirChat and AirChat Remote reach Amazon Bedrock through Airbnb’s gateway, with CloudWatch metrics and eval results setting model defaults and effort ceilings

Figure 1: Coding agents and cloud agents call Amazon Bedrock through our LLM gateway. Our usage records and CloudWatch metrics show real use, and we use eval results to set defaults and effort ceilings

The gateway records the model, client, team, and user for each request. We use these records only in aggregate, for cost and capacity planning. CloudWatch supplies token counts, latency, errors, and throttles. We compare those production signals with our eval scores and cost per task to decide when defaults or effort ceilings need to change.

Prerequisites

On the AWS side, you need:

  • Access to the models in Amazon Bedrock you plan to run and evaluate. For model availability by AWS Region, see Regional availability by models in the Amazon Bedrock User Guide.
  • Streaming calls through ConverseStream or InvokeModelWithResponseStream. Bedrock publishes TimeToFirstToken only for the streaming APIs.
  • Model invocation logging, turned on in every AWS Region you call. It’s off by default. The logs can hold full prompts and responses along with the caller’s identity, so restrict who can read them and set a retention period.

Tagged IAM roles for your callers, with cost allocation tags activated. Activate them before you start collecting, because activation doesn’t backfill earlier costs.

You also need three things of your own:

  • A set of tasks from your own engineering work. Give each one a starting repository state, the prompt an engineer would write and a grader you trust. A grader is the program or model that scores the agent’s work. We build our tasks from real code changes, the review comments on those changes and the decisions that the original ticket didn’t spell out. Step 2 shows one.
  • A gateway or proxy in front of Bedrock that records the model, the client, the team, the user and the token counts for each request.
  • A table of public list prices for every model you run, with a date on each entry, so you can price eval tokens and production tokens on the same basis.

Before rollout, define what the agents can access and which actions need human review. Run code in isolated workspaces and test generated changes before merging. Amazon Bedrock Guardrails can help filter harmful content and sensitive information in prompts and responses. For responses based on supplied reference material, contextual grounding checks can flag unsupported claims. Test these controls on your own tasks and include their latency and cost in your measurements.

Walkthrough

Start by measuring what a complete task costs. Then use those results to choose model defaults, check production behavior, manage capacity, and attribute spend.

Step 1: What does one finished task cost you?

A coding agent takes many requests to finish one task, and on each request the model re-reads the context so far. So we price the whole task.

Bedrock reports four kinds of tokens for each request: uncached input, cache reads, cache writes and output. In an eval run we price all four for every request and add up the requests in the task. Every cost in this post uses the public list prices dated for that run. We record the average cost next to the score for every model, reasoning effort and agent client we test, and the average includes low-scoring attempts.

Reasoning effort changes both score and cost. On one model we tested, moving from low to high effort raised the score by 0.8 points. Cost per task was 2.5 times as high. We use these results to set effort ceilings, which the gateway enforces.

Step 2: How fast can you evaluate a new model?

On the afternoon a new model came out, a single request to our agents started an eval campaign, which runs the new model through all our task suites. A suite is a fixed set of tasks of one type, and we have seven. Agents spread hundreds of tasks across the suites and several cloud workspaces, and each result appeared as soon as a grader scored it. First results arrived within hours. By the next morning, every suite had a score and a cost per task, and we had what we needed for a deployment decision for the entire org.

That speed depends on tasks that are already written down. Each task is a folder that holds everything one run needs. Here is a trimmed task from our machine learning work:

honest-holdout-bakeoff/
  instruction.md     what the agent is told
  task.toml          time limit, CPUs and memory
  environment/       Dockerfile for the sandbox
  tests/test.sh      runs the grader and writes the reward
  tests/grader.py    the checks
  solution/          a reference solution that must pass the grader

instruction.md (excerpt):
  We have two candidate models for a prediction task, and we need to
  know which one to ship.
  ...
  Then answer the question we actually have to decide: is either
  candidate worth rolling out over the system already in production?

The grader recomputes the right answer from frozen data on every run, and the agent earns a reward of 1 only if every check passes. To test one model at every effort, list each effort as its own agent in a job config:

job_name: model-a-effort-sweep
n_attempts: 1                # one attempt per task at each effort
datasets:
  - path: tasks/ml           # each task brings its own grader
agents:
  - name: model-a-low
    model_name: vendor/model-a
    kwargs: {reasoning_effort: low}
  - name: model-a-medium
    model_name: vendor/model-a
    kwargs: {reasoning_effort: medium}
  - name: model-a-high
    model_name: vendor/model-a
    kwargs: {reasoning_effort: high}

Both examples use the formats of Harbor, an open-source framework for running agent evals in sandboxes. It’s one option if you’re starting out.

Running the evals can be optimized. Writing the rubric, the scoring guide that a grader follows, is the hardest part. Our long tasks run over several rounds, with a simulated reviewer pushing back between rounds. A model grader scores each run on eight weighted dimensions. Each dimension gets a score from 0 to 3.

Correctness of delivery   25     Judgment                 10
Verification quality      15     Evidence and reporting   10
Autonomy                  15     Readability              10
Design and workload fit   10     Operational discipline    5

Ceiling: if the reviewer had to rescue the agent repeatedly
(autonomy 0 or 0.5), the total is capped at 25 of 100.

Because of ceilings like this one, an agent that needed rescuing can’t score well on polish alone.

Step 3: What do you watch after a rollout?

When latency rises, chart three signals before you open a support case: TimeToFirstToken for the wait before generation starts, OutputTokenCount for response length, and OTPS for generation speed. Bedrock has no native OTPS metric. You derive it with metric math, as the Amazon Bedrock User Guide describes:

OTPS = OutputTokenCount / (InvocationLatency - TimeToFirstToken) * 1000

In the example from our session, latency rose 40 percent while time to first token and OTPS stayed flat. Someone had changed the system prompt to ask for more thorough answers, and output tokens per request rose 50 percent. The model generated at the same speed and wrote longer responses. Treat prompt changes and model upgrades as deploys, and mark them on your dashboard.

Each week we put the CloudWatch runtime metrics next to the usage records from our gateway for each model. For cost and capacity planning, we also look at usage in aggregate: cost per request and the average number of requests per user each week. A new model can move both. After every rollout, we compare them with what we expected from the eval’s cost per task.

Step 4: The evals looked good. What did production show?

One model, which we call model A, looked good in our evals. On our coding tasks, it scored more than three points above our previous best model. Cost per task was about the same, and the same grader scored every model. We rolled it out, capped effort at high, and asked people to use low or medium for most work.

A few weeks after launch, we priced model A’s production usage at list price and looked at it in aggregate. Average weekly spend per user was several times higher on model A than on the model before it. Each model A request cost about three times as much, and the average model A user sent about three times as many requests.

Output tokens made up only a tenth of a request’s cost. Most of the cost went to cached context, which the model re-read on every request. Model A could work on its own for a long time, so people started long, self-directed sessions with it.

The eval measured the model on a task. Production measured how people chose to work with it.

Later, a newer model scored higher than model A on the same tasks and grader, at about a sixth of its cost per task. We moved model A’s users to it with a configuration change.

We then checked the change in aggregate on the same user group, the heavy users of model A. In their first full week on the newer model, the group sent about the same number of requests to their main model, and the group’s spend on that model fell to about a seventh.

Step 5: Why you get throttled, and where traffic goes next

On the bedrock-runtime endpoint, Bedrock checks your tokens-per-minute quota when a request arrives, before the model writes anything. It reserves your uncached input tokens and your cache writes, plus max_tokens, the most output the request is allowed. It adjusts that reservation as the model generates and returns the unused part when the request settles. Cache reads never count against the quota, so a request can be cheap against your quota and expensive on your bill. Output tokens settle at a burndown rate that is 1:1 on most models and several times that on some current models, so check your model’s rate.

A dashboard of billed tokens can show room while requests are being throttled. In our session’s example, the quota was 1 million tokens per minute and traffic was 300 requests per minute. The dashboard showed about 30 percent of quota. After applying a 5:1 output-token burndown rate, completed requests consumed about 77 percent.

Admission uses a different calculation. Each request initially reserved 600 uncached input tokens plus a max_tokens of 4,096, for a total of 4,696 tokens. Responses were about 800 tokens at p99.

Lowering max_tokens to 1,024 cut the initial reservation to 1,624 tokens, about 65 percent smaller. The limit was still above p99. For unchanged input and output, final quota consumption stays the same. The smaller reservation leaves more room to admit requests. Monitor stopReason to catch truncated responses. Use backoff and jitter, and cap retries.

Demand can also outrun the headroom you have on any one model. You want to know where work can go before you need it, and our evals answer that. The answer can be a model from another model family. For every major model we use, we keep at least one equivalent that performs as well on our evals and that we can load-balance to. Without those evals, choosing where the work goes would be guesswork.

If headroom on a model stays tight, we can rebalance by changing a default in our configuration. First check that the equivalent model has capacity for the extra load and that your quota covers it. Keep enough traffic on it to hold your access, too: after a model enters its Legacy period, existing customers can lose access after 15 days of inactivity. A Legacy model works until end-of-life, and after that Bedrock removes it from every Region, requests to it fail, and Bedrock doesn’t move your traffic automatically. Alarm when LegacyModelInvocations rises above zero.

Step 6: Whose spend is it?

Bedrock runtime metrics are dimensioned by ModelId, so you can see which model served a request but not which team sent it. Bedrock publishes these metrics both with and without the ModelId dimension, and a request that fails before Bedrock resolves the model appears only in the series without it, so filtering by ModelId undercounts errors. Bedrock offers three ways to attribute spend:

  • Which team or user? Bedrock records the caller’s IAM identity on every request. Tag your roles, or pass session tags, to see cost by tag in AWS Cost Explorer and the AWS Cost and Usage Report (CUR 2.0).
  • Which application? Application inference profiles are model-specific, so create one tagged profile per application and model, and call the matching profile’s ARN.
  • Which feature or prompt? Pass key-value pairs in requestMetadata on Converse. Bedrock writes them to the model invocation logs and leaves them off the bill. To price them, multiply the token counts in the logs by the published rates. Neither classic CUR nor CUR 2.0 carries a per-request ID, so reconcile the two by comparing totals by model, usage type and time period.

Behind a gateway, Bedrock sees the gateway’s role, so have the gateway assume a role for each user with its own RoleSessionName or session tags. Cache those credentials for the session, because assuming a role on every request can exceed your AWS Security Token Service (AWS STS) rate quotas. Requests keep succeeding when attribution breaks, so set an alarm for when Bedrock fails to deliver invocation logs.

Our usage records let us attribute usage by model, client, team and user, which we use only in aggregate for cost and capacity planning. We compute cost per task from the same four token counts, so we can compare token mix and cost between our evals and production.

Results

We can go from a model release to a deployment decision overnight. After one rollout, average weekly spend per user on model A was several times that on the previous model. We moved its heavy users to a newer model. In their first full week, the group’s spend on its main model fell to about a seventh, with roughly the same number of requests. For every major model we use, our evals identify an equivalent we can rebalance to.

Clean up

After testing, stop the eval jobs and agent processes you started. If you enabled model invocation logging only for this walkthrough, disable it in each Region using DeleteModelInvocationLoggingConfiguration.

After callers no longer use them, delete the application inference profiles you created using DeleteInferenceProfile. Remove IAM roles, policies, and resource tags created only for this walkthrough. Deactivate cost allocation tags used only for these tests. Delete test CloudWatch log groups and S3 log objects you no longer need.

Conclusion

We run four checks every time a model ships:

  • Release day: run the new model on your own tasks at every effort, and record a score and a cost per task together.
  • Before rollout: choose the default and cap effort from those numbers. Block any effort that costs more and scores less than a lower effort.
  • First weeks: compare cost per request and average requests per user with the eval’s cost per task, pricing both from the same four token counts.
  • Before you need it: know which models score the same on your evals, from any model family, and check their capacity and your quota before you rebalance.

We chose Amazon Bedrock for access to multiple model families through one API. It lets us use our existing AWS access controls and logging while AWS manages the model infrastructure. We use evals to choose models for our work and observability to know when those choices need another look.

To go further, see these pages in the Amazon Bedrock User Guide:


About the authors

原文版权归 AWS Machine Learning 及原作者所有,本站转载仅供阅读。如需删除,请联系 support@aisa.one。