Amazon Bedrock 通过通用 API 提供对来自多个提供商的基础模型的访问,并由 AWS 管理模型基础设施。它与 AWS 访问控制和监控集成,并在 AWS 内处理提示、代码和模型响应。
随着编码代理进入生产环境,模型和使用情况的变化可能会影响延迟、容量和成本。Amazon CloudWatch 中的 Amazon Bedrock 运行时指标帮助团队调查延迟增加、限流以及令牌消耗的变化。评估(evals)通过评估代理在固定的一组代表性任务上的工作质量,补充了这些运营指标。
在这篇文章中,我们展示了 Airbnb 如何为编码代理评估模型并监控其在 Amazon Bedrock 上的使用情况。我们解释了每个任务的成本如何指导模型选择,以及生产数据如何帮助我们随着使用情况的变化重新审视这些选择。我们还描述了用于调查延迟、限流和支出的运行时信号。
在 Airbnb,每周有数千名工程师及工程相关人员使用 AirChat——这是我们运行编码代理的内部工具。这些代理通过我们的语言模型(LLM)网关发送其模型请求,该网关运行在 Amazon Bedrock 上。Bedrock 在 AWS 内处理 Airbnb 的代码、提示和模型响应。大约五周内发布了八个新模型。到十月中旬,工程师通过编码代理发送的请求中约有三分之二发往这些新模型。
每次发布都可能同时改变输出令牌、成本、延迟和容量,而随发布发布的基准图表并未显示模型如何处理工程师的工作。因此,我们并行运行两个系统。在部署之前,我们在自己的任务上运行评估,以测量每个模型的得分和每个任务的成本。每个任务的成本是代理完成一个任务所使用的所有令牌的列表价格成本。
在这些指标之上,我们添加了每个任务的成本。我们的评估为允许推理的每种模型和每种推理力度测量得分和每个任务的成本。推理力度是一个控制模型在响应前进行多少推理的设置。我们使用这些结果来选择每个默认值(即除非工程师选择其他选项,否则编码工具使用的模型和力度),以及每个力度上限(即为某个模型允许的最高力度)。
工程师在 AirDev 工作区上运行 AirChat,AirDev 工作区是云中的个人开发机器。他们还可以将任务交给 AirChat Remote,后者在工程师做其他事情时在单独的云工作区中运行云代理。两条路径都通过我们的 LLM 网关进入 Amazon Bedrock。
图 1 显示了 AirChat 和 AirChat Remote 如何通过我们的网关到达 Amazon Bedrock。它还显示了评估和生产监控如何指导模型默认值和力度上限。
图 1:编码代理和云代理通过我们的 LLM 网关调用 Amazon Bedrock。我们的使用记录和 CloudWatch 指标显示实际使用情况,我们使用评估结果来设定默认值和力度上限
网关记录每个请求的模型、客户端、团队和用户。我们仅在聚合层面使用这些记录,用于成本和容量规划。CloudWatch 提供令牌计数、延迟、错误和限流信息。我们将这些生产信号与我们的评估得分和每个任务的成本进行比较,以决定何时需要更改默认值或力度上限。
Bedrock 为每个请求报告四种类型的令牌:未缓存输入、缓存读取、缓存写入和输出。在评估运行中,我们为每个请求的所有四种类型定价,并累加任务中的请求。本文中的所有成本都使用该运行日期的公开列表价格计算。我们为测试的每个模型、推理力度和代理客户端记录平均成本,旁边是其得分,平均值包括低分尝试。
推理力度既改变得分也改变成本。在我们测试的一个模型上,从低力度切换到高力度使得分提高了 0.8 分。每个任务的成本则高出 2.5 倍。我们使用这些结果来设定力度上限,网关会强制执行这些上限。
在新模型发布的当天下午,向我们的代理发送单个请求便启动了一项评估活动,该活动让新模型通过我们所有的任务套件运行。套件是一组固定类型的任务,我们有七个。代理将数百个任务分散到各个套件和几个云工作区中,每个结果一旦评分器评分就会立即出现。第一批结果在几小时内到达。到第二天早上,每个套件都有了得分和每个任务的成本,我们拥有了整个组织部署决策所需的一切。
评分器在每次运行中都从冻结的数据重新计算正确答案,只有当所有检查都通过时,代理才能获得 1 的奖励。要在每个力度下测试一个模型,请在作业配置中将每个力度列为独立的代理:
运行评估可以进行优化。编写评分标准(评分器遵循的评分指南)是最困难的部分。我们的长任务跨越几个回合,模拟评审员在回合之间提出反对意见。模型评分器根据八个加权维度对每次运行进行评分。每个维度的得分为 0 到 3。
查看原文
Amazon Bedrock provides access to foundation models from multiple providers through a common API, with AWS managing the model infrastructure. It integrates with AWS access controls and monitoring, and processes prompts, code, and model responses within AWS.
As coding agents move into production, changes in models and usage can affect latency, capacity, and cost. Amazon Bedrock runtime metrics in Amazon CloudWatch help teams investigate latency increases, throttling, and changes in token consumption. Evaluations, or evals, complement these operational metrics by assessing the quality of an agent’s work on a fixed set of representative tasks.
In this post, we show how Airbnb evaluates models for coding agents and monitors their use on Amazon Bedrock. We explain how cost per task guides model selection and how production data helps us revisit those choices as usage changes. We also describe the runtime signals used to investigate latency, throttling, and spend.
At Airbnb, thousands of engineers and engineering-adjacent people use AirChat, our internal tool for running coding agents, every week. The agents send their model requests through our large language model (LLM) gateway, which runs on Amazon Bedrock. Bedrock processes Airbnb’s code, prompts, and model responses within AWS. Eight new models came out in about five weeks. By mid-October, about two thirds of the requests our engineers sent through coding agents went to those new models.
Each release can change output tokens, cost, latency and capacity at once, and the benchmark charts published with a launch don’t show how a model handles our engineers’ work. So we run two systems side by side. Before a rollout, we run evals on our own tasks to measure each model’s score and cost per task. Cost per task is the list-price cost of all the tokens an agent uses to finish one task.
After a rollout, observability shows us what changed in production. When production usage doesn’t match the eval results, we find out why.
Solution overview
You can diagnose most production problems with a few signals in the AWS/Bedrock namespace in CloudWatch:
- Latency:
InvocationLatency, TimeToFirstToken, OutputTokenCount, and output tokens per second (OTPS), which you derive.
- Throttling:
InvocationThrottles, plus the tokens each request reserves against your quota, which you publish yourself. Bedrock also publishes EstimatedTPMQuotaUsage, which measures your estimated tokens per minute (TPM) quota consumption, including cache write tokens and output burndown multipliers, across all inference APIs (Converse, InvokeModel, ConverseStream, and InvokeModelWithResponseStream). You can use this metric to set proactive alarms before reaching your quota limit, track your quota consumption across your models, and request further quota increases before usage is rate limited.
- Cost:
InputTokenCount (uncached input), CacheReadInputTokenCount, CacheWriteInputTokenCount and OutputTokenCount, each priced at its own rate. Cache reads cost less than uncached input, and cache writes can cost more depending on the model. A lower cache hit rate can increase cost per request.
On top of these metrics, we add cost per task. Our evals measure a score and a cost per task for every model and every reasoning effort we allow. Reasoning effort is a setting that controls how much a model reasons before it responds. We use the results to choose each default, which is the model and effort that our coding tools use unless an engineer picks another, and each effort ceiling, which is the highest effort we allow for a model.
Architecture
Engineers run AirChat on their AirDev workspaces, which are personal development machines in the cloud. They can also hand a task to AirChat Remote, which runs cloud agents in separate cloud workspaces while the engineer does something else. Both paths go through our LLM gateway to Amazon Bedrock.
Figure 1 shows how AirChat and AirChat Remote reach Amazon Bedrock through our gateway. It also shows how evals and production monitoring guide model defaults and effort ceilings.

Figure 1: Coding agents and cloud agents call Amazon Bedrock through our LLM gateway. Our usage records and CloudWatch metrics show real use, and we use eval results to set defaults and effort ceilings
The gateway records the model, client, team, and user for each request. We use these records only in aggregate, for cost and capacity planning. CloudWatch supplies token counts, latency, errors, and throttles. We compare those production signals with our eval scores and cost per task to decide when defaults or effort ceilings need to change.
Prerequisites
On the AWS side, you need:
- Access to the models in Amazon Bedrock you plan to run and evaluate. For model availability by AWS Region, see Regional availability by models in the Amazon Bedrock User Guide.
- Streaming calls through
ConverseStream or InvokeModelWithResponseStream. Bedrock publishes TimeToFirstToken only for the streaming APIs.
- Model invocation logging, turned on in every AWS Region you call. It’s off by default. The logs can hold full prompts and responses along with the caller’s identity, so restrict who can read them and set a retention period.
Tagged IAM roles for your callers, with cost allocation tags activated. Activate them before you start collecting, because activation doesn’t backfill earlier costs.
You also need three things of your own:
- A set of tasks from your own engineering work. Give each one a starting repository state, the prompt an engineer would write and a grader you trust. A grader is the program or model that scores the agent’s work. We build our tasks from real code changes, the review comments on those changes and the decisions that the original ticket didn’t spell out. Step 2 shows one.
- A gateway or proxy in front of Bedrock that records the model, the client, the team, the user and the token counts for each request.
- A table of public list prices for every model you run, with a date on each entry, so you can price eval tokens and production tokens on the same basis.
Before rollout, define what the agents can access and which actions need human review. Run code in isolated workspaces and test generated changes before merging. Amazon Bedrock Guardrails can help filter harmful content and sensitive information in prompts and responses. For responses based on supplied reference material, contextual grounding checks can flag unsupported claims. Test these controls on your own tasks and include their latency and cost in your measurements.
Walkthrough
Start by measuring what a complete task costs. Then use those results to choose model defaults, check production behavior, manage capacity, and attribute spend.
Step 1: What does one finished task cost you?
A coding agent takes many requests to finish one task, and on each request the model re-reads the context so far. So we price the whole task.
Bedrock reports four kinds of tokens for each request: uncached input, cache reads, cache writes and output. In an eval run we price all four for every request and add up the requests in the task. Every cost in this post uses the public list prices dated for that run. We record the average cost next to the score for every model, reasoning effort and agent client we test, and the average includes low-scoring attempts.
Reasoning effort changes both score and cost. On one model we tested, moving from low to high effort raised the score by 0.8 points. Cost per task was 2.5 times as high. We use these results to set effort ceilings, which the gateway enforces.
Step 2: How fast can you evaluate a new model?
On the afternoon a new model came out, a single request to our agents started an eval campaign, which runs the new model through all our task suites. A suite is a fixed set of tasks of one type, and we have seven. Agents spread hundreds of tasks across the suites and several cloud workspaces, and each result appeared as soon as a grader scored it. First results arrived within hours. By the next morning, every suite had a score and a cost per task, and we had what we needed for a deployment decision for the entire org.
That speed depends on tasks that are already written down. Each task is a folder that holds everything one run needs. Here is a trimmed task from our machine learning work:
honest-holdout-bakeoff/
instruction.md what the agent is told
task.toml time limit, CPUs and memory
environment/ Dockerfile for the sandbox
tests/test.sh runs the grader and writes the reward
tests/grader.py the checks
solution/ a reference solution that must pass the grader
instruction.md (excerpt):
We have two candidate models for a prediction task, and we need to
know which one to ship.
...
Then answer the question we actually have to decide: is either
candidate worth rolling out over the system already in production?
The grader recomputes the right answer from frozen data on every run, and the agent earns a reward of 1 only if every check passes. To test one model at every effort, list each effort as its own agent in a job config:
job_name: model-a-effort-sweep
n_attempts: 1 # one attempt per task at each effort
datasets:
- path: tasks/ml # each task brings its own grader
agents:
- name: model-a-low
model_name: vendor/model-a
kwargs: {reasoning_effort: low}
- name: model-a-medium
model_name: vendor/model-a
kwargs: {reasoning_effort: medium}
- name: model-a-high
model_name: vendor/model-a
kwargs: {reasoning_effort: high}
Both examples use the formats of Harbor, an open-source framework for running agent evals in sandboxes. It’s one option if you’re starting out.
Running the evals can be optimized. Writing the rubric, the scoring guide that a grader follows, is the hardest part. Our long tasks run over several rounds, with a simulated reviewer pushing back between rounds. A model grader scores each run on eight weighted dimensions. Each dimension gets a score from 0 to 3.
Correctness of delivery 25 Judgment 10
Verification quality 15 Evidence and reporting 10
Autonomy 15 Readability 10
Design and workload fit 10 Operational discipline 5
Ceiling: if the reviewer had to rescue the agent repeatedly
(autonomy 0 or 0.5), the total is capped at 25 of 100.
Because of ceilings like this one, an agent that needed rescuing can’t score well on polish alone.
Step 3: What do you watch after a rollout?
When latency rises, chart three signals before you open a support case: TimeToFirstToken for the wait before generation starts, OutputTokenCount for response length, and OTPS for generation speed. Bedrock has no native OTPS metric. You derive it with metric math, as the Amazon Bedrock User Guide describes:
OTPS = OutputTokenCount / (InvocationLatency - TimeToFirstToken) * 1000
In the example from our session, latency rose 40 percent while time to first token and OTPS stayed flat. Someone had changed the system prompt to ask for more thorough answers, and output tokens per request rose 50 percent. The model generated at the same speed and wrote longer responses. Treat prompt changes and model upgrades as deploys, and mark them on your dashboard.
Each week we put the CloudWatch runtime metrics next to the usage records from our gateway for each model. For cost and capacity planning, we also look at usage in aggregate: cost per request and the average number of requests per user each week. A new model can move both. After every rollout, we compare them with what we expected from the eval’s cost per task.
Step 4: The evals looked good. What did production show?
One model, which we call model A, looked good in our evals. On our coding tasks, it scored more than three points above our previous best model. Cost per task was about the same, and the same grader scored every model. We rolled it out, capped effort at high, and asked people to use low or medium for most work.
A few weeks after launch, we priced model A’s production usage at list price and looked at it in aggregate. Average weekly spend per user was several times higher on model A than on the model before it. Each model A request cost about three times as much, and the average model A user sent about three times as many requests.
Output tokens made up only a tenth of a request’s cost. Most of the cost went to cached context, which the model re-read on every request. Model A could work on its own for a long time, so people started long, self-directed sessions with it.
The eval measured the model on a task. Production measured how people chose to work with it.
Later, a newer model scored higher than model A on the same tasks and grader, at about a sixth of its cost per task. We moved model A’s users to it with a configuration change.
We then checked the change in aggregate on the same user group, the heavy users of model A. In their first full week on the newer model, the group sent about the same number of requests to their main model, and the group’s spend on that model fell to about a seventh.
Step 5: Why you get throttled, and where traffic goes next
On the bedrock-runtime endpoint, Bedrock checks your tokens-per-minute quota when a request arrives, before the model writes anything. It reserves your uncached input tokens and your cache writes, plus max_tokens, the most output the request is allowed. It adjusts that reservation as the model generates and returns the unused part when the request settles. Cache reads never count against the quota, so a request can be cheap against your quota and expensive on your bill. Output tokens settle at a burndown rate that is 1:1 on most models and several times that on some current models, so check your model’s rate.
A dashboard of billed tokens can show room while requests are being throttled. In our session’s example, the quota was 1 million tokens per minute and traffic was 300 requests per minute. The dashboard showed about 30 percent of quota. After applying a 5:1 output-token burndown rate, completed requests consumed about 77 percent.
Admission uses a different calculation. Each request initially reserved 600 uncached input tokens plus a max_tokens of 4,096, for a total of 4,696 tokens. Responses were about 800 tokens at p99.
Lowering max_tokens to 1,024 cut the initial reservation to 1,624 tokens, about 65 percent smaller. The limit was still above p99. For unchanged input and output, final quota consumption stays the same. The smaller reservation leaves more room to admit requests. Monitor stopReason to catch truncated responses. Use backoff and jitter, and cap retries.
Demand can also outrun the headroom you have on any one model. You want to know where work can go before you need it, and our evals answer that. The answer can be a model from another model family. For every major model we use, we keep at least one equivalent that performs as well on our evals and that we can load-balance to. Without those evals, choosing where the work goes would be guesswork.
If headroom on a model stays tight, we can rebalance by changing a default in our configuration. First check that the equivalent model has capacity for the extra load and that your quota covers it. Keep enough traffic on it to hold your access, too: after a model enters its Legacy period, existing customers can lose access after 15 days of inactivity. A Legacy model works until end-of-life, and after that Bedrock removes it from every Region, requests to it fail, and Bedrock doesn’t move your traffic automatically. Alarm when LegacyModelInvocations rises above zero.
Step 6: Whose spend is it?
Bedrock runtime metrics are dimensioned by ModelId, so you can see which model served a request but not which team sent it. Bedrock publishes these metrics both with and without the ModelId dimension, and a request that fails before Bedrock resolves the model appears only in the series without it, so filtering by ModelId undercounts errors. Bedrock offers three ways to attribute spend:
- Which team or user? Bedrock records the caller’s IAM identity on every request. Tag your roles, or pass session tags, to see cost by tag in AWS Cost Explorer and the AWS Cost and Usage Report (CUR 2.0).
- Which application? Application inference profiles are model-specific, so create one tagged profile per application and model, and call the matching profile’s ARN.
- Which feature or prompt? Pass key-value pairs in
requestMetadata on Converse. Bedrock writes them to the model invocation logs and leaves them off the bill. To price them, multiply the token counts in the logs by the published rates. Neither classic CUR nor CUR 2.0 carries a per-request ID, so reconcile the two by comparing totals by model, usage type and time period.
Behind a gateway, Bedrock sees the gateway’s role, so have the gateway assume a role for each user with its own RoleSessionName or session tags. Cache those credentials for the session, because assuming a role on every request can exceed your AWS Security Token Service (AWS STS) rate quotas. Requests keep succeeding when attribution breaks, so set an alarm for when Bedrock fails to deliver invocation logs.
Our usage records let us attribute usage by model, client, team and user, which we use only in aggregate for cost and capacity planning. We compute cost per task from the same four token counts, so we can compare token mix and cost between our evals and production.
Results
We can go from a model release to a deployment decision overnight. After one rollout, average weekly spend per user on model A was several times that on the previous model. We moved its heavy users to a newer model. In their first full week, the group’s spend on its main model fell to about a seventh, with roughly the same number of requests. For every major model we use, our evals identify an equivalent we can rebalance to.
Clean up
After testing, stop the eval jobs and agent processes you started. If you enabled model invocation logging only for this walkthrough, disable it in each Region using DeleteModelInvocationLoggingConfiguration.
After callers no longer use them, delete the application inference profiles you created using DeleteInferenceProfile. Remove IAM roles, policies, and resource tags created only for this walkthrough. Deactivate cost allocation tags used only for these tests. Delete test CloudWatch log groups and S3 log objects you no longer need.
Conclusion
We run four checks every time a model ships:
- Release day: run the new model on your own tasks at every effort, and record a score and a cost per task together.
- Before rollout: choose the default and cap effort from those numbers. Block any effort that costs more and scores less than a lower effort.
- First weeks: compare cost per request and average requests per user with the eval’s cost per task, pricing both from the same four token counts.
- Before you need it: know which models score the same on your evals, from any model family, and check their capacity and your quota before you rebalance.
We chose Amazon Bedrock for access to multiple model families through one API. It lets us use our existing AWS access controls and logging while AWS manages the model infrastructure. We use evals to choose models for our work and observability to know when those choices need another look.
To go further, see these pages in the Amazon Bedrock User Guide:
About the authors