TapTap Maker Benchmark logo TapTap Maker Benchmark agentic game-dev on the UrhoX engine

Methodology

TapTap Maker Benchmark measures how well AI agents develop games on a real engine (UrhoX, scripted in Lua). Every score on this site is produced by running the agent's code on the UrhoX engine and checking it with deterministic assertions — never by an LLM judge, and never reported by a vendor. This page explains what we measure, how scores are produced, and why you can trust a benchmark whose harness is closed source.

What we measure#

Each task asks an AI agent to build or modify a Lua-scripted game feature for the UrhoX engine — physics gameplay, 3D scene construction, UI, animation, networking, and more — inside a realistic development workspace with the engine's documentation and helper libraries. The agent writes code; the engine decides whether it works.

Our scope is deliberately narrow and deep: rigor of real engine execution over breadth of game genres. We cover one engine and one scripting language thoroughly — a broad task set spanning categories from physics gameplay to UI, across three difficulty tiers — rather than many engines shallowly.

How scoring works#

Every attempt is graded by a three-level pipeline:

  • L0 — deliverables. Did the agent produce the files the task requires?
  • L1 — static analysis. Does the code pass a Lua language-server check?
  • L2 — engine execution. The agent's scene is executed headlessly by the real UrhoX runtime and checked against deterministic Lua assertions (positions, physics behavior, UI state, network wiring, …).

The headline number on the main board is the L2 pass rate. L2 is the level that matters: the code doesn't just look right, it runs right. There is no LLM in the judging loop, so a score cannot be influenced by a judge model's preferences, prompt sensitivity, or style bias.

Statistics#

Each score is the pass rate over multiple independent repetitions of every task, and every rate on this site carries a precomputed 95% confidence interval (case-weighted bootstrap; an exact Clopper–Pearson interval on pooled attempts when a model sits at a 0%/100% boundary where the bootstrap degenerates). The interval width reflects the sample size behind each score.

Every rank carries a plausible range. A leaderboard position computed from sampled runs is itself uncertain, so we bootstrap the entire board: resample the measurement noise, re-rank all models, repeat — and take the 2.5–97.5 percentile of each model's rank. The board shows that range under each rank (a 3 with 1–9 beneath it means: point estimate 3rd, but anything from 1st to 9th is consistent with the data). Where two models' ranges overlap, the ordering between them is simply not resolved at the current sample size — we would rather show that honestly than over-claim a separation.

# rank ranges (precomputed in the snapshot — the frontend does no statistics)
bootstrap: resample every cell's measurement noise → re-rank the board → repeat
each model's 2.5–97.5 percentile of rank = the range shown under its number

Contamination resistance#

Any public benchmark eventually leaks into training data. Our answer is a two-layer dataset:

  • Public tasks — drive the category and difficulty boards.
  • Held-out tasks — task prompts, ground truth, and transcripts are never published. The Frontier board shows scores only: task identities, the set's size, and its composition never leave the harness.

The dataset evolves under semantic versioning, with new held-out tasks added over time, and each edition pins an exact dataset version and content hash.

Why trust a closed benchmark#

The harness is closed source, so we earn trust through mechanism, not disclosure:

  • All scores are run by us — a uniform agent driver, fixed inference parameters, and recorded reasoning modes. Nothing on the board is vendor-reported.
  • Full traceability chain. Every task carries a content fingerprint (task_hash) and every grading battery a fingerprint (battery_hash); each edition pins the engine commit, the dataset version, and a dataset content hash. Raw results are never modified — corrections are appended to an audit ledger, so any reversal is itself on the record.
  • Isolation. The agent's generation phase runs in a per-attempt hardened container that physically cannot read the grading surface (ground truth, assertion scripts).
  • Determinism. Grading is engine execution plus deterministic assertions — re-running the same code yields the same verdict.

Policies#

  • Model inclusion. We benchmark widely available frontier and cost-efficient models under their pinned provider versions. A model enters the board only through a full benchmark run executed by us.
  • Versioning. Models are identified by their exact pinned version at run time; when a provider updates a model, a new run is required for the new version to appear.
  • Appeals. Believe a score is wrong? Contact the maintainers. Substantiated corrections go through the append-only ledger described above — raw results are never edited in place.
  • Editions are permanent. Published editions are never deleted, and models are never silently removed from an edition. Every snapshot stays reachable from the edition archive.

Limitations#

  • One engine, one language. Results speak to UrhoX/Lua game development; they do not directly generalize to other engines or stacks.
  • Sampling variance. Agents are stochastic. Confidence intervals quantify this, but close models should be read as close — the rank ranges exist for exactly this reason.
  • Cost figures are list-price estimates from provider token counts, not billed amounts; un-metered channels are shown as N/A rather than estimated.
  • Timing is measured on our runners, not a vendor SLA. Time per task is end-to-end wall clock — reasoning, token emission and the agent harness's tool round-trips — so it depends on our network location, the agent scaffold and provider load at run time. Attempts that hit an infrastructure timeout or hung are excluded from the timing sample entirely (they would otherwise report a kill deadline as a model's speed), which is why per-task time is compared on the shared set of tasks every model completed at least once.
  • Single agent driver per run. A model's score reflects its pairing with a specific agent scaffold; a different scaffold may shift results.

评测方法

TapTap Maker Benchmark 衡量 AI agent 在真实游戏引擎（UrhoX，Lua 脚本）上的开发能力。本站每个分数 都由 agent 产出的代码在 UrhoX 引擎上真实执行、再由确定性断言判定得出——不使用 LLM 打分， 也不采用任何厂商自报成绩。本页说明我们测什么、分数如何产生，以及为什么一个闭源 harness 的 榜单值得信任。

测什么#

每道题要求 AI agent 在配备引擎文档与辅助库的真实开发工作区中，为 UrhoX 引擎用 Lua 实现或修改一个游戏功能——物理玩法、3D 场景搭建、UI、动画、网络同步等。agent 写代码， 引擎判定它是否真的工作。

我们的范围刻意窄而深：真实引擎执行的严谨度优先于游戏品类的覆盖广度。 我们把一个引擎、一门脚本语言做透——覆盖从物理玩法到 UI 的多个类别、三个难度档位的 大量题目——而不是浅尝多个引擎。

怎么判分#

每次尝试经过三级流水线判分：

  • L0 —— 交付物。agent 是否产出了题目要求的文件？
  • L1 —— 静态分析。代码是否通过 Lua 语言服务器检查？
  • L2 —— 引擎执行。agent 的场景由真实 UrhoX 运行时无头执行，并用确定性 Lua 断言校验（位置、物理行为、UI 状态、网络接线……）。

主榜头条数字是 L2 通过率。L2 才是要紧的那一级： 代码不只是"看起来对"，而是"跑起来对"。判分回路中没有 LLM，分数不受裁判模型的偏好、 prompt 敏感性或风格偏好的影响。

统计方法#

每个分数都是每道题多次独立重复的通过率，且本站所有通过率都带有预先算好的 95% 置信区间（按题加权的 bootstrap；当模型全过或全败、bootstrap 退化为 零宽时，回退到按合并尝试次数的 Clopper–Pearson 精确区间）。区间宽度自然反映每个分数背后的 样本量。

每个名次都带可能范围。由采样轮次算出的榜单位置本身就有不确定性， 因此我们对整张榜做 bootstrap：重采样测量噪声、重新排名、反复进行，取每个模型名次的 2.5–97.5 分位。榜单在每个名次下方标注该范围（3 下方的 1–9 表示：点估计第 3，但第 1 到第 9 都与数据相容）。当两个模型的范围重叠时，说明在当前 样本量下它们之间的先后尚未被解析——我们选择如实展示这一点，而不是过度断言名次差距。

# 名次范围（由快照预计算——前端不做任何统计）
bootstrap：重采样每个 cell 的测量噪声 → 重新排名 → 反复进行
每个模型名次的 2.5–97.5 分位 = 名次下方标注的范围

抗污染设计#

任何公开榜单最终都会渗入训练数据。我们的答案是双层数据集：

  • 公开题 —— 用于分类榜与难度榜。
  • 私有题（held-out） —— 题面、标准答案与运行记录永不公开。前沿榜只展示 分数：题目身份、题集规模与构成都不会离开 harness。

数据集按语义化版本演进，私有题持续增补；每个 edition 固定确切的数据集版本与内容哈希。

为什么信任一个闭源榜单#

harness 闭源，所以我们靠机制而非公开代码来赢得信任：

  • 全部分数由我们自跑 —— 统一的 agent 驱动、固定推理参数、记录推理模式。 榜上没有任何厂商自报成绩。
  • 完整可追溯链。每道题有内容指纹（task_hash）、每套判分电池有 指纹（battery_hash）；每个 edition 固定引擎 commit、数据集版本与数据集内容哈希。 原始结果永不改动——勘误以追加方式写入台账，任何翻案本身也留有记录。
  • 隔离。agent 的生成阶段在每次尝试独立的硬化容器中运行，物理上读不到 判分面（标准答案、断言脚本）。
  • 确定性。判分 = 引擎真实执行 + 确定性断言——同一份代码重跑得到同样的 判决。

政策#

  • 模型收录。我们评测广泛使用的前沿模型与高性价比模型，均以固定版本 pin。 模型只能通过我们执行的完整跑分进入榜单。
  • 版本管理。模型以跑分时的确切固定版本标识；厂商更新模型后，新版本需要 重新跑分才会出现。
  • 申诉。认为分数有误？联系维护者。经核实的更正走上述只增不改的台账—— 原始结果绝不被原地修改。
  • 版本永久存档。已发布的 edition 永不删除，模型不会从 edition 中静默下架。 每个快照都可以在版本存档中访问。

局限性#

  • 单引擎、单语言。结果针对 UrhoX/Lua 游戏开发，不能直接推广到其他引擎 或技术栈。
  • 采样方差。agent 是随机的。置信区间量化了这一点，但分数接近的模型就应 读作"接近"——名次范围正是为此存在。
  • 成本数字是按刊例价的估算，来自厂商 token 计数，并非账单金额；未计量的 通道显示为 N/A 而非估算值。
  • 耗时是我们机器上的实测，不是厂商 SLA。单题耗时为端到端 wall-clock—— 推理时间 + 吐字 + agent 脚手架的工具往返都算在内——因此受我们的网络位置、agent 脚手架、 跑分时厂商负载影响。命中 infra 超时或挂死的 attempt 完全不进入耗时样本（否则等于把我们的 强杀时限当成模型速度上报）；也正因如此，单题耗时只在"每个模型都至少完整做完过一次"的 共同题集上比较。
  • 每次跑分使用单一 agent 驱动。模型的分数反映它与特定 agent 脚手架的 组合，换脚手架结果可能变化。