所有基准测试

VGI-Bench

VGI-Bench(Video General Intelligence Bench,由 Seldon 开发)针对长视频提出单项选择题:什么最先出现、什么与其他事物同时出现在屏幕上、某个特定镜头出现时旁白说了什么,以及前提从未发生过的问题。下方每个样例都是通过 OpenRouter 进行的一次真实运行。模型在采样温度为 0 的情况下,在一轮中接收完整视频和问题,评分所依据的答案是其最后一个 Answer: 行中的字母。

上次基准测试运行

模型对比

最佳性价比

准确率
代表性运行的准确率,按从高到低排序。
每个问题的成本
每个问题的平均成本,按从低到高排序。
每题耗时
每道题的平均实际耗时,按耗时从短到长排序。

成本效率

准确率与成本(帕累托前沿)
每个点代表一个模型,在可用时使用默认路由(不固定到某个提供商)。曲线为帕累托前沿:没有模型能在准确率和成本两方面同时优于这些模型。

排行榜

在公开测试集上的准确率(共 439 道题,运行至少需回答 395 道题才会列出),以及模型在各提供商端点上运行时每道题的成本、耗时、输入和输出 token 数。分数汇总了最近 90 天内的成功运行结果。

VGI-Bench 排行榜:按提供商列出模型准确率、成本、耗时及每道题的输入和输出 token 数
#模型标准差
166.2%±1.0个百分点US$0.03964s141k2.03k
265.5%--US$0.0464.6m412k11.5k
363.7%±4.8个百分点US$0.0732.5m241k6.5k
463.7%--US$0.558.6m613k20.1k
563.6%--US$1.033.1m233k8.62k
6
Google AI Studio
61.4%±0.7个百分点US$0.0611.8m3416.04k
761.1%--US$0.224.2m227k14.3k
858.0%--US$0.338.2m579k13.6k
9
Meta
57.4%--US$0.0231.6m231k9.18k
1056.8%----1.9m313102
1156.8%--US$0.116.2m1.52M24.1k
1254.8%±0.5个百分点US$0.144.7m111k40.2k
1354.8%±0.5个百分点US$0.275.6m1.48M10.9k
1454.2%±0.5个百分点US$0.242.6m151k21.8k
1552.8%±2.3个百分点US$0.163.7m98.6k14.1k
1651.3%--US$0.0086.0m7.53k13.7k
1751.0%±0.6个百分点US$0.162.0m150k15.3k
1850.6%±0.4个百分点US$0.217.1m532k20.9k
19
Google AI Studio
50.0%±0.3个百分点US$0.0241.9m3419.42k
2048.0%--US$0.0122.4m34029.7k
2147.6%±1.9个百分点US$0.0262.3m36.7k7.15k
2246.5%±2.9个百分点US$0.0186.4m272k13.2k
2346.2%±0.1个百分点US$0.0082.7m46.4k817
2446.2%±1.2个百分点US$0.104.9m1.49M20.2k
2545.6%--US$0.0274.6m77.4k14.7k
2645.5%±1.0个百分点US$0.958.6m1.45M21.4k
2745.5%±0.7个百分点US$0.177.5m532k21.3k
2845.4%--US$0.0345.2m337k11.9k
2945.2%±0.8个百分点US$1.146.7m1.49M19.4k
3045.1%±0.6个百分点US$0.174.9m440k18.6k
3143.4%±2.9个百分点US$0.0271.9m232k9.34k
3243.4%--US$0.343.2m77k21.1k
3342.3%--US$0.193.5m515k7.53k
3440.9%±7.2个百分点US$0.704.4m38.3k81k
3534.8%±2.9个百分点US$0.0217.5m199k16.4k
3634.6%±2.7个百分点US$0.0288.3m58.2k43.9k
3730.1%±2.6个百分点US$0.0819.8m53.4k75.3k
3828.9%--US$0.0191.6m49k7.48k
3928.5%±0.8个百分点US$0.0161.9m14.6k30.3k
4024.7%±7.3个百分点US$0.0042.8m4.59k9.91k
4122.2%--US$0.00644s130k3.99k
42
Reka Edge
帕累托
Reka
21.4%±0.2个百分点US$0.0001531s1.49k30.5
4319.2%--US$0.0011.8m5.71k9.46k
4415.5%--US$0.05044s1.58k4.7k
4514.7%--US$0.323.0m1.63k15.5k
4614.4%--US$0.03976s1.42k10.3k
4713.8%--US$0.02247s6.28k6.46k
4812.8%±0.4个百分点US$0.0233.9m38k6.22k
4912.7%±0.0个百分点US$0.06851s1.58k8.24k
5012.4%--US$0.01324s1.19k3.29k
5112.4%±0.4个百分点US$0.01556s38.3k6.56k
5212.3%±0.5个百分点US$0.136.9m124k2.42k
5311.8%±0.4个百分点US$0.0313.9m32.7k5.87k
5411.8%--US$0.04182s5.06k3.17k
55
Google AI Studio
11.5%--US$0.01624s1.39k3.97k
5611.4%--US$0.111.9m1.58k10.6k
5710.9%--US$0.09859s1.39k7.96k
5810.8%±0.3个百分点US$0.00544s4.57k5.08k
5910.6%--US$0.09860s1.39k8.02k
60
Google AI Studio
10.1%--US$0.08839s1.38k9.59k
618.4%--US$0.00613s1.39k2.08k
628.2%--US$0.03858s1.39k25.1k
638.2%--US$0.03858s1.38k25.1k
646.7%±0.3个百分点US$0.0075.7m5.07k24.3k
656.4%--US$0.2657s1.57k12.9k
666.3%±5.7个百分点US$0.02536s5.46k1.69k
675.6%±4.3个百分点US$0.0171.9m69.2k804

轨迹

推理轨迹体现了模型的感知:它声称看到的片段,以及为这些片段标注的时间戳。将推理轨迹与视频对照阅读,可以了解答案是如何得出的,以及在失败的运行中,描述在何处偏离了视频画面。

正确

指针悬停时间最长的窗口

正常流程: 模型定位到了该时刻,并正确解读了它。

问题 1187 global-graph/interaction_partner Qwen3.8 27B

发送给模型的完整视频,由智能体运行框架所使用的 OpenRouter 镜像提供。

鼠标指针最常悬停在哪个选项上?

  1. A)Windhawk application window密钥,模型
  2. B)Install button
  3. C)Adobe Premiere Pro window
  4. D)Windows taskbar
回答了 A ✓按键 A

该追踪会在整个录制过程中跟踪指针,而不是只看一帧,将接近结尾时短暂出现的 Premiere Pro 标为干扰项,并最终确定在整个会话中占据主导地位的窗口。

The question asks which option the mouse pointer hovers over most often. Let me think about the video content. The video is a tutorial about Windhawk, a customization tool for Windows. Throughout the video, the user is constantly interacting with the Windhawk application window - browsing mods, installing them, configuring settings, etc. Let me consider each option: - A) Windhawk application window - This is the main focus of the video. The user opens Windhawk, browses mods, clicks on details, settings, installs mods, etc. The mouse is frequently over this window. - B) Install button - The install button is clicked a few times but not hovered over most often. - C) Adobe Premiere Pro window - This appears only briefly near the end (around 13:20-13:40) as a joke about the title bar color. - D) Windows taskbar - The taskbar is shown but the mouse isn't constantly hovering over it. The dominant interaction is with the Windhawk application window. The mouse pointer is over it for the majority of the video as the user navigates through mods, settings, etc. So the answer is A. Let me double check by reviewing timestamps. From 1:20 onwards, the Windhawk window is open and the mouse moves around it constantly - clicking Mods, Explore, scrolling, clicking Details, Settings, etc. This continues through most of the video. The taskbar is shown at times but the mouse hovers over the Windhawk window more. Thus A is correct.