MIT Technology Review reported that the updated plot adds more tasks, including some that are more open-ended, and that METR researchers have worked to include fuzzier, judgment-laden tasks such as having models design new AI evaluations and score philosophical arguments.[1] Even the most advanced models fell short on these tasks, with none of their proposed evaluations deemed useful by METR staff, though newer models like Opus 5.5 are showing "borderline above the bar" performance compared to all models being "clearly below the bar" a year ago, the report states.[1] The original plot tracked progress on verifiable tasks like coding, where every four months the length of the tasks that frontier models could successfully complete doubled.[1] Judgment-laden tasks could be most relevant to AI research and development, and METR researchers have included some of these in their new suite of tasks.[1]
METR adds judgment-heavy tasks to its AI progress plot, top models fall short
None of the models' proposed evaluations were deemed useful by METR staff, though newer models like Opus 5.5 show "borderline above the bar" performance, against all models "clearly below the bar" a year ago.
0 primary · 1 independent · 0 commentary · 0 vendorSources
- The most misunderstood graph in AI just got a major update MIT Technology Review Independent