We have recently been rethinking one question: what kind of benchmark is still worth building today?
Over the past few years, the development of large models has almost always been pulled forward by benchmarks. GLUE and SuperGLUE pushed NLP pre-training forward. MMLU turned “general knowledge capability” into a comparable score. HumanEval pushed code generation into the mainstream. SWE-bench then moved coding agents from “writing functions” to solving real GitHub issues.
A benchmark has never been just a leaderboard. It is more like a way of modeling problems: it tells us what capabilities models currently lack, where the gaps are, and where the next batch of training data should be created.
So the most important thing right now is not to keep looking for one more leaderboard where every model can score 90. It is to find benchmarks that fit our own business, our own products, and our own organizations, and then use them to guide evaluation and training-data construction.
For this issue, we selected several new benchmarks in 2026 that are worth watching.
They all point to the same shift: the next generation of benchmarks should no longer reward models that merely “know how to answer questions.” They should reward systems that can get things done reliably in the real world.
User Experience | UXBench
Benchmarking User Experience in AI Assistants
One-sentence summary: UXBench is a user-experience benchmark proposed by the Tencent Yuanbao team. It uses real AI assistant interaction logs to evaluate whether a model can understand user feedback, preferences, and failure recovery.
Overview
UXBench evaluates whether an AI assistant can make users feel that it is “good to use.” It contains three tasks: UX Judge, which predicts user feedback; UX Eval, which evaluates whether a regenerated response creates a satisfactory experience; and UX Recovery, which evaluates whether the model can recover after a failure. The data comes from more than 70,000 real Chinese AI assistant interaction logs, eventually forming 7,400 test samples across eight scenarios and 83 domains.
The core question it cares about is highly product-oriented: why is the user dissatisfied? Which response made the experience worse? Can the model identify a failure and recover the experience in the next round?
In the test results, the top overseas models generally scored higher, while Hunyuan 3 performed at a medium level across the three tasks, with scores of 64.3%, 48.8%, and 7.6%, respectively.
Team
The work comes from the Hong Kong University of Science and Technology and the Tencent Yuanbao team. The paper authors include Mengze Hong, Xia Zeng, Zeyang Lei, Sheng Wang, and others. The paper was submitted on June 8, 2026, and updated on June 9.
Why it matters
Past model evaluations often asked whether the answer was correct, whether the reasoning was strong, or whether the code could run. But in a real AI assistant, user satisfaction often depends on much finer-grained factors: whether the assistant is verbose, whether it can “steadily” absorb the user’s emotions, whether it can understand implicit needs, and whether it can repair the experience after a failure.
The value of UXBench is that it turns “user experience” into something that can be trained, evaluated, and iterated on. This type of benchmark may directly influence the construction of reward models and post-training data for assistant-like products in the future.
Its advantage is that it does not rely on a large amount of manual annotation. Instead, it continuously produces test samples through an automated pipeline from real user interactions with a mainstream Chinese AI assistant.
This also means UXBench looks more like a product-specific, customized benchmark that grew out of the product itself. It is well suited for optimizing the experience of Tencent Yuanbao-like Chinese AI assistants. But if it is directly transferred to other assistants, biases may appear because user groups, product forms, interaction styles, and feedback habits can all differ.
Building a benchmark for one’s own product - and then using it to serve the iteration of one’s own model and product - is a critical path forward.
Memory | MemLens
Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models
One-sentence summary: MemLens is a multimodal long-term memory benchmark designed to test whether models can truly remember and update information across multi-turn, multi-session, multimodal conversations.
Overview
MemLens is a memory evaluation set for multi-turn, cross-session, multimodal dialogue. It contains 789 questions and covers five memory capabilities: information extraction, cross-session reasoning, temporal reasoning, knowledge updating, and refusal - meaning actively saying “I don’t know” when there is no evidence. It tests models under four context lengths: 32K, 64K, 128K, and 256K, using a unified cross-modal token count to compare text and images on the same ruler.
The team evaluated 27 vision-language models and seven memory-enhanced agents.
The highlights are twofold. First, the paper ran an image ablation experiment: after removing the evidence images, the accuracy of two frontier LVLMs dropped below 2% on 80.4% of the questions, proving that visual evidence is truly needed. Second, it found that long-context LVLMs can be accurate on short conversations, but their performance decays as the dialogue becomes longer. Memory agents are more stable at longer lengths, but storage compression causes them to lose visual fidelity. This is one of the key structural trade-offs today.
Team
The work comes from Professor Song Yangqiu’s team at the Hong Kong University of Science and Technology, together with the Chinese University of Hong Kong, Nvidia, and Hillbot Intelligence. The first author is Xiyu Ren. The dataset and evaluation code have been open-sourced. Paper: https://arxiv.org/abs/2605.14906. Code: https://github.com/xrenaf/MEMLENS.
Why it matters
MemLens fills an evaluation gap in multimodal long-term memory.
Its core contribution is that it compares the two technical routes for “long memory” - long-context LVLMs and memory-enhanced agents - on problems that genuinely require visual evidence. The conclusion is that neither route works well on its own; a hybrid architecture is necessary.
For memory agents, the usual path is “store, then compress”: compress an image into a description or vector representation, and the date on a boarding pass or the amount on a ticket disappears the moment it is written. As a result, a base model may reason directly to 49%, while after wrapping it with a memory system, the score can drop to 15% - because when it is time to answer, the key information was never preserved.
Long-context multimodal models show the opposite pattern. In short conversations, simply looking at the original image gives high accuracy; but once the conversation becomes long, performance starts to decay.
MemLens turns “remembering, finding, and not hallucinating” into engineering metrics that can be diagnosed and optimized separately. For anyone trying to build multimodal memory into a product, it draws a clear map of the missing capabilities.
Long-Horizon Coding | RoadmapBench
Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
One-sentence summary: Software engineering evaluation is finally moving from “fixing bugs” to “doing version upgrades.” RoadmapBench uses version-upgrade tasks from real open-source projects to evaluate whether coding agents can complete long-horizon, multi-goal software development.
Overview
RoadmapBench contains 115 long-horizon coding tasks from 17 real open-source repositories and five programming languages. Each task gives an agent a snapshot of the old version of the codebase, then provides a roadmap for a target version, asking the agent to implement the new functionality required by that version.
The tasks are very difficult. On average, each task requires modifying 3,700 lines of code across 51 files. Even the strongest model, Claude Opus 4.7, solves only 39.1% of tasks, while weaker models solve only 5.2%. By contrast, these models often achieve over 80% on SWE-bench Verified.
Team
The work was led by Xin Dou from the AI startup UniPat AI, in collaboration with Peking University, and involved researchers from Fudan University, the University of Hong Kong, Tsinghua University, OG Labs, and Pipeline Lab. It was submitted on May 15, 2026, and updated on May 19.
Why it matters
SWE-bench is already very important, but it is still mainly about fixing bugs at the issue level. It is also highly concentrated in a small set of repeatedly used Python packages, which is not realistic enough and carries the risk of data contamination. Real software development requires cross-version migration, architectural changes, new functionality, regression tests, and compatibility handling. These are exactly the capabilities RoadmapBench tries to test.
Its value has three layers. First, it is realistic: it uses real version upgrades, five languages, and performs instruction-test consistency checks, reducing the space for benchmark gaming and contamination. Second, it has fine-grained scoring: it makes statements such as “the model completed 70% of a long-horizon task” measurable, comparable, and usable as training signals. Third, it points clearly to missing capabilities in coding agents: long-horizon planning, codebase localization, cross-file consistency, and turning exploration into precise edits.
The paper also quantifies an intuitive gap: designing a new abstraction, such as component creation, with an average pass rate of 36%, is far harder than locating and repairing an existing defect, or bug fixing, at 64%. That captures the essential difference between “developing” and “fixing bugs.”
Planning Capability | Agent Planning Benchmark
A Diagnostic Framework for Planning Capabilities in LLM Agents
One-sentence summary: Planning capability needs to be separated from execution results. APB is a benchmark focused on diagnosing agent planning. It tests whether a model can decompose tasks, select tools, handle broken tools, and identify unsolvable tasks.
Overview
APB contains 4,209 multimodal cases across 22 domains and five types of settings: overall planning, stepwise planning under feedback conditions, no tools, broken tools, and unsolvable tasks. Its core point is that it does not only care whether the model succeeds in the end. It asks: if the model fails, was it a planning failure or an execution failure?
The paper evaluated 12 multimodal models, and the gaps are large. For overall planning, GPT-5 reaches 74.5% plan accuracy, Gemini 3 Pro reaches 71.3%, while GPT-4o reaches only 19.5%.
The most convincing step is the post-hoc experiment. On multiple tasks, for GPT-4o, Qwen3-VL-235B, and Gemini 2.5 Flash, correcting the plan according to APB improves not only the plan itself, but also the final execution performance.
This turns APB from “a diagnostic leaderboard” into an upstream signal that can connect to real execution and guide improvement.
Team
The authors include Haoyu Sun, Wenxuan Wang, Mingyang Song, Yu Cheng, and others, from Shanghai AI Lab, Harbin Institute of Technology, Fudan University, and related institutions. The paper was submitted on June 3, 2026.
Why it matters
Many agent benchmark pass rates only tell us “the agent did not succeed,” but not why it failed. The value of APB is that it isolates planning capability, allowing researchers to see specific weaknesses in goal decomposition, tool selection, refusal calibration, and feedback correction.
It meets a core requirement of a good benchmark: it should not only assign scores, but also identify failure types. Only then can a benchmark reverse-guide how training data should be built.
Localization | K-BrowseComp
A Web Browsing Agent Benchmark Grounded in Korean Contexts
One-sentence summary: Deep Research needs localized benchmarks. K-BrowseComp is a Korean-context web-browsing agent evaluation set. It tests a model’s deep-search ability on the non-English internet: the answers are hidden in Korean-language websites, Korean local knowledge, and local information sources, and cannot be solved by relying only on the English-language web.
Overview
K-BrowseComp contains 400 questions. Among them, 300 questions in the Verified subset were written and cross-validated by native Korean speakers. Each question requires multi-hop retrieval, has a clear chain of evidence, and has a uniquely verifiable answer.
The results: GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1 score only 30% to 45.67% on the Verified subset, showing a clear drop compared with their performance on the English BrowseComp. Meanwhile, Korean-native models released under the “Korean Sovereign AI Foundation Model” initiative, such as K-EXAONE, HCX-SEED, A.X, and Kanana, score only 0% to 10.33%.
Team
This is a Korean academia-industry collaboration led by the Korean research community, including Chung-Ang University, KAIST, Seoul National University, OnelineAI, NAVER Cloud AI, and Carnegie Mellon University. The first author is Nahyun Lee from Chung-Ang University, and the corresponding author is Seungone Kim from CMU.
The paper has a strong flavor of “building sovereign evaluation capability for Korean.” It was submitted on June 1, 2026, and the data and code are open.
Why it matters
OpenAI’s BrowseComp has already shown that deep research agents need persistent search, cross-validation, and information integration. K-BrowseComp adds another layer on top of this: localization itself is a capability. Two things are worth highlighting here.
First, the Chinese-language world may also need its own Deep Research benchmark. The Chinese internet contains WeChat public accounts, Xiaohongshu, Zhihu, Bilibili, government websites, corporate websites, forums, and a large amount of high-value information hidden in PDFs, images, screenshots, and closed platforms. Much of this content is not indexed by general-purpose search engines.
Second, this also exposes the shortcomings of models in specific languages. Even the strongest global models still have significant room to improve if they want to truly serve global markets within a specific language and local information ecosystem. This is also one of the underlying drivers behind countries competing to build “sovereign evaluation” and “sovereign models.”
In-House Evaluation | GenAI Assembling Agent Eval Series
One-sentence summary: The five benchmarks above were built by others. The last one is something we are building ourselves. The GenAI Assembling AI frontier team’s Agent Eval series is essentially a “sudden exam”: on a regular basis, we pull several of the hottest general-purpose agents on the market into the same exam room and test them with different exam papers.
Overview
Over the past year, general-purpose agents have become a product form that technology companies must fight for: ChatGPT Deep Research, Gemini Deep Research, Claude Research, Genspark, Manus, Kimi, MiniMax, GLM, and ByteDance-related products are all competing over the ability to “autonomously search, reason in multiple steps, and output structured reports.”
But do these agents actually work on real tasks? Many existing benchmarks are either academically closed questions or self-scored evaluations that lack objective grading.
Agent Eval wants to do something different:
Real tasks: tasks with objective award or result moments, such as product launches, college entrance exams, sports events, financial reports, and decided court cases.
Same-time prompts: all participating agents receive the same prompt in the same time window.
Pre-locked process scoring: the process score is completed before the result is announced, and no post-hoc modification is allowed.
Fully open evaluation rules: prompts, ground truth, and scoring rules are all open.
Follow-up mechanism: each agent receives three standardized follow-up questions, testing self-checking, epistemic humility, and the ability to correct mistaken answers.
Team
The GenAI Assembling AI frontier team. GitHub: https://github.com/pingwest-ai/agent-eval.
Why it matters
Isn’t this worth paying attention to?
Fine, let’s say the serious reason.
Paper-style evaluations are like recorded shows, while what we are doing is a live broadcast: put the agents on stage seconds before the event happens, take the exam on the spot, and announce the results on the spot. This naturally gets closer to real-world scenarios, because real-world tasks come with uncertainty and time-limited submission.
Beyond live evaluation, GenAI Assembling also plans to expand the exam room into more strange but valuable places: fuzzy-intention image tests, real user statements that are basically a tangled mess, testing whether an agent can extract what the user actually wants from “I don’t even know what I want,” or game exams that pull a group of agents together to play a game. Such tasks can show results directly while also testing planning, strategy, execution, memory, and more. In short, the core is to find more real user scenarios and needs.
Another advantage of our evaluation series is that everyone is welcome to participate.
We began this issue by saying: the next generation of models needs the next generation of benchmarks. Then perhaps we should build those benchmarks together.
If you have run into a pitfall while using agents, if there is a place where you most want models to improve, or if there is a capability that still feels unreliable today, you can propose it as the next exam question and directly participate in the evaluation process.
Evaluation has never been just about scoring. It is about telling every user which direction models should evolve toward.
Summary
The six benchmarks in this issue all replace the old question “can the model answer the question?” with a harder question: “can the system get things done in the real world?”
UXBench asks whether users are satisfied. MemLens asks whether the model can remember, and whether it knows when not to answer. RoadmapBench asks whether it can complete a version upgrade. APB asks whether the plan itself is wrong. K-BrowseComp asks whether the model still works in another language. Silicon Star Eval asks how far models are from real-world scenarios.
They are no longer testing only knowledge points. They are testing the ability to create experiences, preserve memory, execute over long horizons, localize, plan, and actually “do things.”
More importantly, they are no longer satisfied with giving a single score. They are beginning to provide failure types: telling us whether the model is stuck in writing, retrieval, reasoning, planning, or execution.
Evaluation is moving from “scoring” to “diagnosis.” That is the real turning point of this generation of benchmarks.
The direction of model evolution should not be defined by an outdated leaderboard that happens to go viral, but by better questions.
And the people who raise these questions should not be only labs. They should also include you, the people who are actually using these systems.