GenAI Assembling Eval Eps. 1 | “AI Predicted 98% of the Exam”? We Had 8 AIs Predict the Gaokao Math Paper
Every year, in the month before China’s national college entrance exam, the Gaokao (China’s notoriously grueling, high-stakes academic exam that determines university admissions for millions of students), the same thing goes on sale across the internet: prediction papers.
The most outrageous example in 2025 was a new-media claim that its “AI prediction paper” had a 98% hit rate. Later, official fact-checking platforms in Shanghai and the China Association for Science and Technology both pushed back: gaokao questions are kept under strict confidentiality, AI cannot access the confidential exam-question data, and because old “prediction papers” usually hedge in every direction, a real AI hit rate is essentially impossible.
Prediction papers are anxiety packaged as a product. What we wanted to do was reverse-engineer that anxiety and puncture it.
This year, shortly before gaokao, the GenAI Assembling AI frontier team sent the same prompt to eight mainstream global AI agent products. Each was asked to do three things: analyze the question-setting patterns in recent Beijing math papers, predict what the 2026 paper might look like, and then produce a full 2026 mock paper from scratch. We then anonymized and shuffled all eight papers, asked the eight AIs to grade one another, and finally invited a math teacher who has taught many rounds of Beijing high-school seniors to review each paper after the real exam and check the actual hit rate question by question.
There was no “98%.” The teacher’s judgment was that, excluding the relatively fixed, low-difficulty points in the multiple choice, fill-in-the-blank questions, and the first question of the long-answer section, the knowledge points genuinely predicted by all AIs added up to barely more than 20%.
This is the second issue of our Agent Eval series. In the previous issue, we predicted whether Google I/O would leak any cues. This time, the Gaokao prediction was a sealed box: there was no standard answer in advance, and the models had to manufacture new questions. Before explaining how we tested it and why we chose Gaokao math, here are the results.
Who Predicted More Accurately?
After the June 7 exam, we scored the eight systems from two angles. The first was objective: did their predictions hit the knowledge points in the real paper, item by item? The second was subjective: we asked the math teacher to grade each paper on how many genuinely useful highlights it contained and how good the paper felt as a whole.
First, the objective hit rate. Across 21 questions, counted by knowledge points that appeared in the real exam, the ranking was:
The separation was more obvious than we expected: the difference between 9 questions and 4 questions was more than twofold. Everyone hit the fixed knowledge points, so there was little distance there. The real gap came from the dozen or so floating, non-fixed items in the middle.
Two results were especially interesting: one was a correct bet, and one was a collective mistake.
The correct bet was on T21, the final question. The real paper included a new-definition problem about a ±1 number table. Its direction was combinatorics rather than sequences. Before the exam, Claude, Gemini, Genspark, and Manus had already moved away from “still a sequence problem” and toward “combination.” ChatGPT, MiniMax, and Kimi insisted it would still be a sequence problem. GLM did not generate anything close to the new-definition pressure of the real final question and simply produced an ordinary derivative problem.
The collective failure was on T17 and T18. The real exam swapped the expected roles of these two long-answer questions: T17 became probability and T18 became solid geometry. No model fully predicted the swap. Most systems still followed the old pattern: T17 solid geometry, T18 probability, and all lost points on those two major items.
Then came the teacher’s subjective highlight scores.
The teacher’s comments included a few representative lines: Genspark’s Question 8 hit a similar problem type, its long-answer questions felt authentic to the gaokao style, and the probability background was rich. Gemini did not merely imitate - it even adapted and reworked a 2022 gaokao problem by changing the angle, increasing the difficulty, and shifting the derivative focus; the teacher called it the hardest of the eight papers. MiniMax had the best model for the ellipse long-answer question, but the derivative question would not have been that simple at the end of senior year. Claude’s Question 10 appeared to be a 2022 gaokao question with only a few numbers changed. ChatGPT’s derivative problem looked intimidating at first glance, but with a little calculation turned out to be very simple. GLM’s paper came with reference formulas on the front page; its long-answer section still used arithmetic sequences, and its analytic geometry question looked like a parabola problem. The teacher suspected it may have crossed over from another regional exam style.
Putting the two rankings side by side was revealing. Genspark ranked first on both lists; GLM was at the bottom on both. But the middle group was more interesting. Kimi tied for first in hit rate, yet only scored 60 in teacher quality. Gemini’s hit rate was middle of the pack, but its paper quality ranked first. Predicting accurately and writing a good paper turned out to be two different things.
A Few Unexpected Findings
AI Was Not Collectively Narcissistic
We anonymized the eight papers, labeled them from Paper 1 to Paper 8, and sent them back to the eight AI systems. Each system was asked, as an education researcher, to grade the papers and rank them. Would they secretly give themselves high marks?
To make the question cleaner, we put several layers of separation in place. We removed traces of source files from every paper, standardized the layout, and made sure the models could not identify which paper they had generated. All reviews were conducted in fresh conversations with memory turned off and privacy mode enabled. We did not let a model carry any memory like “I wrote a paper last week” into the grading process. Internally, we kept a control table mapping each paper number to its true author, and watched the diagonal of the matrix to see how each system ranked itself.
Self-preference in large models is a well-known issue: when asked to judge a pile of content that includes their own outputs, models often unconsciously look more favorably at their own work. After anonymization, that bias almost disappeared - and that was exactly what we wanted to test.
Of the eight AIs, only one ranked itself first. That one was Genspark, and in this case its confidence was not unreasonable: its paper was also the consensus champion among the group, with six other systems ranking it in the top two. In a sense, even this “narcissist” was backed by peer consensus.
More surprising was the opposite direction. GLM ranked its own paper last overall, eighth. Kimi ranked itself fifth. The rest generally placed themselves in the middle and did not obviously push themselves upward. Excluding Genspark as a special case, the average self-ranking of the remaining models was even lower than random expectation. No one clearly defended its own identity.
For these general agents, this was not the legendary self-preference problem. If anything, they were somewhat self-restrained. A more precise way to put it is that they could see the rough edges of their own work. GLM’s paper really was hard and off-style; Kimi itself clearly saw that its three-year data was insufficient. Under anonymity, being able to accurately point out the shortcomings of one’s own paper is actually a judgment worth acknowledging.
There was one outlier: ChatGPT. It went against everyone else and ranked the consensus champion, Genspark’s paper, sixth, while ranking its own paper first. Even the AIs could not agree on grading aesthetics.
One PDF Tested Who Was Honest
The PDF we fed the eight systems contained real exam papers, but two years - 2021 and 2024 - were scanned images. A machine could not directly extract text from those pages. That was an error on our part, but by accident it became the most unexpected part of the evaluation. It posed a very common real-world challenge: the information in front of you is incomplete. What do you do? The agents’ reactions revealed whether they were honest and reliable when faced with flawed input. The eight responses clearly fell into three groups.
The honest group had one member: Kimi. At the beginning of its report, Kimi explicitly stated that it had only read three years - 2022, 2023, and 2025 - and could not access 2021 or 2024. It did not make up five years of analysis just to look complete. That gave it less information, but it also did not pretend.
The middle group included GLM, Manus, and MiniMax. They all claimed to have analyzed five full years. When we checked their annotations for 2021 and 2024, they were actually correct. For example, GLM marked 2021 Question 18 as “checking the probability of nucleic acid testing” and Question 6 as “arithmetic sequences in party-flag specifications,” matching the real paper closely. This suggests they used other methods, such as visual recognition or web search, to fill the gap, or drew on stored knowledge. Their capability was sufficient. The only issue was that throughout the process they never said, “These two years were images, so I found another way to access them.” It made everything look smooth, but skipped a communication step that should have existed.
The most interesting group had one member: Gemini. At first we saw no issue. Only after we asked how it had read the PDF did it admit that it had not truly parsed the file. Instead, it had relied on its training-time memory of Beijing exam questions. The earlier “five-year analysis” was not based on the material we provided. In practice, this is a subtle kind of hallucination: you think it is reading your document, but it is actually free-associating from memory.
Who Was Careful, Who Was Lazy, and Who Crossed the Line?
A quick rundown of the eight systems’ process behavior:
ChatGPT (GPT-5.5 Thinking Extended) was the most conservative. It directly produced a neatly formatted PDF test paper that was ready to use. Its predictions felt “textbook-like,” its structure was logically closed, and the solution writeups were complete. It was like the detached student in front of the class during peer review: it suppressed the consensus champion to sixth place. The questions it generated were conventional and steady, with little flair, but almost no mistakes.
Claude (Opus 4.8 Max) was the most careful. To render mathematical formulas cleanly, it designed its own pipeline: generate Markdown, convert it into MathJax-based HTML, and finally print it through a browser into a PDF. Its thinking time was unexpectedly long. That scholarly diligence also appeared in peer review: it was the only one that manually solved every question and found a math error hidden in another paper, behaving like a veteran teacher revising exam papers.
Gemini (3.1 Pro Extended) loved wrapping questions in tech scenarios. Compute costs, neural-network nodes, robot tests - the questions never left the frontier. It was also the system that failed to read the earlier PDF and answered from memory. In its own paper, some formulas were not properly rendered and were left as unresolved code-like symbols, revealing rough edges.
Genspark (Ultra Mode, powered underneath by Claude Opus 4.7) was the standout paper this time and the consensus champion. Its paper had almost no mathematical errors, the only one paper with no point deductions, and strong situational design. It accurately picked up the recent trend of battery degradation, low-altitude economy, drones, and autonomous driving reliability, translating those into plausible gaokao-style contexts. When faced with the incomplete PDF, it was also honest: it proactively said it had not read everything and asked whether it could search the web. After we agreed, it searched openly. Its small flaw was that for the 2025 score-distribution structure, we could not trace enough source support and suspected a possible local reconstruction.
GLM (GLM-5.1) produced the most exam-like formatting: question numbers, scores, and layout all looked polished. But it was also the universally recognized bottom performer. Both the AI peer jury and the teacher placed it last. The formatting was polished, but the content felt imported from another regional exam style. The front page included reference formulas, a habit more associated with Shanghai papers. The long-answer question still used arithmetic sequences, closer to national-paper conventions. Beijing papers do not usually use that style for analytic geometry parabola questions. Its multiple-choice options repeatedly appeared as “A,” which was an obvious formatting bug.
Kimi (k2.6-agent) was honest, but looked like a diligent executor that did not take one extra step. After discovering that two years of material could not be read, it simply continued downward without trying another method. It lacked initiative. Its paper looked decent, but it was too simple. Because it only read three years of data, it was the only system that predicted T16 and T17 in the reverse order, biased by the 2023 T16/T17 swap.
MiniMax (MiniMax-M3) produced the prettiest and most regular template. It could be used directly for a typeset exam paper. But it was also the slowest among the four Chinese models and ran for a long time. The questions were simple; the teacher said its derivative question was closer to a post-lesson exercise for the end of the second semester of senior year. It also made a detour: the paper started as Beijing, drifted toward Shanghai, and even inserted its own product name halfway through.
Manus (Manus 1.6 Max) was balanced and complete, with no obvious outliers, but also no major strengths. In this batch of generally simple outputs, its solutions were judged by the teacher as one of the more reasonable sets. It was, in effect, the best of the “ordinary” group.
Another small habit became popular: six of the eight papers tried to insert AI, computing, new energy, or other technology scenarios into the questions. Manus used a charging-pile coverage logarithmic model; Gemini turned neural-network layer nodes into sequences; Genspark asked students to compute the probability that an autonomous-driving algorithm A or B would be reliable. The most meta one was ChatGPT: it generated a probability problem in which models A, B, and C all solve the same math problem, then asked an AI paper to evaluate another AI paper. The reality is that Beijing has only used an AI-themed context once in the past five years. AI-generated exam questions love to cue themselves.
The Teacher Reviewed All Eight Papers. We Got Scolded Too.
Looking only at the scores, the results do not seem terrible. But then we listened to the teacher who reviewed all eight papers. His verdict was blunt: the papers were “overall too simple.” The papers generated by these AIs were not as hard as a paper for the second semester of the junior year.
That judgment did not stand alone. A study on AI-generated questions for high-stakes medical exams has found that AI-written questions tend to be simpler, more biased toward factual recall and other low-level cognition, and contain more factual errors. This matched the front-line teacher’s experience almost exactly.
The most worth mentioning is the three-way corroboration. The AI peer jury placed GLM at the bottom. The teacher, without knowing how the AIs had reviewed it, also graded GLM as the weakest paper, for reasons consistent with the earlier criticisms. The human expert, the AI peer reviewers, and our programmatic checks all independently pointed to the same bottom-ranked answer.
As for why the whole group did not perform well, the teacher offered four explanations: the reference set was too small, the systems could mostly imitate numeric forms, they could not generate genuinely new problem types, and they failed to create innovative combinations of knowledge points. The first two were visible in the data: ChatGPT and Genspark’s derivative long-answer questions nearly collided, and both had sources close to the 2025 real paper. The latter two were more experiential: models are biased toward high-frequency patterns and avoid low-probability novel structures.
The teacher’s most memorable line was: “If I write one, it will definitely be much better than what they wrote. But if I write it, I will absolutely be criticized too.” Because there are so few truly testable points in a prediction paper that even a human stepping into the ring may not score much higher. In other words, the pressure of gaokao prediction is not mainly on the AI side. The task itself is nearly unsolvable.
The Form Was There. The Spirit Was Not.
Even the systems criticized most harshly by the teacher could imitate the skeleton, question types, and score distribution of a Beijing math paper fairly well.
But once the final award distance was measured, the result was clear: no system truly hit the paper. Genspark ranked first in hit rate, AI peer review, and teacher score, but part of its edge came from proactively searching for additional real-paper data midway through. That is different from Gemini quietly not reading the PDF. Still, even Genspark was far from truly predicting a gaokao paper.
The eight systems could imitate the “shape” of a Beijing paper, but they could not create its “spirit.” The final challenge problem involving a new definition that appears every year and leaves candidates seeing stars - that is the soul of the paper, and also a blind spot for this AI collective. They can imitate old wounds, but it is still hard for them to create a new one. This time, the AIs did not cross that threshold.
Appendix: How We Tested
Why gaokao math? In our first evaluation, the press-conference prediction still left room for leaks along the industry chain. Gaokao questions are the opposite extreme: a genuinely sealed box. Outsiders cannot obtain any internal information. The only option is to infer patterns from past papers. Harder still, the AI must truly create questions. Search cannot help, and simple memorization is useless because the 2026 paper does not yet exist. Reading, inferring, and creating are all required, and if any step is unstable, the final paper collapses. This is where a “bookish” AI and a “thinking” AI begin to separate.
How did we test? We selected the same eight systems as in the previous evaluation, all set to their highest reasoning tier and allowed to use the web. Each received the exact same prompt and input materials: a collection of 2021-2025 Beijing gaokao math papers and analyses. The test had three stages. Stage 1: go through the papers and annotate knowledge points and patterns. Stage 2: predict each question type and direction for 2026. Stage 3: generate a full 150-minute mock paper.
How did we score? We fixed five dimensions. The first four could be evaluated before the exam: prediction logic, paper quality and difficulty, AI peer review, and PDF parsing/honesty. The fifth dimension, hit rate, was calculated only after the real paper came out.
The prediction logic itself was worth looking at. For the long-answer section, all eight systems appeared to refer to the same teaching-research consensus: T16 triangle, T17 solid geometry, T18 probability, T19 ellipse, T20 derivative, and T21 final challenge problem involving a new definition. Everyone hit this overall structure. Even the scoring structure was aligned. But for the short-answer questions, T3 through T14, the floating zone was chaotic: almost no question was predicted identically by all systems.
Note: The official Beijing math paper had not yet been released when the test began. The reference paper used here was reconstructed and cross-checked by several post-exam memories. Some detailed items may vary, but the knowledge-point framework is reliable. The hit rate and bright-spot scores were reviewed and approved by the math teacher. The scoring details and the eight original test papers are available on GitHub: https://github.com/pingwest-ai/agent-eval/tree/main/cases/EVAL-002-gaokao-math-2026