For four months, we improved everything around the AI model: the ontology, the semantic layer, the agentic harness. Accuracy climbed, then it stopped climbing. In October, we changed a single piece: the model. Here is what the numbers say.
This article follows an experiment run in real conditions and measured every night on the same business questions. Two phases, one method, and a lesson that goes beyond our case: a good architecture moves AI forward up to a point, and then the model's capability sets the limit.
We changed a single piece. Accuracy went from 87% to 98%.
The context: a graph, an ontology and an agent
Graphicient connects engineering and manufacturing data in a knowledge graph, guided by an ontology. On top of it, a conversational agent answers business questions in natural language: “which vehicles use this part?”, “how many battery references are fitted on vehicles built in India?”.
The ground is demanding: the bill of materials (BoM) of a major carmaker. More than 10 million connected objects, up to 15 levels of nesting in a bill of materials, and around 1,400 business terms and synonyms in the ontology. Here, a plausible answer is not enough: an approximate number is a wrong number.
The agent never works alone. Four building blocks surround it, and each one was built to fix a measured failure.
The problem: a ceiling the architecture could no longer break
From late June to early October, the model at the heart of the agent was Mistral Medium 3.5. An efficient, fast and affordable model, which we surrounded with all our attention.
- A richer ontology and a semantic layer: business words find the right concepts. In our September test, 24 business concepts out of 24 were found at first rank.
- A stronger agentic harness: around fifteen automatic fixes and checks on every query, tools run in parallel, know-how written by the business teams themselves. On hard questions, failed queries fell from 44% to 7%.
- Controlled memory: the model receives a precise, stable briefing rather than raw data. Most of it is served from cache, which makes every answer faster and cheaper.
Results followed. Accuracy rose from about 76% in late June to 87% in early October. The average time per question was divided by three, from about 110 s to 37 s, and token consumption fell by a third.
Then the curve flattened: evaluation after evaluation, accuracy plateaued around 87%. Each new rule fixed one question… and weakened another.
The harness could no longer make up for what the model could not do.
The remaining errors had something in common. They required holding a long line of reasoning without losing the thread: chaining several levels of a bill of materials, crossing two sets, following a business rule to the end. The model skipped a step of the method, or forgot a rule along the way. On the Artificial Analysis Intelligence Index, an independent evaluation of models' reasoning capability, Mistral Medium 3.5 scores 14 points [1].
The proof: measuring every night, on the same questions
To tell real progress from an impression, we evaluate the full agent, not the model alone, on a set of reference questions. The method did not change between the two phases.
Two identical evaluations can vary by 3 to 5 points: we read trends, not isolated scores. Phase 1 reads as a steady climb up to a plateau. Phase 2 reads as a change of level.
The twist: a new model, and everything else unchanged
In mid-October, a new model became available in our environment: GLM-5.3, an open-weight model released by Z.ai in August 2026. On the same Intelligence Index, it scores 34 points in its most economical setting, “low effort”, against 14 for Mistral Medium 3.5 [1].
GLM-5.3 · max effort45
Mistral Large 4 Preview38GLM-5.3 · low effort (phase 2)34Mistral Medium 3.5 (phase 1)14We plugged it in as is. Same ontology, same tools, same instructions, same 98 questions, same judge. Both models then ran on the same day, one after the other, on the same question set.
Mistral Medium 3.5GLM-5.3 (low effort)And the result held over the following evaluations: 98% correct answers on average.
More accurate, and faster. With short reasoning (“low effort”), the model makes fewer round trips (3.6 steps per question against 4.6) and goes straight to the point. It uses about a third fewer tokens, for an estimated cost around 35% lower per question at public prices [1].
The work on the ontology and the harness is not lost: it is what lets the new model reach 98% on day one.
This is the most important lesson of the experiment. The model is an interchangeable piece; structured knowledge stays. The day a more capable model arrives, it immediately benefits from everything built around it.
The outlook: a benchmark to reinvent
At 98%, our reference set no longer makes the difference. A test you pass almost perfectly mostly measures what you already know how to do. The next step is therefore to raise the complexity.
- Harder questions: multi-step analyses, crossing the engineering and manufacturing bills of materials, calculations on large volumes.
- More demanding checks: verifying that answers are consistent with each other and with business rules, not just the right number.
- Ambiguous questions: checking that the agent knows how to ask for clarification rather than guess.
A benchmark is not a trophy. It is an instrument, and it must stay more demanding than the system it measures.
The vision: an agent that can carry more
A more capable model is not only useful for answering the same questions better. It lets us entrust more to the agent: longer tasks, more tools used in parallel, analyses that are computed rather than estimated.
Our conviction stays the same: lasting value lies in structured knowledge and in the guardrails around it. Models will keep improving; a well-designed architecture lets you benefit from them on day one.
Your turn: which question can your graph not answer yet?
Sources
- [1]Artificial Analysis — GLM-5.3 (low) vs Mistral Medium 3.5: Intelligence Index v4.3.2 (34 vs 14), public prices per million tokens (input, output, cache). Retrieved on October 14, 2026. artificialanalysis.ai
- [2]Artificial Analysis — Models leaderboard: Intelligence Index scores for GLM-5.3 (max), Mistral Large 4 Preview and the best model on the leaderboard. Retrieved on October 14, 2026. artificialanalysis.ai/leaderboards
- [3]Graphicient — internal evaluations of the agent on 98 reference questions from a major carmaker's bill of materials, 50 complete evaluations kept (out of 55 run) from June to October 2026, traced in Langfuse.