🔍 Read the full analysis: We Asked 42 AI Setups About Law—The Built-for-Law Model Scored 8/100 on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Lexassay, a legal AI test run by Thorsten Meyer AI, scored GLM-5.3 FlashX Legal at 8 out of 100 across 15 questions, with zero points on the hidden set. The test also found citation errors across published model runs, but its author calls it a spot check rather than a strict benchmark and discloses that Claude agents graded the answers.
GLM-5.3 FlashX Legal, a model marketed by its maker Z.ai for professional legal workflows, scored 8 out of 100 on Lexassay, a test of 15 legal questions run by Thorsten Meyer AI. The test’s author says the result is a limited spot check, not a definitive ranking, and discloses that Claude agents graded the answers while a Claude model placed first.
Lexassay tested 42 AI setups on questions covering German, US and EU law. They included deadline calculations, recent legal changes, court decisions and questions built around false premises. The models answered without internet access or external tools. The test used ten public questions and five hidden ones; only the score for the hidden set is published.
According to the published results, GLM-5.3 FlashX Legal scored zero on the hidden set and fully solved none of the 15 tasks. It produced no answer on two German-law questions after using its full 64,000-token thinking budget. Its citation check recorded 21 real citations, 25 misattributed citations and one invented citation. The underlying GLM-5.3 FlashX model scored 3, according to the article.
The report says Mistral Large 4 scored 10 out of 100, with errors in citations accounting for much of its lost credit. Claude Opus 5.5 led the reported results with 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the leader met every criterion on only eight of the ten public questions, the report says. Across all published runs, Lexassay reports verifying 2,644 citations and finding 804 that were invented or did not support the model’s stated claim.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Citation Errors Matter
The results highlight a practical risk for people using chatbots to interpret legal rules: an answer can sound confident while citing a decision or statute incorrectly. In the test’s scoring system, false or misattributed citations reduced task scores, but the larger concern for a user is whether an error could affect a deadline, claim or legal filing.
The article illustrates that risk with a German civil-procedure question involving judgments served on April 27, 2027, in Munich and Berlin. Because Corpus Christi is a public holiday in Bavaria but not Berlin, the report says the appeal deadline falls on May 28 in Munich and May 27 in Berlin; the deadline for written grounds is June 28 in both. The example is part of the test, not advice for any reader’s case. The report warns that a late appeal can be rejected as inadmissible and advises readers to verify legal citations and not rely on a chatbot for a deadline.
Products Worth Considering
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Hazelnut Taupe
Logitech MX Master 4 for Mac Wireless Ergonomic Mouse, Space Black
Apple Watch SE 3 GPS Smartwatch, 40mm
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
The test grades answers against a prepared reference answer with three to six weighted criteria. Each citation is checked against official databases and classified as real, misattributed, invented or not findable. Lexassay also deducts points when an answer lacks a legal-advice disclaimer or direction to consult a lawyer. Its method page publishes the rules, answers, reference answers and citation checks, according to the report.
The score is not a measure of how a model performs with legal research tools or supplied case documents. Lexassay deliberately tested models without internet access or tools, aiming to measure what they could answer on their own. Effort settings were tested separately, and a model may therefore appear more than once among the 42 setups. The author says there was one answer per model and task, meaning small score differences may reflect noise.
The article also notes that a separate Vals AI legal-agent benchmark, where models work with supplied documents, ranked the same models differently: Mistral Large 4 placed eighth, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th. The differing results reflect distinct test conditions and should not be treated as directly interchangeable rankings.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in its developer documentation
Limits Behind the Scores
The results do not establish how the models perform across the full range of legal work, with browsing enabled or when supplied with relevant documents. The test includes only 15 questions and one answer per model and task; the author says small differences may be noise. The published scores are therefore evidence about performance on this particular test, not proof that a model is generally suitable or unsuitable for legal work.
There is also a disclosed conflict of interest: Thorsten Meyer AI runs Lexassay, and Claude agents graded the answers while a Claude model ranked first. The operator says the method page identifies this conflict and that the answers, reference answers and citation checks are published for review. The source material does not provide independent replication of the scores or explain how the same grading process would perform under an external audit.
The source identifies GLM-5.3 FlashX Legal as seventh among 81 on another measure, but the supplied excerpt cuts off before explaining what that ranking refers to. It cannot be reconciled with the 8/100 result from the information available here. The full details behind that ranking, and whether any models or results have since changed, remain unclear.
Review the Published Test Records
Readers seeking to evaluate the findings can consult Lexassay’s published method, public answers, reference answers and citation checks, which the operator says are available for review. A further comparison would need repeated runs, independent grading and clearly matched test conditions before small differences between scores could be treated as meaningful.
No follow-up test or response from Z.ai is specified in the supplied report. For now, the operator’s guidance is to use AI answers only as a starting point, check each cited law or ruling against the original source, and consult a qualified lawyer about a real deadline, claim, contract or communication with an authority. This article describes a test and is not legal advice.
Key Questions
What did GLM-5.3 FlashX Legal score?
It scored 8 out of 100 on Lexassay’s 15-question test and received zero on the hidden set, according to the report.
How many legal questions were in the test?
Lexassay used 15 questions: ten public questions and five hidden questions. The test covered German, US and EU law, and the models answered without internet access or tools.
Does the test prove that the model cannot do legal work?
No. The author describes Lexassay as a spot check, with one answer per model and task. Its results do not establish performance across all legal work or in tool-assisted settings.
Who graded the answers?
According to the report, several Claude agents graded answers against reference answers. Thorsten Meyer AI discloses that it runs Lexassay and that a Claude model ranked first.
Should I use a chatbot to calculate a legal deadline?
The report advises readers not to rely on a chatbot for a deadline. Check the applicable rules and cited sources, and contact a lawyer about a real case or filing.
Source: ThorstenMeyerAI.com
