TL;DR
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Z.ai says its GLM-5.3-FlashX-Legal model achieved a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark. Vals AI’s leaderboard lists the model seventh among 81 entries, while the benchmark’s strict scoring rule means a task fails if even one required criterion is missed. Z.ai’s open-source comparison and a small reported citation check require additional context.
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on the all-criteria task pass rate of Harvey’s Legal Agent Benchmark, placing seventh among 81 models on Vals AI’s leaderboard. The result measures how often a model met every requirement in a task, so it does not mean the system met only 19.17% of all individual rubric criteria.
Vals AI’s archived leaderboard lists GLM Legal at 19.17%, rounded to 19.2% in Z.ai’s documentation, and gives a test cost of $0.15. Z.ai says the result was independently evaluated by Vals. The company’s documentation describes the model as optimized for professional legal workflows, including document understanding, legal reasoning, research, contract review and drafting initial work product.
Z.ai says the model was built on GLM-5.3-Flash and received legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments modeled on legal work. Its October 13 AutoClaw blog post opened a beta by application. Z.ai recommends using the model as an agent connected to services that provide laws, cases and company information. Its pricing page lists charges of $0.37 per million input tokens and $1.25 per million output tokens.
The reported run also met 92.5% of individual criteria, according to the source material. Z.ai says its score is 12.5 percentage points above the base model. That comparison is consistent with the same leaderboard figures cited in the material. The company also calls its result state of the art among open-source models, but the cited documentation does not specify the comparison set, and the source material reports no published weights for this legal model.
How Z.ai’s Legal AI Ranks Near The Top Despite Passing About 1 In 5 Tasks
GLM-5.3-FlashX-Legal passed every required check on 19.17% of benchmark tasks. Its ranking is strong within the leaderboard, but the score reflects a strict all-criteria rule.
Why the task score is lower
A task counts as complete only when every required check passes.
A work product can include requirements for facts, citations, recommendations, deadlines and formatting.
One missed criterion is enough for the whole task to fail the all-pass rule.
The 92.5% figure counts individual checks met; 19.17% counts tasks with no missed checks.
What the benchmark covers
Harvey LAB evaluates legal work products against a large set of expert-written checks.
Broad legal work
The public set has 1,250 tasks across 24 practice areas.
Binary checks
More than 75,000 rubric criteria check details such as conclusions, citations and deadlines.
Private holdout
Vals AI reports 120 tasks, with internet access disabled and scores averaged across two model judges.
What the result tells buyers
The benchmark offers evidence about a defined evaluation, not a guarantee for a firm’s own matters.
- Ranks seventh among 81 leaderboard entries.
- Often meets individual rubric checks.
- Z.ai positions outputs as drafts for legal professionals.
- Task-by-task results are not provided.
- The open-source comparison set is unspecified, and the source material reports no published weights.
- Client-data terms and performance on a firm’s own work remain unresolved.
Practical evaluation
Z.ai recommends connecting the model to services that provide laws, cases and company information.
- Review the benchmark methodology and procurement terms.
- Test the model on the firm’s own matters and documents.
- Verify citations and have lawyers review work product.
Why Legal Task Scores Look Low
The score matters because legal work products often depend on several requirements being satisfied together. In Harvey LAB, a report can fail as a complete task even when it gets most of its component checks right. A missed citation, deadline, recommendation or formatting requirement can be enough to prevent an all-pass result. The 19.17% figure is therefore a measure of fully completed benchmark tasks, rather than a direct estimate of the model’s overall accuracy or usefulness.
That distinction cuts in two directions. The 92.5% individual-criteria rate suggests the model often met many rubric checks, but it does not establish that a lawyer could rely on a given output. A task with one important error still fails the benchmark’s all-pass rule; in practice, the importance of an error depends on the matter and the work product. Conversely, describing the result as simply “failing four out of five tasks” can obscure how close some outputs may have come to passing.
For firms considering legal AI, the result is one piece of evidence about performance under a defined evaluation. It does not settle whether the model is appropriate for client work. Review of citations and conclusions, data handling, and procurement terms remain practical questions for buyers. Z.ai describes the output as drafts for legal professionals, according to the source material.
Products Worth Considering
Apple Watch SE (2nd Gen) GPS 40mm with Starlight Aluminum Case and Sport Band (S/M)
DJI Mic 3 Wireless Microphone Kit (2 Transmitters, 1 Receiver, Charging Case)
Apple Watch Series 8 GPS, 41mm, Midnight Aluminum Case with Midnight Sport Band (S/M)
As an affiliate, we earn on qualifying purchases.
How Harvey LAB Scores Work
Harvey released the Legal Agent Benchmark (LAB) as open source on May 6, 2026. Its public set contains 1,250 tasks across 24 legal practice areas. Each task combines an instruction, a client matter containing relevant materials and a requested work product. More than 75,000 expert-written rubric criteria cover items such as facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts and formatting.
Harvey describes those criteria as atomic, binary pass-or-fail checks. Under its all-pass grading rule, every required criterion for a task must pass for the task to count as complete. Harvey has said that identifying eight of ten risks in a deal-team report does not make the report 80% complete if the missing risks leave it materially incomplete. The example in the source material illustrates the breadth of a task: one change-of-control assignment has 57 criteria.
The public leaderboard uses a private holdout set intended to mirror the public task mix. Vals AI says that set contains 120 tasks and that internet access is disabled. Two language-model judges, GPT-5.5 and Claude Sonnet 4.6, grade each submission; the final score averages their task pass rates. These design choices describe the conditions behind the reported result. The score is tied to this benchmark, its task set and its judging process.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%.””
— Z.ai documentation
What the Benchmark Cannot Resolve
The available material does not identify which individual tasks GLM Legal passed or failed, or provide a task-by-task breakdown of its 92.5% individual-criteria result. It also does not specify the comparison set behind Z.ai’s description of state-of-the-art performance among open-source models. The source material reports no published weights for the legal model, leaving that open-source characterization difficult to assess from the cited pages alone.
The article’s author reports a four-task spot check, explicitly described as too small to serve as a benchmark. In that check, the legal model cited rulings the author could not find. The author says the tested OpenAI and Claude models invented no rulings, while two Claude models cited a real ruling for the opposite of what it held. These observations are limited to that check and do not establish error rates across the benchmark or typical use.
The source material says the Z.ai pages reviewed describe outputs as drafts for legal professionals and carry no “not legal advice” notice. It does not provide detail on client-data processing, retention, or applicable contractual and regulatory terms. Those details, along with how firms should verify citations before using work product, remain unresolved in the material.
Beta Access and Buyer Checks
Z.ai’s AutoClaw post says the beta is open by application. The source material does not give a broader release date or say when the model might become generally available. Further leaderboard entries or updated benchmark results could change its position, but no future update is specified in the cited material.
For organizations evaluating the model, the immediate next steps are to review the benchmark methodology, check the terms governing client data and test the system against the firm’s own work. Citation verification and lawyer review are especially relevant given the reported spot-check observations and the benchmark’s strict treatment of missed criteria. The available information does not establish how the model performs across a firm’s particular matters, document sets or review practices.
Key Questions
What does the 19.2% score mean?
It is the share of benchmark tasks on which GLM Legal passed every required rubric criterion, according to the reported result. A task with one or more missed criteria does not count as complete.
How can the model meet 92.5% of criteria but pass 19.2% of tasks?
The figures use different units. The 92.5% figure refers to individual checks across the run, while the 19.2% figure counts tasks that passed all checks assigned to each task. A high rate on individual checks can coexist with a lower all-pass task rate.
Where did the model rank?
Vals AI’s archived leaderboard lists GLM Legal seventh among 81 models, with a score of 19.17% and a listed test cost of $0.15.
Does the result show that the model is ready for legal work?
No. The result describes performance on a particular benchmark and does not establish suitability for a specific firm or matter. Z.ai describes the system’s output as drafts for legal professionals, according to the source material.
When can users try it?
Z.ai’s AutoClaw post, dated October 13, 2026, says the beta is open by application. The source material does not state when general availability will begin.
Source: Thorsten Meyer AI
