🔍 Read the full analysis: Which AI Harness Fits Your Needs: CUGA, Hermes, Or Deep Agents? on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
IBM reports that CUGA completed 73 of 90 AppWorld tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The test suggests harness configuration can affect results on this workload, but it does not establish a general ranking or show how the systems perform on production enterprise tasks.
IBM reports that its CUGA agent harness completed 73 of 90 AppWorld tasks, compared with 42 for Hermes and 27 for Deep Agents, when all three used the same open-weight model, GLM-5.3-flash, as discussed in the original analysis. The results point to harness design as a factor in performance on this particular multi-step benchmark, but do not establish an overall winner across tasks or real-world deployments.
The comparison ran each harness on the same 90 tasks across 30 scenarios, using the Harbor evaluation framework and one attempt per task, according to IBM. The reported task completion rates were 81.1% for CUGA, 46.7% for Hermes and 30.0% for Deep Agents. On a separate scenario measure, which counted a scenario only if all three of its variants passed, CUGA passed 20 of 30 scenarios, Hermes eight and Deep Agents two.
The harnesses did not use identical configurations. CUGA used task decomposition, app-focused tools, verification before execution and automatic reflection. Hermes ran with Harbor’s default setup and AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP alongside LangChain middleware for tool selection, context summarization, to-do lists, retries and call limits. IBM says it made two fixes to Deep Agents so malformed model replies would not stop runs.
IBM says the experiment setup is available in harbor-eval, its Harbor fork. The supplied material does not include an independent replication or comparative figures for latency, cost, or performance under repeated attempts. The scores are outcomes from the stated benchmark setup, not evidence that the harnesses would perform in the same order on other workloads.
Why Harness Design Affects Scores
An agent harness coordinates the model’s work: it can divide a task into steps, select tools, manage context and check actions. The gap in IBM’s results suggests that those choices can materially affect whether an agent completes a particular sequence of tasks, even when the underlying model is held constant. For organizations comparing agents, a model score alone may not predict how a configured system will perform on their work.
The practical implication is to evaluate the model, harness and workload together. A team whose workflows involve navigating APIs across several applications may find AppWorld’s results relevant as a signal to test harness choices. But the scores do not show that CUGA will be better for personal assistance, research or other jobs associated with Hermes and Deep Agents.
The benchmark also does not test policy enforcement. IBM describes CUGA as a harness with capabilities for applying rules, but the AppWorld experiment did not use policies and therefore offers no evidence about how effectively any system enforces them. Security controls and approval requirements need their own evaluation.
Products Worth Considering
EcoFlow Delta 3 Ultra 3072Wh Portable Power Station – 3600W AC Output
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
Amazon Kindle Scribe Colorsoft, 64GB, 11-Inch Color Display with Premium Pen
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Work
The comparison is framed as a question of fit, not simply a contest. The source describes CUGA, short for ConfigUrable Generalist Agent, as IBM’s open-source harness for enterprise work across systems such as customer relationship management and payments. Hermes, from Nous Research, is presented as designed for always-on personal assistants, while LangChain’s Deep Agents is described as suited to long, multi-step work such as research.
AppWorld tests multi-step tasks across nine apps and 457 APIs, including email, payments and file storage. IBM characterizes the benchmark as a proxy for some capabilities relevant to enterprise automation, such as reading API documentation, paging through results and changing state across apps. However, the benchmark uses consumer apps and does not directly establish performance on real enterprise systems.
The source also points to community directories that track nearly 200 agent harnesses and cites a separate directory analysis in which a GLM-5.2 score on SWE-bench Pro rose from 23% to 52% after a harness change. That is a distinct reported comparison, not part of IBM’s AppWorld test, and should not be treated as independent confirmation of the AppWorld findings.
“This is not a verdict on Hermes or Deep Agents.”
— IBM, describing the AppWorld results
What the AppWorld Test Leaves Open
The results cover one model, one 90-task sample and one attempt per task. It remains unclear whether the performance gap would hold with other models, different task sets, repeated runs or production workloads. The harness configurations also differed, so the test does not isolate every implementation choice as the cause of the score differences.
The supplied material gives no comparative data on cost, latency, reliability at larger scale or deployment outcomes. It also does not show how the systems handle enterprise access controls, approval steps, data boundaries or policy enforcement. Those issues are distinct from task completion and are not established by the AppWorld scores.
How Teams Can Test Their Workflows
Teams considering these systems can use IBM’s public harbor-eval setup to examine the reported experiment, then run evaluations on workflows that resemble their own. A useful comparison would hold task conditions steady, document each harness configuration and include repeated or varied tasks to assess consistency as well as successful completion.
For enterprise use, evaluation would also need to cover access controls, approval requirements, data boundaries and performance in the intended deployment environment. The supplied material gives no timeline for a broader comparison or independent replication, so it remains unclear when further evidence may become available.
Key Questions
Which harness completed the most AppWorld tasks?
In IBM’s reported test, CUGA completed 73 of 90 tasks. Hermes completed 42 and Deep Agents completed 27, with all three using GLM-5.3-flash.
Does the test prove CUGA is the best agent harness?
No. The result is limited to one benchmark, one model and one set of configurations. It does not establish a general ranking across different tasks or deployments.
What does AppWorld measure?
AppWorld tests multi-step work across nine apps and 457 APIs, including tasks involving email, payments and file storage. IBM describes it as a proxy for some enterprise-automation capabilities, but it uses consumer apps.
Did the experiment test security policies or enterprise controls?
No. The supplied description says the experiment did not use policies. It therefore does not demonstrate how well the systems enforce rules, access controls or approval steps.
Where can teams review the experiment setup?
IBM says the setup is publicly available in harbor-eval, its Harbor fork. Teams can use it as a reference while also testing workflows and configurations relevant to their own work.
Primary source: Hugging Face · via ThorstenMeyerAI.com
