One News Pipeline, 71 AI Configurations, And A Choice To Make

🔍 Read the full analysis: One News Pipeline, 71 AI Configurations, And A Choice To Make on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he tested 71 AI model configurations on real news-writing assignments and selected GPT-6 Luna for production, despite another model scoring higher. In his October 13 report, Meyer said GPT-6 Astra earned the top marks, but cost about $0.12 per article, compared with about $0.0012 for Luna; he also removed two backup models after the test found fabricated quotations or unusable output.

The evaluation ran from September 8 to October 13, 2026, using the production prompt from Meyer’s news pipeline. Configurations included API and subscription services, free-tier options accessed through OpenRouter, and open-weight models running on Meyer’s own hardware. Each source was turned into an article in English and German, with two AI judges assessing fidelity to the source, fabrication, clarity about uncertainty, news value, structure and polish. German articles also received a language-quality score.

Meyer reports that GPT-6 Astra scored 4.00 out of 5 in both languages in the standard test and had perfect source-fidelity scores there. In a follow-up run of 12 articles, it averaged 3.93, with a reported 95% bootstrap interval of 3.83 to 4.02. GPT-6 Luna scored 3.67 in English and 3.68 in German in the standard test; across the robustness run it averaged 3.67, with source fidelity of 4.71 out of 5. Meyer said every Luna article was valid for his pipeline.

The cost and speed gap shaped the choice: Meyer puts Astra at roughly $0.12 per article and 50 seconds, versus about $0.0012 and 17 seconds for Luna. He also tested Luna at six reasoning-effort settings. The reported scores varied, but he said the differences were within the test’s noise; the highest setting cost about 12 times as much and took about 12 times as long as the production setting, with no clear quality gain.

Why Luna Won the Production Slot

The report highlights a practical trade-off for publishers using AI at scale: the top-scoring model may not be the most workable choice when the quality improvement carries a steep per-article cost. Meyer chose GPT-6 Luna because he judged its output consistently usable at a much lower price, not because it led the rankings. That decision reflects his stated threshold for this workflow: use the least expensive model that reliably meets the publication’s quality bar.

The test also points to a risk beyond the primary model. A failover system can send work to a backup when the preferred service is unavailable, but a cheaper or locally hosted fallback may still create editorial problems if it invents quotations or cannot return valid output. Meyer’s reported decision to remove two such models shows why publishers need to evaluate the entire model route, including backups, rather than only the headline choice.

These results are specific to Meyer’s prompt, sources, languages and judging method. They do not establish that Luna is the best model for other publishers, or that Astra’s higher score would be reproduced on a different article mix. The useful takeaway is methodological: benchmark models on the work they will actually do, and weigh quality against cost and reliability.

How the News Pipeline Was Tested

Meyer used two source examples: a 6,340-character product filing and a 619-character political news item. He said the shorter source was more representative of much of his production work and therefore the more demanding test of whether a model would stay within limited evidence. Each configuration produced English and German articles, which were assessed by GPT-6.1 Sol and Claude Opus 5.5. Meyer says both judges saw the original source.

The initial comparison was followed by a robustness run in which the strongest configurations produced 10 additional articles, mostly from short sources. Meyer included 95% bootstrap intervals and said that, with only two articles in the initial test, score gaps smaller than about 0.3 points should be treated as noise. He regarded the larger follow-up run as the more reliable basis for ranking. An article counted only if it produced valid output after the same repair step used in production.

The report also compared models across tasks. Meyer said Claude Opus 5.5, which he described as the strongest coding model on a separate benchmark, scored 3.46 in English and 3.39 in German as a news writer, with source fidelity of 3.75. He cited that result to argue that performance on coding benchmarks does not necessarily predict performance on news writing. The scores and model descriptions are claims from Meyer’s own evaluation, not an independent comparison of all available systems.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13 report

Limits of the Model Comparison

The report describes a publisher’s own test, not a controlled independent audit. The initial comparison used only two source examples, and the follow-up run covered 12 articles for the strongest configurations. Meyer acknowledges that small score differences may be noise and says the larger run should carry more weight. The article does not provide the full set of underlying scores, judge-by-judge results or article outputs for independent review.

It is also unclear how the results would change with other prompts, topics, publication standards or source lengths. The evaluation used AI judges, including Claude Opus 5.5, which Meyer also scored as a candidate writer. The report does not detail how judge disagreement was handled or whether human editors independently checked the ratings. Its conclusions should be read as findings for this specific pipeline rather than a general ranking of AI systems.

Meyer says GPT-4.1 nano produced invented quotations and that gpt-oss-120b yielded no valid articles in four attempts. Those are serious findings within the tested route, but the report does not establish how frequently those failures occur in other settings or with different prompts. It also does not identify Luna’s current fallback arrangement beyond stating that the two earlier backups were removed.

How Meyer Plans to Run Production

Meyer’s report says GPT-6 Luna is the production choice at the default reasoning-effort setting, which he labels “none.” He said actual production use over the prior week cost about $0.0013 per article, with a median generation time of 17 seconds. Astra remains the highest-scoring option in his test, but Meyer said he had decided on October 7 that its cost made it unsuitable for his news workflow.

The report does not announce a new test date or a replacement for the removed fallbacks. The next relevant developments would be further production results, a revised failover route, or repeat testing as models and prices change. For now, the published evidence supports only the decision Meyer describes: Luna will write the articles in this pipeline, while the route’s backup arrangements and the durability of the scores remain open questions.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It averaged 3.93 across 12 articles in the follow-up run, according to Meyer.

Why did Meyer choose GPT-6 Luna instead?

Meyer said Luna’s quality was consistent enough for production at a much lower reported cost: about $0.0012 per article, compared with roughly $0.12 for Astra. He also reported that every Luna article in the test produced valid output.

What happened when reasoning effort was increased?

Meyer tested Luna at six reasoning-effort settings and said score differences were within the test’s noise. The maximum setting cost and took about 12 times as much as the default setting without a clear quality gain.

Why were two fallback models removed?

Meyer said GPT-4.1 nano received very low source-fidelity scores and produced invented quotations in the judged articles. He reported that gpt-oss-120b produced zero valid articles out of four, including output-format problems and invented named quotations. Both were removed from his route on October 7.

Do the results show which model is best for every newsroom?

No. The results come from one publisher’s prompt, sources and pipeline, with a small initial source set and AI judges. They do not establish how the models would perform for other newsrooms or tasks.

Source: ThorstenMeyerAI.com