📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva LLM, built from scratch with extensive Italian data, underperformed on Italian academic benchmarks. This raises questions about the scale of native-language investment needed for effective country-specific models.
Italy’s Minerva-3B, a sovereign language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, demonstrating significant challenges in achieving country-specific language understanding despite large-scale investment.
The Minerva project, led by Sapienza University of Rome and funded through Italy’s national AI strategy, involved training a 7-billion-parameter model from scratch on a dataset of 2.5 trillion tokens, half of which was Italian. The project aimed to create a highly capable Italian language model and was publicly open, with weights, data, and code released from the start.
Despite these extensive efforts, Minerva-3B’s performance on the INVALSI Italian school exams was markedly low, at just 4.9%, which is near chance level. Researchers noted that while dataset composition and scale are important, the overall size of the dataset and the number of parameters are more critical for handling complex language tasks. This empirical result suggests that even large-scale native-language training may be insufficient at current parameter scales to produce models with deep country-specific knowledge.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Fine-Tuning Large Language Models: From Custom Datasets to High-Performance AI Models Using Modern Toolchains
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Kato KAT24-000 Scale Rerailer Unijoiner Tool
EAN 4007246775149
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
OpenAI Evals Cookbook: Designing Benchmarks for Product‑Grade LLM Features
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Mastering Small Language Models: A Practical Guide to Building Lightweight NLP Systems with Python, Transformers, and Quantization Techniques
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-Language Model Strategies
The results from Minerva challenge assumptions that large-scale native-language training alone guarantees high performance on country-specific tasks. They suggest that the European sovereign-LLM movement may need to reconsider the scale of investment and the amount of native-language data required to develop models with meaningful country-level knowledge. The findings highlight a potential scalability gap, even for ambitious projects like Minerva, which could influence future investments and strategic decisions across Europe.
Background on European Sovereign LLM Efforts and Minerva’s Approach
European countries have been pursuing sovereign LLM projects to build language models tailored to national languages and needs. Italy’s Minerva is notable for training from scratch on a massive dataset, contrasting with approaches like Portugal’s AMÁLIA, which layers specialization on multilingual foundations. Minerva’s development involved significant institutional backing, including Italy’s National Research Council, CINECA’s supercomputing resources, and PNRR funding. The project aimed to demonstrate that large native-language datasets and extensive training could produce high-performing models, but the low benchmark score raises questions about this assumption.
“While the dataset size and parameters are critical, our results suggest that even large-scale native-language training at current model sizes may not suffice for complex language understanding.”
— Research team, Minerva project
Unresolved Questions About Model Scaling and Performance
It remains unclear what the optimal scale of native-language data and parameters is to achieve meaningful country-specific language understanding. The performance of Minerva-3B suggests current scales may be insufficient, but further research is needed to determine the thresholds needed for effective models. Additionally, whether alternative architectures or training strategies can improve results is still under investigation.
Next Steps in European Sovereign-Language Model Development
The Minerva team and other European projects are likely to explore larger models, different training techniques, and hybrid approaches combining multilingual and native-language data. Further benchmarking and iterative development will be necessary to establish effective strategies for building models with deep country-specific knowledge. Ongoing research aims to clarify the scale-performance relationship and inform future investments.
Key Questions
Why did Minerva perform poorly on Italian academic tests?
Despite extensive training on a large dataset, the empirical results suggest that current model sizes and native-language data scales may not be sufficient to develop deep country-specific knowledge, especially for complex tasks like academic testing.
Does this mean native-language models are not worth the investment?
Not necessarily. The findings indicate that scale matters, and achieving high performance may require larger models and more native data than currently used. It highlights the need to reassess investment strategies rather than dismissing native-language models altogether.
How does Minerva compare to multilingual models?
Minerva was trained solely on Italian data from scratch, aiming for country-specific knowledge, whereas many multilingual models rely on broader datasets. Despite this, Minerva’s performance on complex tasks was limited, suggesting that scale and data volume are crucial factors.
What are the implications for European AI policy?
The results imply that European sovereign AI initiatives may need to allocate more resources toward larger-scale native-language training to achieve meaningful country-specific capabilities, potentially influencing future funding and strategic priorities.
Source: ThorstenMeyerAI.com
