POSE: Phase II: LMGame: An Integrated Open-Source Ecosystem for Benchmarking and Training Generative AI using Digital Games

Award ID: 2550249

This Pathways to Enable Open-Source Ecosystems (POSE) project establishes LMGame, an open-source ecosystem that repurposes computer games as rigorous instruments for artificial intelligence (AI) evaluation. As AI grows more capable and more widely deployed, society needs ways to measure what AI can and cannot do. Games combine clear rules, measurable outcomes, and real-time challenges. Games can thus be used as instruments for revealing whether AI can reason, plan, perceive, and make decisions. However, existing game-based AI evaluations are ad-hoc and fragmented, with benchmarks scattered across repositories and testbeds that rely on short-term volunteer maintenance. By unifying diverse game environments under a shared open-source ecosystem with standardized metrics, transparent reporting, and reproducible results, the LMGame project lowers the barrier for researchers, students, educators, and practitioners to probe AI capabilities. The project advances the national interest by strengthening open infrastructure for trustworthy AI evaluation, training the next generation of AI researchers, and advancing U.S. leadership in open-source AI. This POSE Phase II project transitions LMGame into a coherent open-source ecosystem through four coordinated aims. First, the project consolidates evaluation and training into a single Large Language Model (LLM)-first platform where all components run on a unified set of game environments and a shared application programming interface (API). Second, the project formalizes governance through a technical core committee that oversees signed versioned releases and responsible-disclosure processes. Third, the project implements continuous development and deployment with reproducible test harnesses for each game and metric. Fourth, the project grows the ecosystem via partner pathways for academia, industry, and non-profits; and through education and outreach activities supported by reference integrations, course modules, and showcases. The vision is for the LMGame project to grow into a healthy ecosystem that includes standardized interfaces and harnesses; auditable, contamination-aware metrics; reproducible pipelines that connect evaluation to post-training; and a managing organization that can curate, refresh, and secure the stack as models and games evolve. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

NSF Program Director: Marlon Pierce