POSE: Phase II: SceneryStack: An Open-Source Ecosystem for Scaling High-Quality, Multi-Modal Interactives for STEM Education and Beyond

Award ID: 2550231

The Web is rapidly evolving into a medium that is multimodal, data-rich, and increasingly shaped by advances in artificial intelligence (AI). Education and research communities are demanding tools that support interactive simulations, immersive learning, and multimodal collaboration, yet most frameworks remain limited in their input/output modes, structured data streams, and scalability. Realizing the Web's potential requires new shared software infrastructure for building high-quality interactive experiences that combine visuals, sound, touch, and gesture. The solutions must support many types of users and scenarios and produce well-organized data that AI systems can use. Today, developers piece together custom solutions, which slows innovation and limits what gets built. This project advances SceneryStack, an open-source framework, into a community resource for advanced interactive media. This project will drive innovation and capabilities in interactive Web technologies, expand participation in open-source development, and support progress in education with potential applications in healthcare, workforce training, and entertainment. SceneryStack is a set of open-source TypeScript libraries that serve as the building blocks for Web-based interactive media, supporting dynamic graphics, sound, description, and mathematical models. Distinctive features include: 1) connecting each static, dynamic, or interactive visual element with a structured, readable HyperText Markup Language (HTML) description that AI systems can use for automated narration, multimodal translation, and intelligent tutoring; 2) delivering spoken description directly through the browser; and 3) supporting gesture-based control for flexible interaction. This project phase will grow SceneryStack into a sustained ecosystem by growing the community of users and contributors, advancing the codebase, expanding the documentation, building partnerships with companies and standards bodies, and developing transparent governance with funding. By the end of Phase II, SceneryStack will be a well-established open-source platform for the innovation and creation of advanced interactive media for science, technology, engineering, and mathematics education and beyond. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

NSF Program Director: Florence Rabanal