PFI-TT: Cooperative Listening with Networked Audio Devices

Award ID: 1919257

The broader impact/commercial potential of this Partnerships for Innovation - Technology Translation (PFI-TT) project is to enable multiple listening devices to work together and improve their individual performance through collaborative signal processing. For both humans and machines, speech comprehension can be difficult in noisy, complex environments with several competing sound sources. Many of these environments may have multiple devices that in turn each contain multiple microphones, such as smart speakers, smart-home appliances, mobile devices, and wearables. Unfortunately, these devices do not currently cooperate to perform spatial sound capture. If they could be connected together to perform large-scale spatial signal processing, such a distributed array could dramatically improve the performance of machine listening, human-computer interaction, and human sensory augmentation tasks. The ability to precisely localize, separate, and enhance sound sources in complex environments would enable new applications that are impossible with current technology. These technologies could be applied to many already-deployed acoustic systems with minimal additional bandwidth and computation requirements. This research has the potential to dramatically impact these and other application areas while training graduate and undergraduate students and post-doctoral researchers from a range of underrepresented groups in lean-startup approaches to technology commercialization. The proposed project will develop technologies to aggregate data from multiple devices containing acoustic arrays, such as smart-home devices and wearables, to improve the spatial sound processing performance of each individual device. The proposed technology uses a hierarchical, distributed processing approach to efficiently aggregate information across networked devices without transmitting full synchronous audio data. The resulting system can leverage the spatial diversity of the distributed array to achieve better performance than a single device in listening tasks, especially in adverse environments with strong noise and interference where current technology often fails. The research objectives of this project are to explore the design trade-offs and scaling behavior of systems at large-scale; determine how best to aggregate distributed array data in the presence of non-idealities such as sample clock mismatch, network latency, and bandwidth constraints; to characterize the performance scaling and design tradeoffs of the system in real-world environments and under various acoustic and network conditions; and to implement and demonstrate the new algorithms on embedded hardware. Source separation and speech recognition experiments will be conducted in both controlled laboratory conditions and real-world environments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

NSF Program Director: Samir Iqbal