Data & AI Methods for Modeling Facial Expressions in Language with Applications to Privacy for the Deaf, ASL Education & Linguistic Research

Award ID: 2040638

The NSF Convergence Accelerator supports use-inspired, team-based, multidisciplinary efforts that address challenges of national importance and will produce deliverables of value to society in the near future. American Sign Language (ASL) is the third most studies foreign language in the United States. This project is building 4-dimensional face-tracking algorithms that could be used to separate facial geometry from facial movement and expression. The work supports an application for teaching American Sign Language (ASL) to ASL-learners, an application for anonymizing the signer when privacy is a concern, and research into the role of facial expressions in both sign and spoken language. The privacy preserving application being developed by this project will enable ASL speakers to have private conversations about sensitive topics they would otherwise. This team of linguists, computer scientists, deaf and hearing experts on ASL, and industry partners will address research and societal challenges through three types of deliverables targeted to diverse user and research communities: 1) Modifications and extension of AI methods and publicly shared ASL data and tools to encompass spoken language. Although facial expressions and head gestures, essential to the grammar of signed languages, also play an important role in speech, this is not well understood because resources of the kind developed by this project have not been available. New data and analyses will open the door to comparative study of the role of facial expressions across modalities, and the role of facial expressions in signed language vs. spoken language. Shared raw data, analyses, and visualizations will open up new avenues for linguistic and computer science research into the role of spatiotemporal synchronization of nonmanual expressions in conjunction with speech and signing. 2) An application to help ASL learners produce facial expressions and head gestures to convey grammatical information in signed languages; and 3) Development of a tool for real-time anonymization of ASL videos to preserve grammatical information expressed non-manually, while de-identifying the signer. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

NSF Program Director: Mike Pozmantier