Lighting Talks
Passed
Disha Rana
Topic: Uncertainty Signals in Generative Chatbots
Hao Yuan
My recent work focuses on exploring computational methods, which include LLM and linguistic features analysis, to find the association between PLWDs' cognitive abilities and dementia stages, AND their writing.
This project idea is developed based on our previous project that investigates PLWDs’ perspectives on their relationship with family caregivers by examining their posts on the dementia subreddit. In addition to expressing needs to maintain emotional connection with their family throughout their trajectory of decline, PLWDs describe how they are experiencing progressive decline in, for example, memory and language. These self-reports suggest that PLWDs’ writing can provide a valuable source of longitudinal data that reflects how their cognitive abilities change over time. This thus motivates us to explore how to make use of computational tools to find the association between PLWDs' cognitive abilities and dementia stages AND their writing.
For the LLM part, we are designing an agentic system that automatically identifies and extracts PLWDs’ self-reported cognitive and functional symptoms of decline and self-expressed emotions in their online forum posts. The extraction process is based on an expert-reviewed codebook that contains definitions of commonly-seen symptoms of decline in PLWDs and their representative examples.
For the linguistic feature analysis part, we extract features that have been used in prior studies for examining associations between PLWDs’ language abilities and dementia diagnosis. We extend these speech data-based studies by exploring these features in the written text of posts made by PLWDs. In particular, we find statistically significant differences in multiple feature scores between PLWDs’ posts and healthy users’ posts. We also found statistically significant differences in linguistic feature scores between PLWDs’ posts written before and after their first self-reported symptoms of decline.
Together, these two approaches demonstrate how PLWDs’ writing can be associated with cognitive abilities and potentially used to infer their stages of decline.
Recap by Zoom AI
This was the first meeting of a reading group event where graduate students presented their research projects. Lu introduced the group and its purpose of providing a regular space for members to practice reading and presentation skills. The first presenter was Disha, who shared her summer research project with Professor Tiffany Li, examining how AI chatbots affect learning when they display uncertainty signals to learners. Disha explained that despite uncertainty signals, novice learners still struggle to catch errors, and her study aims to identify the specific barriers preventing learners from verifying chatbot responses. Hao then presented his computational research on using large language models and linguistic features analysis to find associations between people living with dementia's writing and their cognitive abilities and dementia stages. Hao described his work on developing an agentic system to automatically identify and extract self-reported cognitive symptoms from online forum posts on the dementia subreddit. The meeting included detailed discussions about methodology, data collection challenges, and potential clinical applications of both presentations.