LLM-Measure: Generating Valid, Consistent, and Reproducible Text-Based Measures for Business Research 2026-11-08
Title: LLM-Measure: Generating Valid, Consistent, and Reproducible Text-Based Measures for Business Research
Speaker: Hanyu Duan, Ph.D. Candidate, HKUST
Host: Xuewen Han, Assistant Professor, Antai College of Economics and Management, Shanghai Jiao Tong University
Time: 10:00–11:30, Tuesday, December 8, 2026
Venue: Room A303, Pao Siu-Long Library, Antai College of Economics and Management, Shanghai Jiao Tong University
Brief introduction of the content:
Text is data. In business research, text data are increasingly used to construct measures that are included as variables in econometric models. Recent advances in large language models (LLMs) have made this practice easier, enabling researchers to create text based measures simply by prompting an LLM. Yet this convenience comes with an important but often-overlooked issue: the resulting measures can be highly sensitive to prompt wording. Even seemingly superficial changes to a prompt can yield materially different measures, raising concerns about measurement reliability and reproducibility and, more critically, about the credibility of subsequent research findings. In this work, we call attention to this issue and develop LLM-Measure, an approach that uses an LLM’s latent representations to construct text-based measures with reduced prompt sensitivity. Across three measurement tasks, the measures produced by LLM-Measure align more closely with human judgments and remain more consistent across prompt rephrasings than those produced through conventional LLM prompting. To demonstrate LLM Measure’s practical value in supporting empirical research, we use it to construct measures at the macro, firm, and individual levels and incorporate them into relevant econometric models, successfully replicating findings documented in prior research. Overall, this work addresses a pressing methodological issue in LLM-assisted research and offers a broadly applicable method for text-based measurement within and beyond information systems.
Speaker's profile:
Hanyu Duan is a PhD candidate in the Department of Information Systems, Business Statistics and Operations Management at Hong Kong University of Science and Technology (HKUST). He received his MSc in data-driven modeling from HKUST in 2020 and his BSc in applied physics from Jimei University in 2019. His research focuses on developing natural language processing-based computational methods to study problems in business and the social sciences.
Welcome all!
