EnDex: Evaluation of Dialogue Engagingness at Scale

Abstract

We propose EnDex, the first human-reaction based model to evaluate dialogue engagingness. EnDex is trained on 80k Reddit-based Engagement Dataset (RED) curated using a novel distant-supervision framework. %for training automatic metric. Engagingness is a key measure that captures high-level quality of AI dialogue systems and closely reflects actual user experience. However, data shortage, plus the abstract and extensive definition of engagingness makes it challenging to develop an automatic metric. Our work departs from mainstream approaches that use synthetic negative examples to train binary classifiers, and instead, proposes a solution using distant-supervision from human-reaction feedback.

Publication
In Findings of Empirical Methods in Natural Language Processing Abu Dhabi: EMNLP-IJCNLP 2022, Abu Dhabi, UAE, 2022.

Related