EmoUS is a data-driven user simulator with transformers, generating user emotions, semantic actions, and natural language responses based on the user goal, the dialogue history, and the user persona.
Introduction
We propose a generative transform-based user simulator (EmoUS) in this work. EmoUS consists of an encoder-decoder structure, which can generate user emotions, semantic actions, and natural language responses based on the user goal, the dialogue history, and the user persona. By analysing what kind of system behaviour elicits what kind of user emotions, we show that EmoUS can be used as a probe to evaluate a variety of dialogue systems and in particular their effect on the user's emotional state. Developing such methods is important in the age of large language model chat-bots and rising ethical concerns.
This repo is a snapshot of current convlab-3 and you can find the code of EmoUS is in convlab/policy/emoUS.
If you want to check the latest version you can find in the convlab3 official repo
Usage
Train EmoUS
You need to generate the input files by build_data.py, then train the model by train_model.py.
python3 convlab/policy/emoUS/unify/build_data.py --add-persona --dataset $dataset --add-history --dial-ids-order $dial_ids_order --split2ratio $split2ratio
python3 convlab/policy/genTUS/train_model.py --data-name $dataset --dial-ids-order $dial_ids_order --split2ratio $split2ratio --batch-size 8dataset can be emowoz+dialmage, emowoz, or dialmage. emowoz+dialmage is the whole dataset pf EmoWOZ, which is composed with human-human (emowoz) and human-machine (dialmage) task-oriented dialogues
dial_ids_order can be 0, 1 or 2
split2ratio can be 0.01, 0.1 or 1
The build_data.py will generate three files, train.json, validation.json, and test.json, under the folder convlab/policy/emoUS/unify/data/EmoUS_${dataset}_${dial_ids_order}_${split2ration}.
We trained EmoUS on A100 or RTX6000.
Evaluate EmoUS
python3 convlab/policy/emoUS/evaluate.py --model-checkpoint $model_checkpoint --input-file $in_file The in_file is the file generated by build_data.py.
The results include three parts:
- Natural language generation: Sacre-BLEU score and slot error rate (SER)
- Semantic action prediction: the precision, recall, F1-score, and turn accuracy for full action prediction and intent-domain prediction.
- Emotion prediction: macro F1-score form emotion and sentiment.
Citing
Hsien-Chin Lin, Shutong Feng, Christian Geishauser, Nurul Lubis, Carel van Niekerk, Michael Heck, Benjamin Matthias Ruppik, Renato Vukovic, and Milica Gašić. 2023. EmoUS: Simulating User Emotions in Task-Oriented Dialogues. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23), July 23–27, 2023, Taipei, Taiwan. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3539618.3592092
License
Apache License 2.0