Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance

For the code used in our paper navigate to convlab/dst/setsumbt. See below info regarding the use and installation of the convlab environment.

ConvLab-2

ConvLab-2 is an open-source toolkit that enables researchers to build task-oriented dialogue systems with state-of-the-art models, perform an end-to-end evaluation, and diagnose the weakness of systems. As the successor of ConvLab, ConvLab-2 inherits ConvLab's framework but integrates more powerful dialogue models and supports more datasets. Besides, we have developed an analysis tool and an interactive tool to assist researchers in diagnosing dialogue systems. paper(https://arxiv.org/abs/2002.04793)

• Installation
• Tutorials
• Documents
• Models
• Supported Datasets
• End-to-end Performance on MultiWOZ
• Module Performance on MultiWOZ
• Issues
• Contributions
• Citing
• License

Installation

Require python 3.6.

Clone this repository:

git clone https://gitlab.cs.uni-duesseldorf.de/general/dsml/setsumbt-public.git

Install ConvLab-2 via pip:

cd ConvLab-2
pip install -e .

Tutorials

• Getting Started (Have a try on Colab!)
• Add New Model
• Train RL Policies
• Interactive Tool demo video(https://youtu.be/00VWzbcx26E)

Documents

Our documents are on https://thu-coai.github.io/ConvLab-2_docs/convlab2.html.

Models

We provide following models:

• NLU: SVMNLU, MILU, BERTNLU
• DST: rule, TRADE, SUMBT, SetSUMBT
• Policy: rule, Imitation, REINFORCE, PPO, GDPL, MDRG, HDSA, LaRL
• Simulator policy: Agenda, VHUS
• NLG: Template, SCLSTM
• End2End: Sequicity, DAMD, RNN_rollout

For more details about these models, You can refer to README.md under convlab2/$module/$model/$dataset dir such as convlab2/nlu/jointBERT/multiwoz/README.md.

Supported Datasets

• Multiwoz 2.1
  • We add user dialogue act (inform, request, bye, greet, thank), remove 5 sessions that have incomplete dialogue act annotation and place it under data/multiwoz dir.
  • Train/val/test size: 8434/999/1000. Split as original data.
  • LICENSE: Attribution 4.0 International, url: http://creativecommons.org/licenses/by/4.0/
• CrossWOZ
  • We offers a rule-based user simulator and a complete set of models for building a pipeline system on the CrossWOZ dataset. We correct few state annotation and place it under data/crosswoz dir.
  • Train/val/test size: 5012/500/500. Split as original data.
  • LICENSE: Attribution 4.0 International, url: http://creativecommons.org/licenses/by/4.0/
• Camrest
  • We add system dialogue act (inform, request, nooffer) and place it under data/camrest dir.
  • Train/val/test size: 406/135/135. Split as original data.
  • LICENSE: Attribution 4.0 International, url: http://creativecommons.org/licenses/by/4.0/
• Dealornot
  • Placed under data/dealornot dir.
  • Train/val/test size: 5048/234/526. Split as original data.
  • LICENSE: Attribution-NonCommercial 4.0 International, url: https://creativecommons.org/licenses/by-nc/4.0/

Team

ConvLab-2 is maintained and developed by Tsinghua University Conversational AI group (THU-coai) and Microsoft Research (MSR).

We would like to thank:

Yan Fang, Zhuoer Feng, Jianfeng Gao, Qihan Guo, Kaili Huang, Minlie Huang, Sungjin Lee, Bing Li, Jinchao Li, Xiang Li, Xiujun Li, Lingxiao Luo, Wenchang Ma, Mehrad Moradshahi, Baolin Peng, Runze Liang, Ryuichi Takanobu, Hongru Wang, Jiaxin Wen, Yaoqin Zhang, Zheng Zhang, Qi Zhu, Xiaoyan Zhu.

The additional code in ConvLab-2 required for this work is developed and maintained by the Dialogue Systems and Machine Learning (DSML) group at the Heinrich Heine Universität Düsseldorf.

We would like to thank:

Carel van Niekerk, Andrey Malinin, Christian Geishauser, Michael Heck, Hsien-chin Lin, Nurul Lubis, Shutong Feng and Milica Gašić

Citing

If you use ConvLab-2 in your research, please cite:

@inproceedings{zhu2020convlab2,
    title={ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems},
    author={Qi Zhu and Zheng Zhang and Yan Fang and Xiang Li and Ryuichi Takanobu and Jinchao Li and Baolin Peng and Jianfeng Gao and Xiaoyan Zhu and Minlie Huang},
    year={2020},
    booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
}

@inproceedings{vanniekerk2021-uncertainty,
  title         = "Uncertainty measures in neural belief tracking and the
                   effects on dialogue policy performance",
  author        = "van Niekerk, Carel and Malinin, Andrey and Geishauser,
                   Christian and Heck, Michael and Lin, Hsien-Chin and Lubis,
                   Nurul and Feng, Shutong and Ga{\v s}i{\'c}, Milica",
  year          =  2021,
  copyright     = "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
  archivePrefix = "arXiv",
  primaryClass  = "cs.CL",
  eprint        = "2109.04349"
}

License

Apache License 2.0