Ab 01.06.2025 wird in der Informatik 2FA/MFA erzwungen. Die Einrichtung wird vorher empfohlen, optimalerweise direkt mit Fallback (Hardwaretoken UND TOTP-/Authenticator App).
# GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers
**ConvLab-2** is an open-source toolkit that enables researchers to build task-oriented dialogue systems with state-of-the-art models, perform an end-to-end evaluation, and diagnose the weakness of systems. As the successor of [ConvLab](https://github.com/ConvLab/ConvLab), ConvLab-2 inherits ConvLab's framework but integrates more powerful dialogue models and supports more datasets. Besides, we have developed an analysis tool and an interactive tool to assist researchers in diagnosing dialogue systems. [[paper]](https://arxiv.org/abs/2002.04793)
**GenTUS** is a user simulator for task-oriented dialogues, which consists of an encoder-decoder structure and is able to optimise both the user policy and natural language generation jointly. GenTUS generates both semantic actions and natural language utterances, preserving interpretability and enhancing language variation. In addition, by representing the inputs and outputs as word sequences and by using a large pre-trained language model we can achieve generalisability in feature representation.
-[Installation](#installation)
-[Tutorials](#tutorials)
-[Documents](#documents)
-[Models](#models)
-[Supported Datasets](#Supported-Datasets)
-[End-to-end Performance on MultiWOZ](#End-to-end-Performance-on-MultiWOZ)
-[Module Performance on MultiWOZ](#Module-Performance-on-MultiWOZ)
-[Issues](#issues)
-[Contributions](#contributions)
-[Citing](#citing)
-[License](#license)
## Updates
2021.9.13:
- Add [MultiWOZ 2.3](https://github.com/lexmen318/MultiWOZ-coref) dataset in `data` dir. The dataset adds co-reference annotations in addition to corrections of dialogue acts and dialogue states. [[paper]](https://arxiv.org/abs/2010.05594)
2021.6.18:
- Add [LAUG](https://github.com/thu-coai/LAUG), an open-source toolkit for Language understanding AUGmentation. It is an automatic method to approximate the natural perturbations to existing data. Augmented data could be used to conduct black-box robustness testing or enhancing training. [[paper]](https://arxiv.org/abs/2012.15262)
- Add [SC-GPT](https://github.com/pengbaolin/SC-GPT) for NLG. [[paper]](https://arxiv.org/abs/2002.12328)
-[Getting Started](https://github.com/thu-coai/ConvLab-2/blob/master/tutorials/Getting_Started.ipynb)(Have a try on [Colab](https://colab.research.google.com/github/thu-coai/ConvLab-2/blob/master/tutorials/Getting_Started.ipynb)!)
-[Add New Model](https://github.com/thu-coai/ConvLab-2/blob/master/tutorials/Add_New_Model.md)
For more details about these models, You can refer to `README.md` under `convlab2/$module/$model/$dataset` dir such as `convlab2/nlu/jointBERT/multiwoz/README.md`.
- We add user dialogue act (*inform*, *request*, *bye*, *greet*, *thank*), remove 5 sessions that have incomplete dialogue act annotation and place it under `data/multiwoz` dir.
- Train/val/test size: 8434/999/1000. Split as original data.
- We offers a rule-based user simulator and a complete set of models for building a pipeline system on the CrossWOZ dataset. We correct few state annotation and place it under `data/crosswoz` dir.
- Train/val/test size: 5012/500/500. Split as original data.
*Notice*: The results are for commits before [`bdc9dba`](https://github.com/thu-coai/ConvLab-2/commit/bdc9dba72c957d97788e533f9458ed03a4b0137b)(inclusive). We will update the results after improving user policy.
We perform end-to-end evaluation (1000 dialogues) on MultiWOZ using the user simulator below (a full example on `tests/test_end2end.py`) :
*Notice*: The results are for commits before [`bdc9dba`](https://github.com/thu-coai/ConvLab-2/commit/bdc9dba72c957d97788e533f9458ed03a4b0137b)(inclusive). We will update the results after improving user policy.
By running `convlab2/policy/evalutate.py --model_name $model`
| | Task Success Rate |
| --------- | ----------------- |
| MLE | 0.56 |
| PG | 0.54 |
| PPO | 0.89 |
| GDPL | 0.58 |
### NLG
By running `convlab2/nlg/evaluate.py MultiWOZ $model sys`
| | corpus BLEU-4 |
| -------- | ------------- |
| Template | 0.3309 |
| SCLSTM | 0.4884 |
## Translation-train SUMBT for cross-lingual DST
### Train
With Convlab-2, you can train SUMBT on a machine-translated dataset like this:
evaluation of our pre-trained models are: (joint acc.)
| type | CrossWOZ-en | MultiWOZ-zh |
| ----- | ----------- | ----------- |
| val | 12.4% | 48.5% |
| test | 12.4% | 46.0% |
| human_val | 10.6% | 47.4% |
`human_val` option will make the model evaluate on the validation set translated by human.
Note: You may want to download pre-traiend BERT models and translation-train SUMBT models provided by us.
Without modifying any code, you could:
- download pre-trained BERT models from:
-[bert-base-uncased](https://huggingface.co/bert-base-uncased) for CrossWOZ-en
-[chinese-bert-wwm-ext](https://huggingface.co/hfl/chinese-bert-wwm-ext) for MultiWOZ-zh
extract it to `./pre-trained-models`.
- for translation-train SUMBT model:
-[trained on CrossWOZ-en](https://convlab.blob.core.windows.net/convlab-2/crosswoz_en-pytorch_model.bin.zip)
-[trained on MultiWOZ-zh](https://convlab.blob.core.windows.net/convlab-2/multiwoz_zh-pytorch_model.bin.zip)
- Say the data set is CrossWOZ (English), (after extraction) just save the pre-trained model under `./convlab2/dst/sumbt/crosswoz_en/pre-trained` and name it with `pytorch_model.bin`.
## Issues
You are welcome to create an issue if you want to request a feature, report a bug or ask a general question.
## Contributions
We welcome contributions from community.
- If you want to make a big change, we recommend first creating an issue with your design.
- Small contributions can be directly made by a pull request.
- If you like make contributions to our library, see issues to find what we need.
## Team
**ConvLab-3** is maintained and developed by Tsinghua University Conversational AI group (THU-coai), the [Dialogue Systems and Machine Learning Group](https://www.cs.hhu.de/en/research-groups/dialog-systems-and-machine-learning.html) at Heinrich Heine University, Düsseldorf, Germany and Microsoft Research (MSR).
abstract = "User simulators (USs) are commonly used to train task-oriented dialogue systems via reinforcement learning. The interactions often take place on semantic level for efficiency, but there is still a gap from semantic actions to natural language, which causes a mismatch between training and deployment environment. Incorporating a natural language generation (NLG) module with USs during training can partly deal with this problem. However, since the policy and NLG of USs are optimised separately, these simulated user utterances may not be natural enough in a given context. In this work, we propose a generative transformer-based user simulator (GenTUS). GenTUS consists of an encoder-decoder structure, which means it can optimise both the user policy and natural language generation jointly. GenTUS generates both semantic actions and natural language utterances, preserving interpretability and enhancing language variation. In addition, by representing the inputs and outputs as word sequences and by using a large pre-trained language model we can achieve generalisability in feature representation. We evaluate GenTUS with automatic metrics and human evaluation. Our results show that GenTUS generates more natural language and is able to transfer to an unseen ontology in a zero-shot fashion. In addition, its behaviour can be further shaped with reinforcement learning opening the door to training specialised user simulators.",
}
@inproceedings{zhu2020convlab2,
title={ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems},
author={Qi Zhu and Zheng Zhang and Yan Fang and Xiang Li and Ryuichi Takanobu and Jinchao Li and Baolin Peng and Jianfeng Gao and Xiaoyan Zhu and Minlie Huang},
