Merge branch 'US_readme' into 'github_master'

Us readme

See merge request dsml/convlab/ConvLab3!44

convlab/policy/genTUS/README.md

+**GenTUS** is a data-driven user simulator with transformers, which can generate semantic actions and utterence. It is able to trasfer to a new ontology in a zero-shot fashion.
+
+## Introduction
+We propose a generative transform-based user simulator (GenTUS) in this work. GenTUS consists of an encoder-decoder structure, which can optimise both the user policy and natural language generation jointly. GenTUS generates semantic actions and natural language utterances, preserving interpretability and enhancing language variation. 
+
+The code of TUS is in `convlab/policy/genTUS`.
+
+## Usage
+### Train GenTUS from scratch
+You need to generate the input files by `build_data.py`, then train the model by `train_model.py`.
+```
+python3 convlab/policy/genTUS/unify/build_data.py --dataset $dataset --add-history --dial-ids-order $dial_ids_order --split2ratio $split2ratio
+python3 convlab/policy/genTUS/train_model.py --data-name $dataset --dial-ids-order $dial_ids_order --split2ratio $split2ratio --batch-size 8
+```
+
+`dataset` can be `multiwoz21`, `sgd`, `tm`, `sgd+tm`, or `all`.
+`dial_ids_order` can be 0, 1 or 2
+`split2ratio` can be 0.01, 0.1 or 1
+
+The `build_data.py` will generate three files, `train.json`, `validation.json`, and `test.json`, under the folder `convlab/policy/genTUS/unify/data/${dataset}_${dial_ids_order}_${split2ration}`. 
+We trained GenTUS on A100 or RTX6000.
+
+### Evaluate TUS
+```
+python3 convlab/policy/genTUS/evaluate.py --model-checkpoint $model_checkpoint --input-file $in_file --dataset $dataset --do-nlg
+```
+The `in_file` is the file generated by `build_data.py`.
+
+### Train a dialogue policy with GenTUS
+You can use it as a normal user simulator by `PipelineAgent`. For example,
+```python
+from convlab.dialog_agent import PipelineAgent
+from convlab.util.custom_util import set_seed
+
+model_checkpoint = 'convlab/policy/genTUS/unify/experiments/multiwoz21-exp'
+usr_policy = UserPolicy(model_checkpoint, mode="semantic")
+simulator = PipelineAgent(None, None, usr_policy, None, 'user')
+```
+then you can train your system with this simulator.
+
+You can also change the `mode` to `"language"`, then GenTUS will response in natural language instead of semantic actions.
+
+
+<!---citation--->
+## Citing
+
+```
+@inproceedings{lin-etal-2022-gentus,
+    title = "{G}en{TUS}: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers",
+    author = "Lin, Hsien-chin  and
+      Geishauser, Christian  and
+      Feng, Shutong  and
+      Lubis, Nurul  and
+      van Niekerk, Carel  and
+      Heck, Michael  and
+      Gasic, Milica",
+    booktitle = "Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue",
+    month = sep,
+    year = "2022",
+    address = "Edinburgh, UK",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2022.sigdial-1.28",
+    pages = "270--282",
+    abstract = "User simulators (USs) are commonly used to train task-oriented dialogue systems via reinforcement learning. The interactions often take place on semantic level for efficiency, but there is still a gap from semantic actions to natural language, which causes a mismatch between training and deployment environment. Incorporating a natural language generation (NLG) module with USs during training can partly deal with this problem. However, since the policy and NLG of USs are optimised separately, these simulated user utterances may not be natural enough in a given context. In this work, we propose a generative transformer-based user simulator (GenTUS). GenTUS consists of an encoder-decoder structure, which means it can optimise both the user policy and natural language generation jointly. GenTUS generates both semantic actions and natural language utterances, preserving interpretability and enhancing language variation. In addition, by representing the inputs and outputs as word sequences and by using a large pre-trained language model we can achieve generalisability in feature representation. We evaluate GenTUS with automatic metrics and human evaluation. Our results show that GenTUS generates more natural language and is able to transfer to an unseen ontology in a zero-shot fashion. In addition, its behaviour can be further shaped with reinforcement learning opening the door to training specialised user simulators.",
+}
+
+
+```
+
+## License
+
+Apache License 2.0

convlab/policy/genTUS/stepGenTUS.py

@@ -11,6 +11,8 @@ from convlab.policy.genTUS.unify.Goal import Goal
 from convlab.policy.genTUS.unify.knowledge_graph import KnowledgeGraph
 from convlab.policy.policy import Policy
 from convlab.task.multiwoz.goal_generator import GoalGenerator
+from convlab.util.custom_util import model_downloader
+
 
 DEBUG = False
 
@@ -589,10 +591,10 @@ class UserPolicy(Policy):
                  action_penalty=False,
                  **kwargs):
         # self.config = config
-        # if not os.path.exists(self.config["model_dir"]):
-        #     os.mkdir(self.config["model_dir"])
-        #     model_downloader(self.config["model_dir"],
-        #                      "https://zenodo.org/record/5779832/files/default.zip")
+        if not os.path.exists(os.path.dirname(model_checkpoint)):
+            os.mkdir(os.path.dirname(model_checkpoint))
+            model_downloader(os.path.dirname(model_checkpoint),
+                             "https://zenodo.org/record/7372442/files/multiwoz21-exp.zip")
 
         self.policy = UserActionPolicy(
             model_checkpoint,
@@ -636,11 +638,11 @@ if __name__ == "__main__":
 
     set_seed(20220220)
     # Test semantic level behaviour
-    model_checkpoint = 'convlab/policy/genTUS/unify/experiments/multiwoz21_0_1.0'
+    model_checkpoint = 'convlab/policy/genTUS/unify/experiments/multiwoz21-exp'
     usr_policy = UserPolicy(
         model_checkpoint,
         mode="semantic")
-    usr_policy.policy.load(os.path.join(model_checkpoint, "pytorch_model.bin"))
+    # usr_policy.policy.load(os.path.join(model_checkpoint, "pytorch_model.bin"))
     usr_nlu = None  # BERTNLU()
     usr = PipelineAgent(usr_nlu, None, usr_policy, None, name='user')
     print(usr.policy.get_goal())

convlab/policy/genTUS/unify/Goal.py

@@ -54,8 +54,8 @@ class Goal:
             self.raw_goal = goal.domain_goals
             goal = old_goal2list(goal.domain_goals)
 
-        else:
-            print("unknow goal")
+        # else:
+        #     print("unknow goal")
 
         # be careful of this order
         for domain, intent, slot, value in goal:

convlab/policy/tus/README.md

-**TUS** is a domain-independent user simulator with transformers for task-oriented dialogue systems. It is based on the [ConvLab-2](https://github.com/thu-coai/ConvLab-2) framework. Therefore, you should follow their instruction to install the package.
+**TUS** is a domain-independent user simulator with transformers for task-oriented dialogue systems.
 
 ## Introduction
-Our model is a domain-independent user simulator, which means it is not based on any domain-dependent freatures and the output representation is also domain-independent. Therefore, it can easily adapt to a new domain, without additional feature engineering and model retraining.
+Our model is a domain-independent user simulator, which means its input and output representations are domain agnostic. Therefore, it can easily adapt to a new domain, without additional feature engineering and model retraining.
 
-The code of TUS is in `convlab/policy/tus` and a rule-based DST of user is also created in `convlab/dst/rule/multiwoz/dst.py` based on the rule-based DST in `convlab/dst/rule/multiwoz/dst.py`.
+The code of TUS is in `convlab/policy/tus`.
 
-## How to run the model
-### Train the user simulator
-`python3 convlab/policy/tus/multiwoz/train.py --user_config convlab/policy/tus/multiwoz/exp/default.json`
+## Usage
+### Train TUS from scratch
 
-One default configuration is placed in `convlab/policy/tus/multiwoz/exp/default.json`. They can be modified based on your requirements. For example, the output directory can be specified in the configuration (`model_dir`).
+```
+python3 convlab/policy/tus/unify/train.py --dataset $dataset --dial-ids-order $dial_ids_order --split2ratio $split2ratio --user-config $config
+```
+
+`dataset` can be `multiwoz21`, `sgd`, `tm`, `sgd+tm`, or `all`.
+`dial_ids_order` can be 0, 1 or 2
+`split2ratio` can be 0.01, 0.1 or 1
+Default configurations are placed in `convlab/policy/tus/unify/exp`. They can be modified based on your requirements. 
+
+For example, you can train TUS for multiwoz21 by 
+`python3 convlab/policy/tus/unify/train.py --dataset multiwoz21 --dial-ids-order 0 --split2ratio 1 --user-config "convlab/policy/tus/unify/exp/multiwoz.json"`
+
+### Evaluate TUS
 
 ### Train a dialogue policy with TUS
 You can use it as a normal user simulator by `PipelineAgent`. For example,
 ```python
 import json
 from convlab.dialog_agent.agent import PipelineAgent
-from convlab.dst.rule.multiwoz.usr_dst import UserRuleDST
-from convlab.policy.tus.multiwoz.TUS import UserPolicy
+from convlab.policy.tus.unify.TUS import UserPolicy
 
-user_config_file = "convlab/policy/tus/multiwoz/exp/default.json"
-dst_usr = UserRuleDST()
+user_config_file = "convlab/policy/tus/unify/exp/multiwoz.json"
 user_config = json.load(open(user_config_file))
 policy_usr = UserPolicy(user_config)
-simulator = PipelineAgent(None, dst_usr, policy_usr, None, 'user')
+simulator = PipelineAgent(None, None, policy_usr, None, 'user')
+```
+then you can train your system with this simulator.
+
+There is an example config, which trains a PPO policy with TUS in semantic level, in `convlab/policy/ppo/tus_semantic_level_config.json`.
+You can train a PPO policy as following, 
+```
+config="convlab/policy/ppo/tus_semantic_level_config.json"
+python3 convlab/policy/ppo/train.py --path $config
 ```
+notice: You should name your pretrained policy as `convlab/policy/ppo/pretrained_models/mle` or modify the `load_path` of `model` in the config `convlab/policy/ppo/tus_semantic_level_config.json`.
 
 
 <!---citation--->

convlab/policy/tus/unify/TUS.py

@@ -132,6 +132,8 @@ class UserActionPolicy(Policy):
             self.raw_goal = goal.domain_goals
             goal_list = old_goal2list(goal.domain_goals)
             goal = Goal(goal_list)
+        elif type(goal) == Goal:
+            self.raw_goal = goal.domain_goals
         else:
             goal = ABUS_Goal(self.goal_gen)
             self.raw_goal = goal.domain_goals
@@ -416,7 +418,7 @@ class UserPolicy(Policy):
         if not os.path.exists(self.config["model_dir"]):
             # os.mkdir(self.config["model_dir"])
             model_downloader(os.path.dirname(self.config["model_dir"]),
-                             "https://zenodo.org/record/5779832/files/default.zip")
+                             "https://zenodo.org/record/7369429/files/multiwoz_0.zip")
         self.slot2dbattr = {
             'open hours': 'openhours',
             'price range': 'pricerange',
@@ -451,30 +453,4 @@ class UserPolicy(Policy):
     def get_goal(self):
         if hasattr(self.policy, 'get_goal'):
             return self.policy.get_goal()
-            # workaround: convert goal to old format
-            multiwoz_goal = {}
-            goal = self.policy.get_goal()
-            for domain in goal:
-                multiwoz_goal[domain] = {}
-                for slot_type in ["info", "reqt"]:
-                    if slot_type not in goal[domain]:
-                        continue
-                    if slot_type not in multiwoz_goal[domain]:
-                        multiwoz_goal[domain][slot_type] = {}
-                    for slot in goal[domain][slot_type]:
-                        value = goal[domain][slot_type][slot].lower()
-                        if "book" in slot:
-                            if "book" not in multiwoz_goal[domain]:
-                                multiwoz_goal[domain]["book"] = {}
-                            norm_slot = slot.split(' ')[-1]
-                            multiwoz_goal[domain]["book"][norm_slot] = value
-                        elif slot in self.slot2dbattr:
-                            norm_slot = self.slot2dbattr[slot]
-                            multiwoz_goal[domain][slot_type][norm_slot] = value
-                        else:
-                            multiwoz_goal[domain][slot_type][slot] = value
-            for domain in multiwoz_goal:
-                if "book" in multiwoz_goal[domain]:
-                    multiwoz_goal[domain]["booked"] = '?'
-            return multiwoz_goal
         return None