Skip to content
Snippets Groups Projects
Commit a341b03c authored by zqwerty's avatar zqwerty
Browse files

update github template and tutorial

parent 561b4831
No related branches found
No related tags found
No related merge requests found
Show whitespace changes
Inline Side-by-side
.github/PULL_REQUEST_TEMPLATE/add_datasets.md → .github/ISSUE_TEMPLATE/add_dataset.md
+ 9
− 0
View file @ a341b03c
---
name: Add dataset
about: Add a new dataset in unified format
title: "[Dataset] "
labels: dataset, new
assignees: ''
---
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
......
.github/PULL_REQUEST_TEMPLATE/add_model.md → .github/ISSUE_TEMPLATE/add_model.md
+ 9
− 0
View file @ a341b03c
---
name: Add model
about: Add a new model in unified format
title: "[Model] "
labels: model, new
assignees: ''
---
- **Name:** *name of the model*
- **Description:** *short description of the model (or link to social media or blog post)*
- **Paper:** *link to the model paper if available*
......
.github/PULL_REQUEST_TEMPLATE/custom.md → .github/PULL_REQUEST_TEMPLATE.md 100755 → 100644
+ 0
− 0
View file @ a341b03c
File moved
.github/PULL_REQUEST_TEMPLATE/fix_bug.md deleted 100755 → 0
+ 0
− 23
View file @ 561b4831
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Actual behavior**
A clear and concise description of the actual results.
**Additional context**
Add any other context about the problem here.
**How to fix**
Changes made to fix the bug.
**Reference Issues:** #XX (XX is the issue number you work on)
README.md
+ 1
− 0
View file @ a341b03c
......@@ -65,6 +65,7 @@ docker exec -it CONTAINER_ID bash
## Tutorials
- [Getting Started](https://github.com/ConvLab/ConvLab-3/blob/master/tutorials/Getting_Started.ipynb) (Have a try on [Colab](https://colab.research.google.com/github/ConvLab/ConvLab-3/blob/master/tutorials/Getting_Started.ipynb)!)
- [Introduction to Unified Data Format](https://github.com/ConvLab/ConvLab-3/tree/master/data/unified_datasets)
- [Utility functions for unified datasets](https://github.com/ConvLab/ConvLab-3/blob/master/convlab/util/unified_datasets_util.py)
- [RL Toolkit](https://github.com/ConvLab/ConvLab-3/tree/master/convlab/policy)
......
tutorials/Getting_Started.ipynb
+ 167
− 165
View file @ a341b03c
%% Cell type:markdown id: tags:
# Getting Started
In this tutorial, you will know how to
- use the models in **ConvLab-2** to build a dialog agent.
- use the models in **ConvLab-3** to build a dialog agent.
- build a simulator to chat with the agent and evaluate the performance.
- try different module combinations.
- use analysis tool to diagnose your system.
Let's get started!
%% Cell type:markdown id: tags:
## Environment setup
Run the command below to install ConvLab-2. Then restart the notebook and skip this commend.
Run the command below to install ConvLab-3. Then restart the notebook and skip this commend.
%% Cell type:code id: tags:
``` python
# first install ConvLab-2 and restart the notebook
! git clone https://github.com/thu-coai/ConvLab-2.git && cd ConvLab-2 && pip install -e .
# first install ConvLab-3 and restart the notebook
! git clone --depth 1 https://github.com/ConvLab/ConvLab-3.git && cd ConvLab-3 && pip install -e .
```
%% Cell type:code id: tags:
``` python
# installing en_core_web_sm for spacy to resolve error in BERTNLU
!python -m spacy download en_core_web_sm
```
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
## build an agent
We use the models adapted on [Multiwoz](https://www.aclweb.org/anthology/D18-1547) dataset to build our agent. This pipeline agent consists of NLU, DST, Policy and NLG modules.
First, import some models:
%% Cell type:code id: tags:
``` python
# common import: convlab.$module.$model.$dataset
from convlab.nlu.jointBERT.multiwoz import BERTNLU
from convlab.nlu.milu.multiwoz import MILU
from convlab.dst.rule.multiwoz import RuleDST
from convlab.policy.rule.multiwoz import RulePolicy
from convlab.nlg.template.multiwoz import TemplateNLG
from convlab.dialog_agent import PipelineAgent, BiSession
from convlab.evaluator.multiwoz_eval import MultiWozEvaluator
from pprint import pprint
import random
import numpy as np
import torch
```
%% Cell type:markdown id: tags:
Then, create the models and build an agent:
%% Cell type:code id: tags:
``` python
# go to README.md of each model for more information
# BERT nlu
sys_nlu = BERTNLU()
# simple rule DST
sys_dst = RuleDST()
# rule policy
sys_policy = RulePolicy()
# template NLG
sys_nlg = TemplateNLG(is_user=False)
# assemble
sys_agent = PipelineAgent(sys_nlu, sys_dst, sys_policy, sys_nlg, name='sys')
```
%% Cell type:markdown id: tags:
That's all! Let's chat with the agent using its response function:
%% Cell type:code id: tags:
``` python
sys_agent.response("I want to find a moderate hotel")
```
%% Cell type:code id: tags:
``` python
sys_agent.response("Which type of hotel is it ?")
```
%% Cell type:code id: tags:
``` python
sys_agent.response("OK , where is its address ?")
```
%% Cell type:code id: tags:
``` python
sys_agent.response("Thank you !")
```
%% Cell type:code id: tags:
``` python
sys_agent.response("Try to find me a Chinese restaurant in south area .")
```
%% Cell type:code id: tags:
``` python
sys_agent.response("Which kind of food it provides ?")
```
%% Cell type:code id: tags:
``` python
sys_agent.response("Book a table for 5 , this Sunday .")
```
%% Cell type:markdown id: tags:
## Build a simulator to chat with the agent and evaluate
In many one-to-one task-oriented dialog system, a simulator is essential to train an RL agent. In our framework, we doesn't distinguish user or system. All speakers are **agents**. The simulator is also an agent, with specific policy inside for accomplishing the user goal.
We use `Agenda` policy for the simulator, this policy requires dialog act input, which means we should set DST argument of `PipelineAgent` to None. Then the `PipelineAgent` will pass dialog act to policy directly. Refer to `PipelineAgent` doc for more details.
%% Cell type:code id: tags:
``` python
# MILU
user_nlu = MILU()
# not use dst
user_dst = None
# rule policy
user_policy = RulePolicy(character='usr')
# template NLG
user_nlg = TemplateNLG(is_user=True)
# assemble
user_agent = PipelineAgent(user_nlu, user_dst, user_policy, user_nlg, name='user')
```
%% Cell type:markdown id: tags:
Now we have a simulator and an agent. we will use an existed simple one-to-one conversation controller BiSession, you can also define your own Session class for your special need.
We add `MultiWozEvaluator` to evaluate the performance. It uses the parsed dialog act input and policy output dialog act to calculate **inform f1**, **book rate**, and whether the task is **success**.
%% Cell type:code id: tags:
``` python
evaluator = MultiWozEvaluator()
sess = BiSession(sys_agent=sys_agent, user_agent=user_agent, kb_query=None, evaluator=evaluator)
```
%% Cell type:markdown id: tags:
Let's make this two agents chat! The key is `next_turn` method of `BiSession` class.
%% Cell type:code id: tags:
``` python
def set_seed(r_seed):
random.seed(r_seed)
np.random.seed(r_seed)
torch.manual_seed(r_seed)
set_seed(20200131)
sys_response = ''
sess.init_session()
print('init goal:')
pprint(sess.evaluator.goal)
print('-'*50)
for i in range(20):
sys_response, user_response, session_over, reward = sess.next_turn(sys_response)
print('user:', user_response)
print('sys:', sys_response)
print()
if session_over is True:
break
print('task success:', sess.evaluator.task_success())
print('book rate:', sess.evaluator.book_rate())
print('inform precision/recall/f1:', sess.evaluator.inform_F1())
print('-'*50)
print('final goal:')
pprint(sess.evaluator.goal)
print('='*100)
```
%% Cell type:markdown id: tags:
## Try different module combinations
The combination modes of pipeline agent modules are flexible. We support joint models such as TRADE, SUMBT for word-DST and MDRG, HDSA, LaRL for word-Policy, once the input and output are matched with previous and next module. We also support End2End models such as Sequicity.
Available models:
- NLU: BERTNLU, MILU, SVMNLU
- DST: RuleDST
- Word-DST: SUMBT, TRADE (set `sys_nlu` to `None`)
- Policy: RulePolicy, Imitation, REINFORCE, PPO, GDPL
- Word-Policy: MDRG, HDSA, LaRL (set `sys_nlg` to `None`)
- NLG: Template, SCLSTM
- End2End: Sequicity, DAMD, RNN_rollout (directly used as `sys_agent`)
- Simulator policy: Agenda, VHUS (for `user_policy`)
%% Cell type:code id: tags:
``` python
# available NLU models
from convlab.nlu.svm.multiwoz import SVMNLU
from convlab.nlu.jointBERT.multiwoz import BERTNLU
from convlab.nlu.milu.multiwoz import MILU
# available DST models
from convlab.dst.rule.multiwoz import RuleDST
from convlab.dst.sumbt.multiwoz import SUMBT
from convlab.dst.trade.multiwoz import TRADE
# available Policy models
from convlab.policy.rule.multiwoz import RulePolicy
from convlab.policy.ppo.multiwoz import PPOPolicy
from convlab.policy.pg.multiwoz import PGPolicy
from convlab.policy.mle.multiwoz import MLEPolicy
from convlab.policy.gdpl.multiwoz import GDPLPolicy
from convlab.policy.vhus.multiwoz import UserPolicyVHUS
from convlab.policy.mdrg.multiwoz import MDRGWordPolicy
from convlab.policy.hdsa.multiwoz import HDSA
from convlab.policy.larl.multiwoz import LaRL
# available NLG models
from convlab.nlg.template.multiwoz import TemplateNLG
from convlab.nlg.sclstm.multiwoz import SCLSTM
# available E2E models
from convlab.e2e.sequicity.multiwoz import Sequicity
from convlab.e2e.damd.multiwoz import Damd
```
%% Cell type:markdown id: tags:
NLU+RuleDST or Word-DST:
%% Cell type:code id: tags:
``` python
# NLU+RuleDST:
sys_nlu = BERTNLU()
# sys_nlu = MILU()
# sys_nlu = SVMNLU()
sys_dst = RuleDST()
# or Word-DST:
# sys_nlu = None
# sys_dst = SUMBT()
# sys_dst = TRADE()
```
%% Cell type:markdown id: tags:
Policy+NLG or Word-Policy:
%% Cell type:code id: tags:
``` python
# Policy+NLG:
sys_policy = RulePolicy()
# sys_policy = PPOPolicy()
# sys_policy = PGPolicy()
# sys_policy = MLEPolicy()
# sys_policy = GDPLPolicy()
sys_nlg = TemplateNLG(is_user=False)
# sys_nlg = SCLSTM(is_user=False)
# or Word-Policy:
# sys_policy = LaRL()
# sys_policy = HDSA()
# sys_policy = MDRGWordPolicy()
# sys_nlg = None
```
%% Cell type:markdown id: tags:
Assemble the Pipeline system agent:
%% Cell type:code id: tags:
``` python
sys_agent = PipelineAgent(sys_nlu, sys_dst, sys_policy, sys_nlg, 'sys')
```
%% Cell type:markdown id: tags:
Or Directly use an end-to-end model:
%% Cell type:code id: tags:
``` python
# sys_agent = Sequicity()
# sys_agent = Damd()
```
%% Cell type:markdown id: tags:
Config an user agent similarly:
%% Cell type:code id: tags:
``` python
user_nlu = BERTNLU()
# user_nlu = MILU()
# user_nlu = SVMNLU()
user_dst = None
user_policy = RulePolicy(character='usr')
# user_policy = UserPolicyVHUS(load_from_zip=True)
user_nlg = TemplateNLG(is_user=True)
# user_nlg = SCLSTM(is_user=True)
user_agent = PipelineAgent(user_nlu, user_dst, user_policy, user_nlg, name='user')
```
%% Cell type:markdown id: tags:
## Use analysis tool to diagnose the system
We provide an analysis tool presents rich statistics and summarizes common mistakes from simulated dialogues, which facilitates error analysis and
system improvement. The analyzer will generate an HTML report which contains
rich statistics of simulated dialogues. For more information, please refer to `convlab/util/analysis_tool`.
%% Cell type:code id: tags:
``` python
from convlab.util.analysis_tool.analyzer import Analyzer
# if sys_nlu!=None, set use_nlu=True to collect more information
analyzer = Analyzer(user_agent=user_agent, dataset='multiwoz')
set_seed(20200131)
analyzer.comprehensive_analyze(sys_agent=sys_agent, model_name='sys_agent', total_dialog=100)
```
%% Cell type:markdown id: tags:
To compare several models:
%% Cell type:code id: tags:
``` python
set_seed(20200131)
analyzer.compare_models(agent_list=[sys_agent1, sys_agent2], model_name=['sys_agent1', 'sys_agent2'], total_dialog=100)
```
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment