Test Report
Model Name: SVMNLU-RuleDST-RulePolicy-TemplateNLG
Dataset: multiwoz
Time: 2020-04-24 22:13:40
Overall Results
Success Rate: 70.4 %
(Precision, Recall, F1) : (0.791, 0.888, 0.815)
Average Dialog Turn (Succ): 14.759
Average Dialog Turn (All): 17.736
Metric
| Total Num | Succ Rate | Precision | Recall | F1 | Dialog Loop Failed Rate | Dialog Turn (Succ) | Dialog Turn (All) |
---|
hotel | 425 | 0.605 | 0.539 | 0.647 | 0.563 | 0.122 | 7.230 | 8.551 |
attraction | 333 | 0.931 | 0.891 | 0.940 | 0.907 | 0.015 | 6.361 | 6.541 |
taxi | 167 | 0.844 | 0.886 | 0.865 | 0.872 | 0.156 | 7.362 | 10.060 |
restaurant | 410 | 0.849 | 0.771 | 0.807 | 0.781 | 0.054 | 5.891 | 6.941 |
train | 389 | 0.895 | 0.825 | 0.879 | 0.842 | 0.054 | 11.247 | 12.036 |
police | 23 | 1.000 | 1.000 | 1.000 | 1.000 | 0.000 | 5.652 | 5.652 |
hospital | 30 | 1.000 | 1.000 | 1.000 | 1.000 | 0.000 | 4.067 | 4.067 |
Domain hotel
Domain attraction
Domain taxi
Domain restaurant
Domain train
Domain police
Overall Results
Success Rate: 100.0 %
(Precision, Recall, F1) : (1.000, 1.000, 1.000)
Average Dialog Turn (Succ): 5.652
Average Dialog Turn (All): 5.652
System NLU Failed Dialog Act:- Request-Police-Addr-?
- Occur Num: 19
- NLU Output
- Request-Police-Phone-? Occur Num: 10
- Request-Attraction-Phone-? Occur Num: 4
- Request-Restaurant-Phone-? Occur Num: 2
- Inform-Police-none-none Occur Num: 1
- Request-Attraction-Addr-? Occur Num: 1
- Request-Police-Phone-?
- Occur Num: 14
- NLU Output
- Request-Police-Addr-? Occur Num: 5
- Request-Attraction-Phone-? Occur Num: 5
- Request-Restaurant-Phone-? Occur Num: 2
- Inform-Police-none-none Occur Num: 1
- Request-Restaurant-Addr-? Occur Num: 1
- Inform-Police-none-none
- Occur Num: 6
- NLU Output
- Recommend-Hotel-Name-Parkside Police Station Occur Num: 5
- Inform-Hotel-Name-Parkside Police Station Occur Num: 1
User NLU Failed Dialog Act:- Inform-Police-Addr-Parkside, Cambridge
- Occur Num: 20
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 17
- Inform-Police-Phone-01223358966 Occur Num: 3
- Inform-Police-Phone-01223358966
- Occur Num: 5
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 3
- Inform-Police-Phone-01223902088 Occur Num: 2
Dialog LoopNothing
Bad Inform Dialog ActNothing
Request But Not Inform Dialog ActNothing
Inform But Not Request Dialog ActNothing
Domain hospital
Overall Results
Success Rate: 100.0 %
(Precision, Recall, F1) : (1.000, 1.000, 1.000)
Average Dialog Turn (Succ): 4.067
Average Dialog Turn (All): 4.067
System NLU Failed Dialog Act:- Request-Hospital-Phone-?
- Occur Num: 3
- NLU Output
- Request-Restaurant-Phone-? Occur Num: 3
- Inform-Hospital-Department-children's oncology and haematology
- Occur Num: 2
- NLU Output
- Inform-Hospital-Department-oncology Occur Num: 1
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-haematology and haematological oncology
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-oncology Occur Num: 1
- Inform-Hospital-Department-oncology
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-haematology day unit
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-haematology Occur Num: 1
- Inform-Hospital-Department-hepatobillary and gastrointestinal surgery regional referral centre
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-surgery Occur Num: 1
- Inform-Hospital-Department-neurology neurosurgery
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-neurology Occur Num: 1
- Inform-Hospital-Department-oral and maxillofacial surgery and ent
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-surgery Occur Num: 1
User NLU Failed Dialog Act:- Inform-Hospital-Choice-66
- Occur Num: 57
- NLU Output
- Inform-Train-Ticket-66 Occur Num: 21
- Inform-Train-Ref-66 Occur Num: 7
- Inform-Hospital-Department-paediatric intensive care unit Occur Num: 2
- Inform-Hospital-Department-66 Occur Num: 2
- Inform-Hospital-Department-clinical decisions unit Occur Num: 2
- Recommend-Hospital-Phone-01223217667
- Occur Num: 4
- NLU Output
- Inform-Hospital-Phone-01223217667 Occur Num: 2
- Inform-Train-Ticket-66 Occur Num: 1
- Inform-Hospital-Department-postnatal Occur Num: 1
- Recommend-Hospital-Department-paediatric intensive care unit
- Occur Num: 3
- NLU Output
- Inform-Hospital-Department-paediatric intensive care unit Occur Num: 2
- Inform-Hospital-Phone-01223217715 Occur Num: 1
- Recommend-Hospital-Phone-01223217715
- Occur Num: 2
- NLU Output
- Inform-Hospital-Department-paediatric intensive care unit Occur Num: 1
- Inform-Hospital-Phone-01223217715 Occur Num: 1
- Recommend-Hospital-Department-major trauma unit
- Occur Num: 2
- NLU Output
- Inform-Hospital-Department-66 Occur Num: 1
- Inform-Hospital-Department-major trauma unit Occur Num: 1
- Recommend-Hospital-Department-clinical decisions unit
- Occur Num: 2
- NLU Output
- Inform-Hospital-Department-clinical decisions unit Occur Num: 2
- Recommend-Hospital-Phone-01223217216
- Occur Num: 2
- NLU Output
- Inform-Train-Ticket-66 Occur Num: 1
- Inform-Hospital-Phone-01223217216 Occur Num: 1
- Recommend-Hospital-Phone-01223217231
- Occur Num: 2
- NLU Output
- Inform-Train-Ticket-66 Occur Num: 1
- Inform-Hospital-Phone-01223217231 Occur Num: 1
- Recommend-Hospital-Phone-01223596066
- Occur Num: 2
- NLU Output
- Inform-Train-Ticket-66 Occur Num: 1
- Inform-Hospital-Phone-01223596066 Occur Num: 1
- Recommend-Hospital-Phone-01223217297
- Occur Num: 2
- NLU Output
- Inform-Hospital-Phone-01223217297 Occur Num: 1
- Inform-Hospital-Department-coronary care unit Occur Num: 1
Dialog LoopNothing
Bad Inform Dialog ActNothing
Request But Not Inform Dialog ActNothing
Inform But Not Request Dialog ActNothing