Test Report
Model Name: BERTNLU-RuleDST-GDPLPolicy-TemplateNLG
Dataset: multiwoz
Time: 2020-04-27 02:32:42
Overall Results
Success Rate: 49.5 %
(Precision, Recall, F1) : (0.670, 0.764, 0.682)
Average Dialog Turn (Succ): 11.479
Average Dialog Turn (All): 24.328
Metric
| Total Num | Succ Rate | Precision | Recall | F1 | Dialog Loop Failed Rate | Dialog Turn (Succ) | Dialog Turn (All) |
---|
hotel | 400 | 0.378 | 0.406 | 0.478 | 0.420 | 0.532 | 7.338 | 21.630 |
restaurant | 390 | 0.674 | 0.531 | 0.779 | 0.605 | 0.310 | 7.658 | 15.036 |
attraction | 302 | 0.881 | 0.764 | 0.920 | 0.807 | 0.099 | 6.541 | 9.642 |
taxi | 104 | 0.558 | 0.596 | 0.577 | 0.583 | 0.433 | 7.276 | 15.308 |
train | 344 | 0.820 | 0.904 | 0.914 | 0.906 | 0.102 | 5.809 | 8.331 |
police | 23 | 1.000 | 0.587 | 1.000 | 0.733 | 0.000 | 2.000 | 2.000 |
hospital | 30 | 0.933 | 0.933 | 0.933 | 0.933 | 0.067 | 4.000 | 6.400 |
Domain hotel
Domain restaurant
Domain attraction
Domain taxi
Domain train
Domain police
Overall Results
Success Rate: 100.0 %
(Precision, Recall, F1) : (0.587, 1.000, 0.733)
Average Dialog Turn (Succ): 2.000
Average Dialog Turn (All): 2.000
System NLU Failed Dialog Act:- Inform-Police-none-none
- Occur Num: 7
- NLU Output
- Inform-Police-Name-Parkside Police Station Occur Num: 6
- Request-Police-Addr-? Occur Num: 1
User NLU Failed Dialog Act:- Inform-Police-Addr-Parkside, Cambridge
- Occur Num: 68
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 23
- Inform-Police-Phone-01223358966 Occur Num: 23
- Inform-Police-Post-none Occur Num: 22
- Inform-Police-Post-none
- Occur Num: 46
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 23
- Inform-Police-Phone-01223358966 Occur Num: 23
- Inform-Police-Phone-01223358966
- Occur Num: 45
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 23
- Inform-Police-Post-none Occur Num: 22
- Inform-Police-Name-none
- Occur Num: 2
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 1
- Inform-Police-Phone-01223358966 Occur Num: 1
Dialog LoopNothing
Bad Inform Dialog Act- inform-police-postcode Occur Num: 22
Request But Not Inform Dialog ActNothing
Inform But Not Request Dialog Act- inform-police-phone Occur Num: 3
Domain hospital
Overall Results
Success Rate: 93.3 %
(Precision, Recall, F1) : (0.933, 0.933, 0.933)
Average Dialog Turn (Succ): 4.000
Average Dialog Turn (All): 6.400
System NLU Failed Dialog Act:- Request-Hospital-Phone-?
- Occur Num: 5
- NLU Output
- Request-Police-Phone-? Occur Num: 2
- Request-Restaurant-Phone-? Occur Num: 2
- Request-Attraction-Phone-? Occur Num: 1
- Inform-Hospital-Department-plastic and vascular surgery plastics
- Occur Num: 2
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-plastic and vascular surgery Occur Num: 1
- Inform-Hospital-Department-haematology and haematological oncology
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-neurosurgery Occur Num: 1
- Inform-Hospital-Department-oncology
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-surgery
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-intermediate dependancy area
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-intermediate dependancy Occur Num: 1
- Inform-Hospital-Department-children's oncology and haematology
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-emergency department
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-emergency department department Occur Num: 1
User NLU Failed Dialog Act:- Inform-Hospital-Addr-none
- Occur Num: 84
- NLU Output
- Request-Hospital-Department-? Occur Num: 28
- Inform-Hospital-Department-neurosciences critical care unit Occur Num: 28
- Inform-Hospital-Phone-01223216297 Occur Num: 28
- Inform-Hospital-Post-none
- Occur Num: 84
- NLU Output
- Request-Hospital-Department-? Occur Num: 28
- Inform-Hospital-Department-neurosciences critical care unit Occur Num: 28
- Inform-Hospital-Phone-01223216297 Occur Num: 28
- Inform-Hospital-Department-neurosciences critical care unit
- Occur Num: 56
- NLU Output
- Request-Hospital-Department-? Occur Num: 28
- Inform-Hospital-Phone-01223216297 Occur Num: 28
- Inform-Hospital-Phone-01223216297
- Occur Num: 56
- NLU Output
- Request-Hospital-Department-? Occur Num: 28
- Inform-Hospital-Department-neurosciences critical care unit Occur Num: 28
- Request-Hospital-Department-?
- Occur Num: 56
- NLU Output
- Inform-Hospital-Department-neurosciences critical care unit Occur Num: 28
- Inform-Hospital-Phone-01223216297 Occur Num: 28
Dialog Loop- Request-Hospital-Phone-? Occur Num: 2
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-hospital-phone Occur Num: 2
Inform But Not Request Dialog ActNothing