Test Report
Model Name: BERTNLU-RuleDST-PGPolicy-TemplateNLG
Dataset: multiwoz
Time: 2020-04-27 13:55:52
Overall Results
Success Rate: 43.3 %
(Precision, Recall, F1) : (0.619, 0.668, 0.604)
Average Dialog Turn (Succ): 14.693
Average Dialog Turn (All): 29.068
Metric
| Total Num | Succ Rate | Precision | Recall | F1 | Dialog Loop Failed Rate | Dialog Turn (Succ) | Dialog Turn (All) |
---|
hotel | 345 | 0.400 | 0.323 | 0.465 | 0.353 | 0.490 | 7.522 | 21.154 |
restaurant | 383 | 0.603 | 0.497 | 0.713 | 0.556 | 0.389 | 11.247 | 21.770 |
train | 314 | 0.710 | 0.810 | 0.790 | 0.790 | 0.280 | 12.404 | 18.930 |
attraction | 269 | 0.844 | 0.764 | 0.897 | 0.799 | 0.141 | 6.907 | 11.346 |
taxi | 86 | 0.721 | 0.767 | 0.744 | 0.752 | 0.267 | 7.258 | 12.558 |
police | 23 | 1.000 | 0.935 | 1.000 | 0.957 | 0.000 | 2.000 | 2.000 |
hospital | 30 | 0.933 | 0.933 | 0.933 | 0.933 | 0.067 | 38.429 | 38.533 |
Domain hotel
Domain restaurant
Domain train
Domain attraction
Domain taxi
Domain police
Overall Results
Success Rate: 100.0 %
(Precision, Recall, F1) : (0.935, 1.000, 0.957)
Average Dialog Turn (Succ): 2.000
Average Dialog Turn (All): 2.000
System NLU Failed Dialog Act:- Inform-Police-none-none
- Occur Num: 7
- NLU Output
- Inform-Police-Name-Parkside Police Station Occur Num: 6
- Request-Police-Addr-? Occur Num: 1
User NLU Failed Dialog Act:- Inform-Police-Addr-Parkside, Cambridge
- Occur Num: 46
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 23
- Inform-Police-Phone-01223358966 Occur Num: 23
- Inform-Police-Name-none
- Occur Num: 46
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 23
- Inform-Police-Phone-01223358966 Occur Num: 23
- Inform-Police-Post-none
- Occur Num: 46
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 23
- Inform-Police-Phone-01223358966 Occur Num: 23
- Inform-Police-Phone-01223358966
- Occur Num: 23
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 23
Dialog LoopNothing
Bad Inform Dialog ActNothing
Request But Not Inform Dialog ActNothing
Inform But Not Request Dialog Act- inform-police-phone Occur Num: 3
Domain hospital
Overall Results
Success Rate: 93.3 %
(Precision, Recall, F1) : (0.933, 0.933, 0.933)
Average Dialog Turn (Succ): 38.429
Average Dialog Turn (All): 38.533
System NLU Failed Dialog Act:- Inform-Hospital-Department-plastic and vascular surgery plastics
- Occur Num: 5
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-plastic and vascular surgery Occur Num: 1
- Inform-Hospital-Department-plastic Occur Num: 1
- Inform-Hospital-Department-vascular surgery Occur Num: 1
- Inform-Hospital-Department-neurosurgery Occur Num: 1
- Request-Hospital-Phone-?
- Occur Num: 4
- NLU Output
- Request-Police-Phone-? Occur Num: 2
- Request-Restaurant-Phone-? Occur Num: 2
- Inform-Hospital-Department-emergency department
- Occur Num: 4
- NLU Output
- Inform-Hospital-Department-emergency department department Occur Num: 2
- Inform-Hospital-Department-neurosurgery Occur Num: 2
- Inform-Hospital-Department-hepatobillary and gastrointestinal surgery regional referral centre
- Occur Num: 2
- NLU Output
- Inform-Hospital-Department-neurosurgery Occur Num: 1
- Inform-Hospital-Department-hepatobillary Occur Num: 1
- Inform-Hospital-Department-haematology and haematological oncology
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-neurosurgery Occur Num: 1
- Inform-Hospital-Department-acute medicine for the elderly
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-neurosurgery Occur Num: 1
- Inform-Hospital-Department-neonatal unit
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-neurosurgery Occur Num: 1
- Inform-Hospital-Department-oncology
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-trauma and orthopaedics
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-trauma and orthopaedics neurosurgery Occur Num: 1
- Inform-Hospital-Department-transplant high dependency unit
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-neurosurgery Occur Num: 1
User NLU Failed Dialog Act:- Inform-Hospital-Addr-none
- Occur Num: 58
- NLU Output
- Inform-Hospital-Department-neurosciences critical care unit Occur Num: 28
- Inform-Hospital-Phone-01223216297 Occur Num: 28
- Request-Hospital-Department-? Occur Num: 2
- Inform-Hospital-Post-none
- Occur Num: 58
- NLU Output
- Inform-Hospital-Department-neurosciences critical care unit Occur Num: 28
- Inform-Hospital-Phone-01223216297 Occur Num: 28
- Request-Hospital-Department-? Occur Num: 2
- Inform-Hospital-Department-neurosciences critical care unit
- Occur Num: 30
- NLU Output
- Inform-Hospital-Phone-01223216297 Occur Num: 28
- Request-Hospital-Department-? Occur Num: 2
- Inform-Hospital-Phone-01223216297
- Occur Num: 30
- NLU Output
- Inform-Hospital-Department-neurosciences critical care unit Occur Num: 28
- Request-Hospital-Department-? Occur Num: 2
- Request-Hospital-Department-?
- Occur Num: 4
- NLU Output
- Inform-Hospital-Department-neurosciences critical care unit Occur Num: 2
- Inform-Hospital-Phone-01223216297 Occur Num: 2
Dialog Loop- Request-Hospital-Phone-? Occur Num: 2
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-hospital-phone Occur Num: 2
Inform But Not Request Dialog ActNothing