Test Report
Model Name: MILU-RuleDST-RulePolicy-TemplateNLG
Dataset: multiwoz
Time: 2020-04-24 21:12:47
Overall Results
Success Rate: 83.1 %
(Precision, Recall, F1) : (0.783, 0.917, 0.825)
Average Dialog Turn (Succ): 12.144
Average Dialog Turn (All): 13.854
Metric
| Total Num | Succ Rate | Precision | Recall | F1 | Dialog Loop Failed Rate | Dialog Turn (Succ) | Dialog Turn (All) |
---|
hotel | 429 | 0.751 | 0.482 | 0.649 | 0.526 | 0.058 | 5.596 | 6.410 |
restaurant | 411 | 0.954 | 0.801 | 0.850 | 0.818 | 0.032 | 4.949 | 5.781 |
attraction | 331 | 0.940 | 0.878 | 0.953 | 0.905 | 0.027 | 5.633 | 6.278 |
taxi | 173 | 0.879 | 0.908 | 0.893 | 0.898 | 0.121 | 7.026 | 9.399 |
train | 389 | 0.995 | 0.840 | 0.928 | 0.867 | 0.005 | 5.767 | 5.779 |
police | 23 | 0.913 | 0.913 | 0.913 | 0.913 | 0.087 | 5.524 | 8.522 |
hospital | 30 | 1.000 | 1.000 | 1.000 | 1.000 | 0.000 | 4.067 | 4.067 |
Domain hotel
Domain restaurant
Domain attraction
Domain taxi
Domain train
Domain police
Overall Results
Success Rate: 91.3 %
(Precision, Recall, F1) : (0.913, 0.913, 0.913)
Average Dialog Turn (Succ): 5.524
Average Dialog Turn (All): 8.522
System NLU Failed Dialog Act:- Request-Police-Addr-?
- Occur Num: 32
- NLU Output
- Request-Police-Phone-? Occur Num: 12
- Request-Police-Post-? Occur Num: 4
- Request-Attraction-Addr-? Occur Num: 3
- Request-Hospital-Addr-? Occur Num: 2
- Request-Attraction-Post-? Occur Num: 2
- Request-Police-Phone-?
- Occur Num: 28
- NLU Output
- Request-Police-Addr-? Occur Num: 11
- Request-Police-Post-? Occur Num: 4
- Request-Attraction-Addr-? Occur Num: 2
- Request-Restaurant-Addr-? Occur Num: 2
- Request-Restaurant-Phone-? Occur Num: 2
- Inform-Police-none-none
- Occur Num: 6
- NLU Output
- Inform-Police-Name-Parkside Police Station Occur Num: 6
User NLU Failed Dialog Act:- Inform-Police-Addr-Parkside, Cambridge
- Occur Num: 29
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 15
- Inform-Police-Phone-01223358966 Occur Num: 11
- Inform-Attraction-Addr-pool way Occur Num: 1
- Inform-Attraction-Addr-whitehill road Occur Num: 1
- Inform-Attraction-Addr-off newmarket road Occur Num: 1
- Inform-Police-Phone-01223358966
- Occur Num: 11
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 11
- Inform-Police-Post-unknown
- Occur Num: 8
- NLU Output
- Inform-Police-Addr-Parkside , Cambridge Occur Num: 4
- Inform-Police-Phone-01223358966 Occur Num: 4
Dialog Loop- Request-Police-Addr-? Occur Num: 2
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-police-phone Occur Num: 2
- request-police-address Occur Num: 2
Inform But Not Request Dialog ActNothing
Domain hospital
Overall Results
Success Rate: 100.0 %
(Precision, Recall, F1) : (1.000, 1.000, 1.000)
Average Dialog Turn (Succ): 4.067
Average Dialog Turn (All): 4.067
System NLU Failed Dialog Act:- Request-Hospital-Phone-?
- Occur Num: 3
- NLU Output
- Request-Hospital-Post-? Occur Num: 1
- Request-Hospital-Addr-? Occur Num: 1
- Request-Restaurant-Phone-? Occur Num: 1
- Inform-Hospital-Department-acute medicine for the elderly
- Occur Num: 2
- NLU Output
- Inform-Hospital-Department-acute medicine Occur Num: 1
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-paediatric clinic
- Occur Num: 2
- NLU Output
- Inform-Hospital-Department-paediatric clinic department Occur Num: 2
- Inform-Hospital-Department-plastic and vascular surgery plastics
- Occur Num: 2
- NLU Output
- Inform-Hospital-Department-and vascular surgery Occur Num: 1
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-neurology
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-neurology neurosurgery Occur Num: 1
- Inform-Hospital-Department-oncology
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-hepatobillary and gastrointestinal surgery regional referral centre
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-hepatobillary and gastrointestinal surgery regional Occur Num: 1
- Inform-Hospital-Department-oral and maxillofacial surgery and ent
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-oral and maxillofacial surgery and ent neurosurgery Occur Num: 1
User NLU Failed Dialog Act:- Inform-Hospital-Choice-66
- Occur Num: 51
- NLU Output
- Inform-Train-Ticket-66 Occur Num: 19
- Inform-Train-Ref-66 Occur Num: 7
- Inform-Hospital-Department-paediatric intensive care unit Occur Num: 2
- Inform-Hospital-Department-66 Occur Num: 2
- Inform-Hospital-Phone-01223217715 Occur Num: 1
- Recommend-Hospital-Department-paediatric intensive care unit
- Occur Num: 3
- NLU Output
- Inform-Hospital-Department-paediatric intensive care unit Occur Num: 2
- Inform-Hospital-Phone-01223217715 Occur Num: 1
- Recommend-Hospital-Phone-01223217715
- Occur Num: 2
- NLU Output
- Inform-Hospital-Department-paediatric intensive care unit Occur Num: 1
- Inform-Hospital-Phone-01223217715 Occur Num: 1
- Recommend-Hospital-Department-major trauma unit
- Occur Num: 2
- NLU Output
- Inform-Hospital-Department-66 Occur Num: 1
- Inform-Hospital-Department-major trauma unit Occur Num: 1
- Recommend-Hospital-Phone-01223217216
- Occur Num: 2
- NLU Output
- Inform-Train-Ticket-66 Occur Num: 1
- Inform-Hospital-Phone-01223217216 Occur Num: 1
- Recommend-Hospital-Phone-01223217231
- Occur Num: 2
- NLU Output
- Inform-Train-Ticket-66 Occur Num: 1
- Inform-Hospital-Phone-01223217231 Occur Num: 1
- Recommend-Hospital-Phone-01223596066
- Occur Num: 2
- NLU Output
- Inform-Train-Ticket-66 Occur Num: 1
- Inform-Hospital-Phone-01223596066 Occur Num: 1
- Recommend-Hospital-Phone-01223217297
- Occur Num: 2
- NLU Output
- Inform-Hospital-Phone-01223217297 Occur Num: 1
- Inform-Hospital-Department-coronary care unit Occur Num: 1
- Recommend-Hospital-Department-coronary care unit
- Occur Num: 2
- NLU Output
- Inform-Hospital-Phone-01223217297 Occur Num: 1
- Inform-Hospital-Department-coronary care unit Occur Num: 1
- Recommend-Hospital-Phone-01223348336
- Occur Num: 2
- NLU Output
- Inform-Train-Ticket-66 Occur Num: 1
- Inform-Hospital-Phone-01223348336 Occur Num: 1
Dialog LoopNothing
Bad Inform Dialog ActNothing
Request But Not Inform Dialog ActNothing
Inform But Not Request Dialog ActNothing