Test Report
Model Name: MDBT_RulePolicy_TemplateNLG
Dataset: multiwoz
Time: 2020-04-28 15:55:48
Overall Results
Success Rate: 21.2 %
(Precision, Recall, F1) : (0.522, 0.410, 0.424)
Average Dialog Turn (Succ): 11.802
Average Dialog Turn (All): 32.096
Metric
| Total Num | Succ Rate | Precision | Recall | F1 | Dialog Loop Failed Rate | Dialog Turn (Succ) | Dialog Turn (All) |
---|
hotel | 334 | 0.225 | 0.171 | 0.173 | 0.163 | 0.599 | 9.573 | 24.581 |
restaurant | 356 | 0.528 | 0.618 | 0.581 | 0.581 | 0.388 | 5.266 | 17.938 |
attraction | 298 | 0.174 | 0.499 | 0.346 | 0.388 | 0.779 | 8.577 | 30.779 |
train | 267 | 0.828 | 0.880 | 0.901 | 0.887 | 0.105 | 7.900 | 8.172 |
police | 23 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 | 0.000 | 40.000 |
hospital | 30 | 0.067 | 0.067 | 0.067 | 0.067 | 0.933 | 2.000 | 37.467 |
taxi | 51 | 0.000 | 0.000 | 0.000 | 0.000 | 0.882 | 0.000 | 35.294 |
Domain hotel
Domain restaurant
Overall Results
Success Rate: 52.8 %
(Precision, Recall, F1) : (0.618, 0.581, 0.581)
Average Dialog Turn (Succ): 5.266
Average Dialog Turn (All): 17.938
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:- Inform-Restaurant-Choice-9
- Occur Num: 153
- NLU Output
- Inform-Restaurant-Choice-13 Occur Num: 7
- Request-Train-Arrive-? Occur Num: 6
- Request-Taxi-Arrive-? Occur Num: 5
- Inform-Restaurant-Choice-44 Occur Num: 5
- Inform-Restaurant-Choice-10 Occur Num: 4
- Inform-Restaurant-Choice-2
- Occur Num: 151
- NLU Output
- Inform-Restaurant-Choice-4 Occur Num: 9
- Inform-Restaurant-Choice-44 Occur Num: 4
- Recommend-Attraction-Name-kings hedges learner pool Occur Num: 3
- Inform-Restaurant-Choice-8 Occur Num: 3
- Recommend-Restaurant-Name-Nandos Occur Num: 3
- Inform-Restaurant-Area-centre
- Occur Num: 147
- NLU Output
- Inform-Attraction-Area-centre Occur Num: 10
- Recommend-Restaurant-Area-centre Occur Num: 9
- Inform-Restaurant-Choice-44 Occur Num: 7
- Inform-Booking-none-none Occur Num: 5
- Inform-Restaurant-Choice-33 Occur Num: 4
- Inform-Restaurant-Choice-5
- Occur Num: 131
- NLU Output
- Inform-Restaurant-Choice-10 Occur Num: 7
- Inform-Restaurant-Choice-8 Occur Num: 6
- Recommend-Restaurant-Food-chinese Occur Num: 3
- Inform-Restaurant-Choice-4 Occur Num: 3
- Recommend-Restaurant-Name-restaurant two two Occur Num: 3
- Inform-Restaurant-Choice-69
- Occur Num: 127
- NLU Output
- Inform-Restaurant-Choice-44 Occur Num: 10
- Request-Train-Arrive-? Occur Num: 6
- Inform-Train-Choice-5 Occur Num: 5
- Request-Taxi-Arrive-? Occur Num: 4
- Recommend-Restaurant-Area-centre Occur Num: 4
- Inform-Restaurant-Choice-33
- Occur Num: 127
- NLU Output
- Inform-Restaurant-Choice-44 Occur Num: 18
- Inform-Booking-none-none Occur Num: 6
- Recommend-Restaurant-Name-The Occur Num: 3
- Inform-Attraction-Choice-33 Occur Num: 3
- Recommend-Attraction-Name-hughes hall Occur Num: 2
- Inform-Restaurant-Choice-3
- Occur Num: 125
- NLU Output
- Inform-Restaurant-Choice-44 Occur Num: 5
- Inform-Restaurant-Choice-10 Occur Num: 3
- Request-Train-Arrive-? Occur Num: 3
- Inform-Restaurant-Choice-13 Occur Num: 3
- Recommend-Restaurant-Name-pizza hut fen ditton Occur Num: 2
- Inform-Restaurant-Food-indian
- Occur Num: 109
- NLU Output
- Recommend-Restaurant-Food-indian Occur Num: 10
- Inform-Restaurant-Choice-4 Occur Num: 7
- Inform-Restaurant-Choice-14 Occur Num: 5
- Inform-Restaurant-Choice-44 Occur Num: 5
- Inform-Restaurant-Choice-5 Occur Num: 3
- Inform-Restaurant-Choice-21
- Occur Num: 109
- NLU Output
- Inform-Restaurant-Choice-44 Occur Num: 17
- Inform-Restaurant-Area-centre Occur Num: 4
- Recommend-Restaurant-Area-centre Occur Num: 3
- Recommend-Restaurant-Food-italian Occur Num: 3
- Recommend-Restaurant-Name-Cotto Occur Num: 3
- Inform-Restaurant-Choice-4
- Occur Num: 103
- NLU Output
- Inform-Restaurant-Food-indian Occur Num: 5
- Inform-Restaurant-Choice-44 Occur Num: 5
- Inform-Restaurant-Area-centre Occur Num: 4
- Inform-Restaurant-Food-chinese Occur Num: 3
- Inform-Restaurant-Price-expensive Occur Num: 3
Dialog Loop- Request-Restaurant-Phone-? Occur Num: 119
- Request-Restaurant-Post-? Occur Num: 4
- Request-Restaurant-Addr-? Occur Num: 3
- Request-Restaurant-Price-? Occur Num: 3
- Inform-Restaurant-Time-11:00 Occur Num: 2
- Request-Restaurant-Area-? Occur Num: 2
- Inform-Restaurant-Time-13:45 Occur Num: 1
- Inform-Restaurant-Time-17:00 Occur Num: 1
- Inform-Restaurant-Time-14:30 Occur Num: 1
- Inform-Restaurant-Time-17:45 Occur Num: 1
Bad Inform Dialog Act- recommend-restaurant-area Occur Num: 2
- inform-restaurant-area Occur Num: 1
- recommend-restaurant-postcode Occur Num: 1
Request But Not Inform Dialog Act- request-restaurant-phone Occur Num: 123
- request-restaurant-address Occur Num: 11
- request-restaurant-postcode Occur Num: 8
- request-restaurant-pricerange Occur Num: 6
- request-restaurant-area Occur Num: 2
Inform But Not Request Dialog Act- inform-restaurant-phone Occur Num: 26
- inform-restaurant-area Occur Num: 11
- inform-restaurant-food Occur Num: 4
Domain attraction
Overall Results
Success Rate: 17.4 %
(Precision, Recall, F1) : (0.499, 0.346, 0.388)
Average Dialog Turn (Succ): 8.577
Average Dialog Turn (All): 30.779
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:- Inform-Attraction-Choice-44
- Occur Num: 369
- NLU Output
- Request-Train-Arrive-? Occur Num: 28
- Inform-Train-Choice-5 Occur Num: 19
- Inform-Restaurant-Choice-69 Occur Num: 16
- Inform-Train-Choice-44 Occur Num: 13
- Request-Train-Leave-? Occur Num: 12
- Inform-Attraction-Choice-4
- Occur Num: 219
- NLU Output
- Request-Train-Arrive-? Occur Num: 28
- Inform-Train-Choice-4 Occur Num: 11
- Request-Train-Leave-? Occur Num: 8
- Inform-Restaurant-Choice-9 Occur Num: 6
- Inform-Train-Choice-5 Occur Num: 5
- Inform-Attraction-Choice-13
- Occur Num: 145
- NLU Output
- Request-Train-Arrive-? Occur Num: 11
- Inform-Attraction-Choice-14 Occur Num: 6
- Recommend-Hotel-Name-huntingdon marriott hotel Occur Num: 4
- Inform-Attraction-Choice-4 Occur Num: 4
- Recommend-Attraction-Fee-free Occur Num: 4
- Inform-Attraction-Fee-?
- Occur Num: 113
- NLU Output
- Inform-Attraction-Post-cb21ab Occur Num: 23
- Inform-Attraction-Addr-Regent Street City Centre Occur Num: 17
- Request-Train-Leave-? Occur Num: 11
- Request-Train-Arrive-? Occur Num: 10
- Inform-Attraction-Type-park Occur Num: 8
- Inform-Attraction-Choice-11
- Occur Num: 99
- NLU Output
- Inform-Train-Choice-5 Occur Num: 8
- Inform-Restaurant-Choice-69 Occur Num: 7
- Request-Train-Leave-? Occur Num: 6
- Request-Train-Arrive-? Occur Num: 6
- Inform-Hotel-Type-guesthouse Occur Num: 4
- Recommend-Attraction-Fee-free
- Occur Num: 95
- NLU Output
- Request-Train-Arrive-? Occur Num: 8
- Inform-Restaurant-Choice-69 Occur Num: 5
- Inform-Attraction-Fee-free Occur Num: 4
- Inform-Attraction-Choice-13 Occur Num: 3
- Inform-Attraction-Choice-5 Occur Num: 3
- Inform-Attraction-Choice-5
- Occur Num: 94
- NLU Output
- Request-Train-Arrive-? Occur Num: 10
- Inform-Train-Choice-5 Occur Num: 9
- Inform-Attraction-Choice-69 Occur Num: 6
- Inform-Hotel-Type-guesthouse Occur Num: 3
- Request-Train-Leave-? Occur Num: 3
- NoOffer-Attraction-Area-centre
- Occur Num: 89
- NLU Output
- NoOffer-Attraction-Type-entertainment Occur Num: 8
- Inform-Attraction-Choice-5 Occur Num: 8
- Request-Train-Arrive-? Occur Num: 7
- Inform-Restaurant-Choice-69 Occur Num: 7
- Inform-Attraction-Choice-69 Occur Num: 5
- Inform-Attraction-Choice-18
- Occur Num: 70
- NLU Output
- Request-Train-Arrive-? Occur Num: 10
- Inform-Train-Choice-18 Occur Num: 7
- Request-Train-Day-? Occur Num: 4
- Inform-Hotel-Type-guesthouse Occur Num: 3
- Recommend-Attraction-Name-trinity college Occur Num: 3
- Recommend-Attraction-Area-centre
- Occur Num: 63
- NLU Output
- Inform-Train-Choice-5 Occur Num: 4
- Inform-Attraction-Choice-5 Occur Num: 3
- Inform-Attraction-Area-centre Occur Num: 3
- Inform-Attraction-Choice-69 Occur Num: 3
- Inform-Restaurant-Choice-69 Occur Num: 3
Dialog Loop- Request-Attraction-Fee-? Occur Num: 82
- Request-Attraction-Addr-? Occur Num: 71
- Request-Attraction-Phone-? Occur Num: 47
- Request-Attraction-Post-? Occur Num: 23
- Request-Attraction-Type-? Occur Num: 9
Bad Inform Dialog Act- recommend-attraction-phone Occur Num: 4
- inform-attraction-phone Occur Num: 3
- inform-attraction-area Occur Num: 2
- recommend-attraction-postcode Occur Num: 1
- inform-attraction-postcode Occur Num: 1
- recommend-attraction-area Occur Num: 1
Request But Not Inform Dialog Act- request-attraction-phone Occur Num: 134
- request-attraction-address Occur Num: 126
- request-attraction-entrance fee Occur Num: 78
- request-attraction-postcode Occur Num: 66
- request-attraction-type Occur Num: 17
- request-attraction-area Occur Num: 6
Inform But Not Request Dialog Act- inform-attraction-phone Occur Num: 16
- inform-attraction-type Occur Num: 12
- inform-attraction-area Occur Num: 3
Domain train
Overall Results
Success Rate: 82.8 %
(Precision, Recall, F1) : (0.880, 0.901, 0.887)
Average Dialog Turn (Succ): 7.900
Average Dialog Turn (All): 8.172
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:- Request-Train-Leave-?
- Occur Num: 231
- NLU Output
- Request-Train-Arrive-? Occur Num: 44
- Request-Taxi-Arrive-? Occur Num: 31
- Request-Taxi-Leave-? Occur Num: 22
- Inform-Train-Choice-5 Occur Num: 8
- Request-Train-Day-? Occur Num: 7
- Request-Train-Arrive-?
- Occur Num: 160
- NLU Output
- Request-Taxi-Arrive-? Occur Num: 40
- Request-Taxi-Leave-? Occur Num: 10
- Inform-Train-Choice-5 Occur Num: 6
- Request-Train-Leave-? Occur Num: 5
- Inform-Hotel-Type-guesthouse Occur Num: 3
- Inform-Train-Ticket-23.60 pounds
- Occur Num: 80
- NLU Output
- Inform-Train-Time-51 minutes Occur Num: 4
- Inform-Hotel-Parking-none Occur Num: 4
- Inform-Train-Time-one Occur Num: 3
- NoOffer-Hotel-Price-moderate Occur Num: 3
- Inform-Hotel-Choice-18 Occur Num: 3
- Inform-Train-Leave-05:00
- Occur Num: 42
- NLU Output
- Select-Train-Leave-05:00 Occur Num: 11
- Select-Train-none-none Occur Num: 10
- Inform-Train-Arrive-05:51 Occur Num: 3
- Inform-Train-Arrive-05:00 Occur Num: 2
- NoOffer-Restaurant-Food-indian Occur Num: 1
- Request-Train-Day-?
- Occur Num: 27
- NLU Output
- Request-Train-Arrive-? Occur Num: 4
- Request-Taxi-Arrive-? Occur Num: 2
- Inform-Attraction-Choice-4 Occur Num: 1
- Recommend-Attraction-Name-aylesbray Occur Num: 1
- Recommend-Attraction-Phone-01223240089 Occur Num: 1
- Inform-Train-Time-51 minutes
- Occur Num: 24
- NLU Output
- Inform-Train-Time-one way Occur Num: 10
- Inform-Train-Ticket-23.60 pounds Occur Num: 5
- Inform-Train-Time-one Occur Num: 3
- Inform-Train-Arrive-05:51 Occur Num: 2
- Inform-Train-Depart-cambridge Occur Num: 1
- Inform-Train-Depart-cambridge
- Occur Num: 9
- NLU Output
- Inform-Train-Ticket-23.60 pounds Occur Num: 1
- Inform-Train-Time-51 minutes Occur Num: 1
- Inform-Train-Time-one Occur Num: 1
- Request-Restaurant-Area-? Occur Num: 1
- Request-Restaurant-Price-? Occur Num: 1
- Inform-Train-Day-monday
- Occur Num: 7
- NLU Output
- OfferBooked-Train-Day-monday Occur Num: 6
- Book-Booking-Day-monday Occur Num: 1
- Inform-Train-Arrive-05:51
- Occur Num: 7
- NLU Output
- Inform-Train-Time-51 minutes Occur Num: 2
- Inform-Train-Arrive-05:00 Occur Num: 2
- Inform-Train-Leave-05:00 Occur Num: 1
- Inform-Train-Leave-would Occur Num: 1
- Inform-Train-Time-one way Occur Num: 1
- Request-Train-Depart-?
- Occur Num: 5
- NLU Output
- Request-Train-Leave-? Occur Num: 3
- Request-Train-Dest-? Occur Num: 1
- Request-Train-Arrive-? Occur Num: 1
Dialog Loop- Inform-Train-People-5 Occur Num: 5
- Inform-Train-People-3 Occur Num: 5
- Inform-Train-People-1 Occur Num: 4
- Inform-Train-People-7 Occur Num: 4
- Inform-Train-People-6 Occur Num: 3
- Inform-Train-People-2 Occur Num: 2
- Inform-Train-People-8 Occur Num: 2
- Inform-Train-People-4 Occur Num: 1
- Request-Train-Ticket-? Occur Num: 1
- Inform-Train-Leave-09:45 Occur Num: 1
Bad Inform Dialog Act- inform-train-duration Occur Num: 13
- inform-train-leaveAt Occur Num: 1
- inform-train-price Occur Num: 1
Request But Not Inform Dialog Act- request-train-duration Occur Num: 4
- request-train-price Occur Num: 3
- request-train-trainID Occur Num: 2
- request-train-leaveAt Occur Num: 1
- request-train-arriveBy Occur Num: 1
Inform But Not Request Dialog ActNothing
Domain police
Overall Results
Success Rate: 0.0 %
(Precision, Recall, F1) : (0.000, 0.000, 0.000)
Average Dialog Turn (Succ): 0.000
Average Dialog Turn (All): 40.000
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Police-Addr-? Occur Num: 17
- Request-Police-Phone-? Occur Num: 6
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-police-phone Occur Num: 20
- request-police-address Occur Num: 17
Inform But Not Request Dialog ActNothing
Domain hospital
Overall Results
Success Rate: 6.7 %
(Precision, Recall, F1) : (0.067, 0.067, 0.067)
Average Dialog Turn (Succ): 2.000
Average Dialog Turn (All): 37.467
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:- Inform-Hospital-Department-neurosciences critical care unit
- Occur Num: 34
- NLU Output
- Request-Train-Arrive-? Occur Num: 4
- Request-Taxi-Leave-? Occur Num: 1
- Inform-Restaurant-Choice-5 Occur Num: 1
- Inform-Hotel-Name-gonville hotel Occur Num: 1
- Inform-Police-Phone-01223366611 Occur Num: 1
Dialog Loop- Request-Hospital-Phone-? Occur Num: 28
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-hospital-phone Occur Num: 28
Inform But Not Request Dialog ActNothing
Domain taxi
Overall Results
Success Rate: 0.0 %
(Precision, Recall, F1) : (0.000, 0.000, 0.000)
Average Dialog Turn (Succ): 0.000
Average Dialog Turn (All): 35.294
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:- Request-Taxi-Leave-?
- Occur Num: 173
- NLU Output
- Inform-Hotel-Type-guesthouse Occur Num: 44
- Request-Taxi-Arrive-? Occur Num: 40
- Request-Train-Arrive-? Occur Num: 9
- Inform-Train-Time-51 minutes Occur Num: 9
- Inform-Train-Leave-05:00 Occur Num: 9
- Request-Taxi-Arrive-?
- Occur Num: 97
- NLU Output
- Request-Taxi-Leave-? Occur Num: 22
- Inform-Hotel-Type-guesthouse Occur Num: 20
- Request-Train-Arrive-? Occur Num: 8
- Inform-Train-Leave-05:00 Occur Num: 8
- Inform-Train-Time-51 minutes Occur Num: 7
Dialog Loop- Request-Taxi-Car-? Occur Num: 30
- Inform-Taxi-Arrive-dontcare Occur Num: 8
- Inform-Taxi-Leave-dontcare Occur Num: 2
- Inform-Taxi-Leave-05:15 Occur Num: 1
- Inform-Taxi-Arrive-17:15 Occur Num: 1
- Inform-Taxi-Arrive-19:30 Occur Num: 1
- Inform-Taxi-Arrive-19:00 Occur Num: 1
- Inform-Taxi-Arrive-13:15 Occur Num: 1
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-taxi-phone Occur Num: 51
- request-taxi-car type Occur Num: 51
Inform But Not Request Dialog ActNothing