Test Report
Model Name: TRADE-RulePolicy-TemplateNLG
Dataset: multiwoz
Time: 2020-04-29 21:10:04
Overall Results
Success Rate: 25.3 %
(Precision, Recall, F1) : (0.493, 0.481, 0.444)
Average Dialog Turn (Succ): 12.735
Average Dialog Turn (All): 24.684
Metric
| Total Num | Succ Rate | Precision | Recall | F1 | Dialog Loop Failed Rate | Dialog Turn (Succ) | Dialog Turn (All) |
---|
hotel | 330 | 0.285 | 0.167 | 0.286 | 0.188 | 0.473 | 9.617 | 16.448 |
attraction | 291 | 0.299 | 0.614 | 0.541 | 0.547 | 0.533 | 6.368 | 20.192 |
restaurant | 361 | 0.335 | 0.479 | 0.490 | 0.465 | 0.427 | 4.744 | 14.964 |
train | 273 | 0.897 | 0.795 | 0.865 | 0.815 | 0.084 | 9.902 | 11.773 |
police | 23 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 | 0.000 | 40.000 |
hospital | 30 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 | 0.000 | 40.000 |
taxi | 46 | 0.935 | 1.000 | 0.967 | 0.978 | 0.065 | 5.209 | 7.478 |
Dialogue Loop

Domain hotel
Domain attraction
Overall Results
Success Rate: 29.9 %
(Precision, Recall, F1) : (0.614, 0.541, 0.547)
Average Dialog Turn (Succ): 6.368
Average Dialog Turn (All): 20.192
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:- Inform-Attraction-Area-centre
- Occur Num: 292
- NLU Output
- Inform-Attraction-Post-cb21ab Occur Num: 23
- Inform-Attraction-Type-museum Occur Num: 18
- Inform-Attraction-Addr-Regent Street City Centre Occur Num: 18
- Inform-Attraction-Area-within Occur Num: 12
- Inform-Attraction-Type-park Occur Num: 11
- Inform-Attraction-Type-museum
- Occur Num: 156
- NLU Output
- Inform-Attraction-Area-centre Occur Num: 20
- Inform-Attraction-Post-cb21ab Occur Num: 16
- Inform-Attraction-Addr-Regent Street City Centre Occur Num: 15
- Inform-Attraction-Fee-free Occur Num: 7
- Recommend-Attraction-Type-museum Occur Num: 6
- Request-Attraction-Type-?
- Occur Num: 138
- NLU Output
- NoOffer-Attraction-Area-centre Occur Num: 13
- NoOffer-Attraction-Type-entertainment Occur Num: 11
- Request-Attraction-Name-? Occur Num: 7
- NoOffer-Attraction-Type-college Occur Num: 6
- Request-Attraction-Area-? Occur Num: 6
- NoOffer-Attraction-Area-centre
- Occur Num: 130
- NLU Output
- NoOffer-Attraction-Type-entertainment Occur Num: 16
- Request-Attraction-Type-? Occur Num: 12
- NoOffer-Attraction-Area-that area of town Occur Num: 7
- NoOffer-Attraction-Type-college Occur Num: 6
- Inform-Hotel-Type-guesthouse Occur Num: 5
- Inform-Attraction-Fee-?
- Occur Num: 117
- NLU Output
- Inform-Attraction-Addr-Regent Street City Centre Occur Num: 17
- Inform-Attraction-Post-cb21ab Occur Num: 17
- Inform-Attraction-Area-centre Occur Num: 7
- Request-Taxi-Leave-? Occur Num: 6
- Inform-Attraction-Area-north Occur Num: 4
- Inform-Attraction-Fee-free
- Occur Num: 102
- NLU Output
- Inform-Attraction-Post-cb21ab Occur Num: 14
- Inform-Attraction-Addr-Regent Street City Centre Occur Num: 12
- Inform-Attraction-Area-centre Occur Num: 11
- Inform-Attraction-Type-park Occur Num: 8
- Request-Train-Arrive-? Occur Num: 4
- Inform-Attraction-Choice-44
- Occur Num: 94
- NLU Output
- reqmore-general-none-none Occur Num: 3
- Request-Attraction-Type-? Occur Num: 2
- Recommend-Attraction-Name-primavera Occur Num: 2
- Recommend-Attraction-Name-scudamores punting co Occur Num: 2
- Recommend-Attraction-Post-cb21qy Occur Num: 2
- NoOffer-Attraction-Type-college
- Occur Num: 93
- NLU Output
- NoOffer-Attraction-Area-west Occur Num: 9
- Request-Attraction-Type-? Occur Num: 8
- NoOffer-Attraction-Type-entertainment Occur Num: 7
- Inform-Attraction-Post-cb21ab Occur Num: 5
- NoOffer-Attraction-Area-centre Occur Num: 5
- Request-Attraction-Name-?
- Occur Num: 84
- NLU Output
- Request-Attraction-Type-? Occur Num: 7
- NoOffer-Attraction-Type-college Occur Num: 5
- Inform-Attraction-Addr-Regent Street City Centre Occur Num: 5
- NoOffer-Attraction-Area-centre Occur Num: 5
- NoOffer-Attraction-Type-entertainment Occur Num: 4
- NoOffer-Attraction-Area-west
- Occur Num: 72
- NLU Output
- NoOffer-Attraction-Type-college Occur Num: 7
- NoOffer-Attraction-Type-entertainment Occur Num: 5
- Inform-Attraction-Post-cb21ab Occur Num: 5
- Request-Attraction-Type-? Occur Num: 4
- NoOffer-Attraction-Type-museum Occur Num: 4
Dialog Loop- Request-Attraction-Phone-? Occur Num: 78
- Request-Attraction-Fee-? Occur Num: 42
- Request-Attraction-Addr-? Occur Num: 23
- Request-Attraction-Post-? Occur Num: 9
- Inform-Attraction-Type-college Occur Num: 1
- Inform-Attraction-Type-nightclub Occur Num: 1
- Request-Attraction-Type-? Occur Num: 1
Bad Inform Dialog Act- inform-attraction-area Occur Num: 33
- recommend-attraction-area Occur Num: 2
- inform-attraction-phone Occur Num: 1
- recommend-attraction-phone Occur Num: 1
Request But Not Inform Dialog Act- request-attraction-phone Occur Num: 140
- request-attraction-entrance fee Occur Num: 59
- request-attraction-address Occur Num: 54
- request-attraction-postcode Occur Num: 28
- request-attraction-type Occur Num: 5
- request-attraction-area Occur Num: 1
Inform But Not Request Dialog Act- inform-attraction-area Occur Num: 47
- inform-attraction-type Occur Num: 43
- inform-attraction-phone Occur Num: 7
Domain restaurant
Overall Results
Success Rate: 33.5 %
(Precision, Recall, F1) : (0.479, 0.490, 0.465)
Average Dialog Turn (Succ): 4.744
Average Dialog Turn (All): 14.964
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:- Inform-Restaurant-Area-centre
- Occur Num: 269
- NLU Output
- Inform-Restaurant-Price-expensive Occur Num: 20
- Inform-Attraction-Area-centre Occur Num: 14
- Request-Taxi-Arrive-? Occur Num: 14
- Inform-Restaurant-Price-moderate Occur Num: 13
- Inform-Restaurant-Price-cheap Occur Num: 11
- Inform-Restaurant-Price-expensive
- Occur Num: 187
- NLU Output
- Inform-Restaurant-Area-centre Occur Num: 19
- Request-Taxi-Arrive-? Occur Num: 8
- Inform-Restaurant-Food-indian Occur Num: 8
- Request-Taxi-Leave-? Occur Num: 6
- Recommend-Restaurant-Name-bangkok city Occur Num: 5
- Request-Restaurant-Food-?
- Occur Num: 173
- NLU Output
- NoOffer-Restaurant-none-none Occur Num: 20
- Request-Restaurant-Price-? Occur Num: 17
- NoOffer-Restaurant-Area-centre Occur Num: 16
- NoOffer-Restaurant-Price-expensive Occur Num: 15
- Request-Restaurant-Area-? Occur Num: 11
- NoOffer-Restaurant-Price-expensive
- Occur Num: 163
- NLU Output
- NoOffer-Restaurant-none-none Occur Num: 24
- NoOffer-Restaurant-Area-centre Occur Num: 15
- Request-Restaurant-Food-? Occur Num: 14
- Request-Restaurant-Price-? Occur Num: 9
- NoOffer-Restaurant-Food-indian Occur Num: 6
- NoOffer-Restaurant-Area-centre
- Occur Num: 151
- NLU Output
- NoOffer-Restaurant-none-none Occur Num: 21
- Request-Restaurant-Food-? Occur Num: 16
- NoOffer-Restaurant-Price-expensive Occur Num: 14
- NoOffer-Restaurant-Price-cheap Occur Num: 8
- NoOffer-Restaurant-Food-indian Occur Num: 8
- Request-Restaurant-Price-?
- Occur Num: 119
- NLU Output
- Request-Restaurant-Food-? Occur Num: 19
- NoOffer-Restaurant-none-none Occur Num: 10
- NoOffer-Restaurant-Price-expensive Occur Num: 10
- NoOffer-Restaurant-Area-centre Occur Num: 8
- Request-Restaurant-Area-? Occur Num: 7
- Inform-Restaurant-Food-indian
- Occur Num: 118
- NLU Output
- Inform-Restaurant-Price-expensive Occur Num: 8
- Inform-Restaurant-Choice-4 Occur Num: 6
- Recommend-Restaurant-Food-indian Occur Num: 5
- Inform-Restaurant-Area-east Occur Num: 5
- Inform-Restaurant-Area-centre Occur Num: 4
- NoOffer-Restaurant-Price-moderate
- Occur Num: 99
- NLU Output
- NoOffer-Restaurant-none-none Occur Num: 14
- Request-Restaurant-Food-? Occur Num: 8
- NoOffer-Restaurant-Food-indian Occur Num: 7
- NoOffer-Restaurant-Area-centre Occur Num: 7
- NoOffer-Restaurant-Area-east Occur Num: 5
- NoOffer-Restaurant-Price-cheap
- Occur Num: 95
- NLU Output
- NoOffer-Restaurant-none-none Occur Num: 15
- NoOffer-Restaurant-Area-centre Occur Num: 10
- Request-Restaurant-Food-? Occur Num: 10
- Request-Restaurant-Price-? Occur Num: 7
- NoOffer-Restaurant-Area-east Occur Num: 4
- Inform-Restaurant-Price-cheap
- Occur Num: 94
- NLU Output
- Inform-Restaurant-Area-centre Occur Num: 10
- Inform-Restaurant-Food-italian Occur Num: 6
- Recommend-Restaurant-Price-cheap Occur Num: 5
- Request-Taxi-Arrive-? Occur Num: 3
- Recommend-Restaurant-Name-la raza Occur Num: 3
Dialog Loop- Request-Restaurant-Phone-? Occur Num: 108
- Request-Restaurant-Addr-? Occur Num: 7
- Request-Restaurant-Post-? Occur Num: 4
- Inform-Restaurant-Day-saturday Occur Num: 3
- Inform-Restaurant-People-4 Occur Num: 2
- Inform-Restaurant-Time-14:15 Occur Num: 2
- Inform-Restaurant-Day-sunday Occur Num: 2
- Request-Restaurant-Price-? Occur Num: 2
- Inform-Restaurant-Time-16:45 Occur Num: 1
- Inform-Restaurant-Time-13:30 Occur Num: 1
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-restaurant-phone Occur Num: 127
- request-restaurant-address Occur Num: 41
- request-restaurant-postcode Occur Num: 19
- request-restaurant-pricerange Occur Num: 3
Inform But Not Request Dialog Act- inform-restaurant-food Occur Num: 56
- inform-restaurant-area Occur Num: 50
- inform-restaurant-phone Occur Num: 18
Domain train
Overall Results
Success Rate: 89.7 %
(Precision, Recall, F1) : (0.795, 0.865, 0.815)
Average Dialog Turn (Succ): 9.902
Average Dialog Turn (All): 11.773
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:- OfferBook-Train-Depart-cambridge
- Occur Num: 712
- NLU Output
- OfferBook-Train-none-none Occur Num: 43
- OfferBook-Train-Dest-cambridge Occur Num: 41
- OfferBook-Train-Day-tuesday Occur Num: 24
- OfferBook-Train-Arrive-05:51 Occur Num: 24
- OfferBook-Train-Day-thursday Occur Num: 22
- OfferBook-Train-Dest-cambridge
- Occur Num: 641
- NLU Output
- OfferBook-Train-none-none Occur Num: 50
- OfferBook-Train-Arrive-05:51 Occur Num: 33
- OfferBook-Train-Depart-cambridge Occur Num: 27
- OfferBook-Train-Day-saturday Occur Num: 26
- OfferBook-Train-Day-tuesday Occur Num: 23
- Inform-Train-Arrive-05:51
- Occur Num: 475
- NLU Output
- OfferBook-Train-Arrive-05:51 Occur Num: 67
- OfferBook-Train-Dest-cambridge Occur Num: 37
- OfferBook-Train-Depart-cambridge Occur Num: 31
- OfferBook-Train-Leave-05:00 Occur Num: 21
- OfferBook-Train-none-none Occur Num: 17
- OfferBook-Train-Day-tuesday
- Occur Num: 256
- NLU Output
- OfferBook-Train-Dest-cambridge Occur Num: 21
- OfferBook-Train-Depart-cambridge Occur Num: 21
- OfferBook-Train-none-none Occur Num: 14
- OfferBook-Train-Arrive-05:51 Occur Num: 12
- OfferBook-Train-Leave-05:00 Occur Num: 7
- OfferBook-Train-Day-saturday
- Occur Num: 254
- NLU Output
- OfferBook-Train-Dest-cambridge Occur Num: 26
- OfferBook-Train-Depart-cambridge Occur Num: 18
- OfferBook-Train-none-none Occur Num: 13
- OfferBook-Train-Arrive-05:51 Occur Num: 12
- OfferBook-Train-Leave-05:00 Occur Num: 8
- Inform-Train-Depart-cambridge
- Occur Num: 253
- NLU Output
- OfferBook-Train-Depart-cambridge Occur Num: 32
- OfferBook-Train-Dest-cambridge Occur Num: 14
- OfferBook-Train-Arrive-05:51 Occur Num: 13
- OfferBook-Train-Day-tuesday Occur Num: 10
- OfferBook-Train-Day-thursday Occur Num: 7
- Inform-Train-Leave-05:00
- Occur Num: 242
- NLU Output
- OfferBook-Train-Leave-05:00 Occur Num: 37
- OfferBook-Train-Arrive-05:51 Occur Num: 16
- OfferBook-Train-Dest-cambridge Occur Num: 13
- OfferBook-Train-Depart-cambridge Occur Num: 10
- OfferBook-Train-Day-saturday Occur Num: 8
- OfferBook-Train-Day-friday
- Occur Num: 237
- NLU Output
- OfferBook-Train-Depart-cambridge Occur Num: 20
- OfferBook-Train-Dest-cambridge Occur Num: 19
- OfferBook-Train-none-none Occur Num: 13
- OfferBook-Train-Arrive-05:51 Occur Num: 11
- OfferBook-Train-Leave-05:00 Occur Num: 7
- OfferBook-Train-Day-thursday
- Occur Num: 233
- NLU Output
- OfferBook-Train-Depart-cambridge Occur Num: 23
- OfferBook-Train-Dest-cambridge Occur Num: 17
- OfferBook-Train-none-none Occur Num: 16
- OfferBook-Train-Arrive-05:51 Occur Num: 9
- OfferBook-Train-Leave-05:00 Occur Num: 5
- Inform-Train-Time-51 minutes
- Occur Num: 194
- NLU Output
- OfferBook-Train-Depart-cambridge Occur Num: 13
- OfferBook-Train-Dest-cambridge Occur Num: 13
- Inform-Train-Time-one way Occur Num: 10
- OfferBook-Train-Arrive-05:51 Occur Num: 9
- OfferBook-Train-People-51 Occur Num: 8
Dialog Loop- Request-Train-Time-? Occur Num: 9
- Inform-Train-People-6 Occur Num: 2
- Inform-Train-People-4 Occur Num: 2
- Inform-Train-People-3 Occur Num: 2
- Inform-Train-People-5 Occur Num: 1
- Inform-Train-People-7 Occur Num: 1
- Inform-Train-Depart-london liverpool street Occur Num: 1
- Inform-Train-Dest-cambridge Occur Num: 1
- Inform-Train-Dest-birmingham new street Occur Num: 1
- Request-Train-Leave-? Occur Num: 1
Bad Inform Dialog Act- offerbook-train-arriveBy Occur Num: 30
- inform-train-duration Occur Num: 14
- offerbook-train-leaveAt Occur Num: 8
- offerbook-train-price Occur Num: 2
- inform-train-price Occur Num: 1
- offerbook-train-trainID Occur Num: 1
Request But Not Inform Dialog Act- request-train-duration Occur Num: 13
- request-train-price Occur Num: 2
- request-train-leaveAt Occur Num: 2
- request-train-arriveBy Occur Num: 1
- request-train-trainID Occur Num: 1
Inform But Not Request Dialog ActNothing
Domain police
Overall Results
Success Rate: 0.0 %
(Precision, Recall, F1) : (0.000, 0.000, 0.000)
Average Dialog Turn (Succ): 0.000
Average Dialog Turn (All): 40.000
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Police-Addr-? Occur Num: 17
- Request-Police-Phone-? Occur Num: 6
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-police-phone Occur Num: 20
- request-police-address Occur Num: 17
Inform But Not Request Dialog ActNothing
Domain hospital
Overall Results
Success Rate: 0.0 %
(Precision, Recall, F1) : (0.000, 0.000, 0.000)
Average Dialog Turn (Succ): 0.000
Average Dialog Turn (All): 40.000
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Hospital-Phone-? Occur Num: 30
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-hospital-phone Occur Num: 30
Inform But Not Request Dialog ActNothing
Domain taxi
Overall Results
Success Rate: 93.5 %
(Precision, Recall, F1) : (1.000, 0.967, 0.978)
Average Dialog Turn (Succ): 5.209
Average Dialog Turn (All): 7.478
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:- Inform-Taxi-Car-skoda
- Occur Num: 27
- NLU Output
- Inform-Taxi-Leave-05:00 Occur Num: 3
- Inform-Hotel-Type-guesthouse Occur Num: 2
- Inform-Train-Time-51 minutes Occur Num: 2
- Inform-Taxi-Phone-62698234072 Occur Num: 1
- Inform-Taxi-Car-toyota Occur Num: 1
- Inform-Taxi-Car-toyota
- Occur Num: 17
- NLU Output
- Inform-Hotel-Type-guesthouse Occur Num: 3
- Inform-Taxi-Leave-05:00 Occur Num: 2
- Inform-Taxi-Phone-62698234072 Occur Num: 1
- Inform-Taxi-Phone-83153889561 Occur Num: 1
- Inform-Taxi-Phone-69714552379 Occur Num: 1
- Inform-Taxi-Car-audi
- Occur Num: 16
- NLU Output
- Inform-Taxi-Leave-05:00 Occur Num: 2
- Inform-Taxi-Phone-73297457818 Occur Num: 1
- Inform-Taxi-Phone-02411795305 Occur Num: 1
- OfferBooked-Train-Time-51 Occur Num: 1
- Inform-Taxi-Phone-17927153736 Occur Num: 1
- Inform-Taxi-Car-ford
- Occur Num: 14
- NLU Output
- Inform-Hotel-Type-guesthouse Occur Num: 3
- Inform-Taxi-Phone-94549609159 Occur Num: 1
- Inform-Taxi-Phone-56629498460 Occur Num: 1
- Inform-Taxi-Phone-44947488614 Occur Num: 1
- Inform-Train-Time-51 minutes Occur Num: 1
- Inform-Taxi-Car-honda
- Occur Num: 13
- NLU Output
- Inform-Taxi-Leave-05:00 Occur Num: 3
- Inform-Taxi-Phone-52547806713 Occur Num: 1
- Inform-Taxi-Phone-42223260114 Occur Num: 1
- Inform-Taxi-Phone-84754151715 Occur Num: 1
- Inform-Taxi-Arrive-05:51 Occur Num: 1
- Inform-Taxi-Car-lexus
- Occur Num: 13
- NLU Output
- Inform-Taxi-Arrive-05:51 Occur Num: 2
- Inform-Hotel-Type-guesthouse Occur Num: 2
- Inform-Taxi-Phone-83144288521 Occur Num: 1
- Inform-Taxi-Phone-69714552379 Occur Num: 1
- Inform-Taxi-Phone-19626794559 Occur Num: 1
- Inform-Taxi-Car-bmw
- Occur Num: 13
- NLU Output
- Inform-Taxi-Leave-05:00 Occur Num: 3
- Inform-Taxi-Phone-79054617689 Occur Num: 1
- Inform-Taxi-Phone-73609361703 Occur Num: 1
- Inform-Taxi-Phone-49920327578 Occur Num: 1
- Inform-Hotel-Type-guesthouse Occur Num: 1
- Inform-Taxi-Car-tesla
- Occur Num: 13
- NLU Output
- Inform-Hotel-Type-guesthouse Occur Num: 3
- Inform-Taxi-Phone-00253869902 Occur Num: 1
- Inform-Taxi-Phone-97488259243 Occur Num: 1
- Inform-Taxi-Leave-05:00 Occur Num: 1
- Inform-Taxi-Phone-36951803842 Occur Num: 1
- Inform-Taxi-Car-volkswagen
- Occur Num: 7
- NLU Output
- Inform-Taxi-Phone-15214714504 Occur Num: 1
- Inform-Taxi-Arrive-05:51 Occur Num: 1
- Inform-Taxi-Phone-53827293134 Occur Num: 1
- Inform-Hotel-Type-guesthouse Occur Num: 1
- Inform-Taxi-Phone-29737767268 Occur Num: 1
- Inform-Taxi-Car-volvo
- Occur Num: 6
- NLU Output
- Inform-Taxi-Phone-26438028145 Occur Num: 1
- OfferBooked-Train-Time-51 Occur Num: 1
- Inform-Taxi-Phone-96368625434 Occur Num: 1
- Inform-Train-Leave-05:00 Occur Num: 1
- Inform-Taxi-Phone-10921657762 Occur Num: 1
Dialog Loop- Request-Taxi-Car-? Occur Num: 3
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-taxi-car type Occur Num: 3
Inform But Not Request Dialog ActNothing