Test Report
Model Name: BERTNLU-RuleDST-HDSA
Dataset: multiwoz
Time: 2020-04-26 00:12:41
Overall Results
Success Rate: 34.0 %
(Precision, Recall, F1) : (0.478, 0.541, 0.476)
Average Dialog Turn (Succ): 15.000
Average Dialog Turn (All): 28.600
Metric
| Total Num | Succ Rate | Precision | Recall | F1 | Dialog Loop Failed Rate | Dialog Turn (Succ) | Dialog Turn (All) |
---|
hotel | 334 | 0.377 | 0.277 | 0.448 | 0.317 | 0.440 | 9.857 | 17.437 |
attraction | 285 | 0.614 | 0.623 | 0.719 | 0.642 | 0.337 | 8.651 | 17.684 |
restaurant | 377 | 0.584 | 0.482 | 0.559 | 0.502 | 0.347 | 9.618 | 17.416 |
train | 320 | 0.631 | 0.754 | 0.670 | 0.697 | 0.362 | 9.356 | 18.812 |
police | 23 | 0.870 | 0.913 | 0.913 | 0.901 | 0.130 | 9.700 | 13.652 |
hospital | 30 | 0.933 | 0.933 | 0.933 | 0.933 | 0.067 | 3.286 | 5.733 |
taxi | 88 | 0.136 | 0.182 | 0.159 | 0.167 | 0.830 | 8.167 | 26.500 |
Dialogue Loop

Domain hotel
Domain attraction
Overall Results
Success Rate: 61.4 %
(Precision, Recall, F1) : (0.623, 0.719, 0.642)
Average Dialog Turn (Succ): 8.651
Average Dialog Turn (All): 17.684
System NLU Failed Dialog Act:- Request-Attraction-Post-?
- Occur Num: 108
- NLU Output
- Request-Attraction-Phone-? Occur Num: 38
- Request-Attraction-Addr-? Occur Num: 31
- Request-Attraction-Fee-? Occur Num: 20
- Request-Attraction-Area-? Occur Num: 4
- Request-Restaurant-Post-? Occur Num: 4
- Request-Attraction-Phone-?
- Occur Num: 101
- NLU Output
- Request-Attraction-Post-? Occur Num: 38
- Request-Attraction-Addr-? Occur Num: 30
- Request-Attraction-Fee-? Occur Num: 14
- Request-Restaurant-Phone-? Occur Num: 6
- Request-Attraction-Type-? Occur Num: 4
- Request-Attraction-Addr-?
- Occur Num: 100
- NLU Output
- Request-Attraction-Post-? Occur Num: 34
- Request-Attraction-Phone-? Occur Num: 30
- Request-Attraction-Fee-? Occur Num: 16
- Request-Restaurant-Addr-? Occur Num: 7
- Request-Attraction-Area-? Occur Num: 5
- Request-Attraction-Fee-?
- Occur Num: 51
- NLU Output
- Request-Attraction-Post-? Occur Num: 20
- Request-Attraction-Addr-? Occur Num: 15
- Request-Attraction-Phone-? Occur Num: 14
- Request-Attraction-Type-? Occur Num: 1
- Request-Restaurant-Post-? Occur Num: 1
- Inform-Attraction-Area-centre
- Occur Num: 28
- NLU Output
- Inform-Attraction-Type-entertainment Occur Num: 12
- Inform-Attraction-Type-museums Occur Num: 3
- Inform-Attraction-Type-boats Occur Num: 2
- Inform-Attraction-Type-museum Occur Num: 2
- Inform-Attraction-Type-colleges Occur Num: 1
- Request-Attraction-Type-?
- Occur Num: 23
- NLU Output
- Request-Attraction-Post-? Occur Num: 8
- Request-Attraction-Phone-? Occur Num: 6
- Request-Attraction-Fee-? Occur Num: 5
- Inform-Attraction-none-none Occur Num: 2
- Request-Attraction-Addr-? Occur Num: 2
- Inform-Attraction-Type-museum
- Occur Num: 21
- NLU Output
- Inform-Attraction-Area-centre Occur Num: 9
- Inform-Attraction-Type-museums Occur Num: 7
- Inform-Attraction-Type-entertainment Occur Num: 3
- Inform-Attraction-Area-west Occur Num: 2
- Inform-Attraction-Type-nightclub
- Occur Num: 18
- NLU Output
- Inform-Attraction-Area-centre Occur Num: 6
- Inform-Attraction-none-none Occur Num: 5
- Inform-Attraction-Area-west Occur Num: 4
- Inform-Attraction-Type-entertainment Occur Num: 3
- Request-Attraction-Area-?
- Occur Num: 17
- NLU Output
- Request-Attraction-Addr-? Occur Num: 4
- Request-Attraction-Post-? Occur Num: 4
- Request-Attraction-Phone-? Occur Num: 3
- Request-Hotel-Area-? Occur Num: 3
- Request-Restaurant-Area-? Occur Num: 1
- Inform-Attraction-Type-boat
- Occur Num: 16
- NLU Output
- Inform-Attraction-Area-centre Occur Num: 6
- Inform-Attraction-Type-boats Occur Num: 5
- Inform-Attraction-Type-entertainment Occur Num: 3
- Inform-Attraction-Area-west Occur Num: 1
- Inform-Attraction-Area-north Occur Num: 1
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Attraction-Fee-? Occur Num: 32
- Request-Attraction-Addr-? Occur Num: 26
- Request-Attraction-Post-? Occur Num: 20
- Request-Attraction-Phone-? Occur Num: 11
- Request-Attraction-Type-? Occur Num: 5
- Request-Attraction-Area-? Occur Num: 1
- Inform-Attraction-Type-museum Occur Num: 1
Bad Inform Dialog Act- inform-attraction-postcode Occur Num: 100
- recommend-attraction-postcode Occur Num: 14
Request But Not Inform Dialog Act- request-attraction-postcode Occur Num: 62
- request-attraction-phone Occur Num: 50
- request-attraction-address Occur Num: 32
- request-attraction-entrance fee Occur Num: 25
- request-attraction-area Occur Num: 9
- request-attraction-type Occur Num: 6
Inform But Not Request Dialog Act- inform-attraction-phone Occur Num: 50
- inform-attraction-area Occur Num: 11
- inform-attraction-type Occur Num: 10
Domain restaurant
Overall Results
Success Rate: 58.4 %
(Precision, Recall, F1) : (0.482, 0.559, 0.502)
Average Dialog Turn (Succ): 9.618
Average Dialog Turn (All): 17.416
System NLU Failed Dialog Act:- Request-Restaurant-Phone-?
- Occur Num: 126
- NLU Output
- Request-Restaurant-Addr-? Occur Num: 41
- Request-Attraction-Phone-? Occur Num: 32
- Request-Restaurant-Post-? Occur Num: 25
- Request-Attraction-Addr-? Occur Num: 10
- Request-Attraction-Post-? Occur Num: 5
- Request-Restaurant-Addr-?
- Occur Num: 125
- NLU Output
- Request-Restaurant-Phone-? Occur Num: 39
- Request-Restaurant-Post-? Occur Num: 35
- Request-Attraction-Addr-? Occur Num: 24
- Request-Attraction-Phone-? Occur Num: 11
- Request-Attraction-Post-? Occur Num: 6
- Request-Restaurant-Post-?
- Occur Num: 86
- NLU Output
- Request-Restaurant-Addr-? Occur Num: 35
- Request-Restaurant-Phone-? Occur Num: 23
- Request-Attraction-Post-? Occur Num: 13
- Request-Attraction-Addr-? Occur Num: 6
- Request-Attraction-Phone-? Occur Num: 5
- Inform-Restaurant-Price-expensive
- Occur Num: 28
- NLU Output
- Inform-Restaurant-Price-expensively Occur Num: 11
- Inform-Restaurant-Area-centre Occur Num: 4
- Inform-Restaurant-Area-south Occur Num: 2
- Inform-Restaurant-Area-do n't care Occur Num: 2
- Inform-Restaurant-Food-indian Occur Num: 1
- Inform-Restaurant-Price-moderate
- Occur Num: 22
- NLU Output
- Inform-Restaurant-Price-moderately Occur Num: 9
- Inform-Restaurant-Area-centre Occur Num: 5
- Inform-Restaurant-Food-italian Occur Num: 2
- Inform-Restaurant-Food-british Occur Num: 2
- Inform-Restaurant-Food-mediterranean Occur Num: 1
- Inform-Restaurant-Area-centre
- Occur Num: 19
- NLU Output
- Inform-Restaurant-Price-moderately Occur Num: 5
- Inform-Restaurant-Food-british Occur Num: 3
- Inform-Restaurant-Food-modern european Occur Num: 2
- Inform-Restaurant-Price-expensively Occur Num: 2
- Inform-Restaurant-Food-indian Occur Num: 1
- Inform-Restaurant-Price-cheap
- Occur Num: 17
- NLU Output
- Inform-Restaurant-Price-cheaply Occur Num: 7
- Inform-Restaurant-Food-italian Occur Num: 3
- Inform-Restaurant-Area-centre Occur Num: 2
- Inform-Restaurant-Food-european Occur Num: 1
- Inform-Restaurant-Area-east Occur Num: 1
- Request-Restaurant-Area-?
- Occur Num: 14
- NLU Output
- Request-Attraction-Area-? Occur Num: 5
- Request-Hotel-Area-? Occur Num: 4
- Request-Restaurant-Addr-? Occur Num: 3
- Request-Restaurant-Post-? Occur Num: 1
- Request-Attraction-Post-? Occur Num: 1
- Inform-Restaurant-Area-dontcare
- Occur Num: 10
- NLU Output
- Inform-Restaurant-Area-do n't care Occur Num: 7
- Inform-Restaurant-Price-expensive Occur Num: 2
- Inform-Restaurant-Price-cheap Occur Num: 1
- Inform-Restaurant-Food-portuguese
- Occur Num: 9
- NLU Output
- Inform-Restaurant-Price-expensive Occur Num: 3
- Inform-Restaurant-none-none Occur Num: 2
- Inform-Restaurant-Area-centre Occur Num: 2
- Inform-Restaurant-Price-moderate Occur Num: 1
- Inform-Restaurant-Price-cheap Occur Num: 1
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Restaurant-Addr-? Occur Num: 44
- Request-Restaurant-Phone-? Occur Num: 22
- Request-Restaurant-Post-? Occur Num: 8
- Inform-Restaurant-Food-italian Occur Num: 4
- Inform-Restaurant-Food-indian Occur Num: 4
- Inform-Restaurant-Food-chinese Occur Num: 4
- Inform-Restaurant-Food-modern european Occur Num: 3
- Inform-Restaurant-Food-international Occur Num: 3
- Request-Restaurant-Area-? Occur Num: 3
- Inform-Restaurant-Time-17:00 Occur Num: 2
Bad Inform Dialog Act- inform-restaurant-postcode Occur Num: 93
- recommend-restaurant-postcode Occur Num: 30
- inform-restaurant-pricerange Occur Num: 16
- inform-restaurant-area Occur Num: 13
- recommend-restaurant-pricerange Occur Num: 2
Request But Not Inform Dialog Act- request-restaurant-address Occur Num: 73
- request-restaurant-postcode Occur Num: 56
- request-restaurant-phone Occur Num: 23
- request-restaurant-area Occur Num: 7
- request-restaurant-pricerange Occur Num: 5
Inform But Not Request Dialog Act- inform-restaurant-food Occur Num: 62
- inform-restaurant-phone Occur Num: 17
- inform-restaurant-area Occur Num: 11
Domain train
Overall Results
Success Rate: 63.1 %
(Precision, Recall, F1) : (0.754, 0.670, 0.697)
Average Dialog Turn (Succ): 9.356
Average Dialog Turn (All): 18.812
System NLU Failed Dialog Act:- Inform-Train-Dest-cambridge
- Occur Num: 136
- NLU Output
- Inform-Train-Depart-cambridge Occur Num: 21
- Inform-Train-Day-friday Occur Num: 8
- Inform-Train-Day-Saturday Occur Num: 7
- Inform-Train-Day-wednesday Occur Num: 7
- Inform-Train-Day-monday Occur Num: 7
- Inform-Train-Depart-cambridge
- Occur Num: 117
- NLU Output
- Inform-Train-Dest-cambridge Occur Num: 20
- Inform-Train-Day-friday Occur Num: 7
- Inform-Train-Day-wednesday Occur Num: 7
- Inform-Train-Day-monday Occur Num: 6
- Inform-Train-Day-Saturday Occur Num: 5
- Request-Train-Ticket-?
- Occur Num: 60
- NLU Output
- Request-Train-Time-? Occur Num: 33
- Request-Restaurant-Price-? Occur Num: 12
- Request-Hotel-Price-? Occur Num: 9
- Request-Train-Id-? Occur Num: 3
- Request-Train-Leave-? Occur Num: 2
- Request-Train-Time-?
- Occur Num: 50
- NLU Output
- Request-Train-Ticket-? Occur Num: 33
- Inform-Train-none-none Occur Num: 8
- Request-Train-Id-? Occur Num: 7
- Request-Train-Arrive-? Occur Num: 2
- Inform-Train-Day-wednesday
- Occur Num: 37
- NLU Output
- Inform-Train-Depart-cambridge Occur Num: 6
- Inform-Train-Dest-cambridge Occur Num: 5
- Inform-Train-Depart-broxbourne Occur Num: 2
- Inform-Train-Day-Saturday Occur Num: 2
- Inform-Train-Depart-london liverpool street Occur Num: 2
- Inform-Train-Day-monday
- Occur Num: 36
- NLU Output
- Inform-Train-Dest-cambridge Occur Num: 7
- Inform-Train-Dest-norwich Occur Num: 3
- Inform-Train-Depart-cambridge Occur Num: 3
- Inform-Train-Dest-kings lynn Occur Num: 2
- Inform-Train-Dest-leicester Occur Num: 2
- Inform-Train-Day-friday
- Occur Num: 29
- NLU Output
- Inform-Train-Depart-cambridge Occur Num: 5
- Inform-Train-Dest-cambridge Occur Num: 5
- Inform-Train-Day-Saturday Occur Num: 3
- Inform-Train-Arrive-11:15 Occur Num: 2
- Inform-Restaurant-Day-friday Occur Num: 1
- Inform-Train-Day-thursday
- Occur Num: 24
- NLU Output
- Inform-Train-Depart-cambridge Occur Num: 5
- Inform-Train-Dest-cambridge Occur Num: 3
- Inform-Train-Dest-stevenage Occur Num: 2
- Inform-Train-Leave-15:30 Occur Num: 2
- Inform-Train-Depart-kings lynn Occur Num: 1
- Inform-Train-Day-tuesday
- Occur Num: 23
- NLU Output
- Inform-Train-Depart-cambridge Occur Num: 3
- Inform-Train-Day-Saturday Occur Num: 2
- Inform-Train-Dest-cambridge Occur Num: 2
- Inform-Train-Leave-16:15 Occur Num: 1
- Inform-Train-Arrive-09:15 Occur Num: 1
- Inform-Train-Depart-norwich
- Occur Num: 19
- NLU Output
- Inform-Train-Dest-norwich Occur Num: 4
- Inform-Train-Dest-cambridge Occur Num: 3
- Inform-Train-Day-monday Occur Num: 2
- Inform-Train-Day-Saturday Occur Num: 2
- Inform-Train-Arrive-08:00 Occur Num: 1
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Train-Ticket-? Occur Num: 75
- Request-Train-Time-? Occur Num: 31
- Inform-Train-People-1 Occur Num: 2
- Request-Train-Leave-? Occur Num: 2
- Inform-Train-People-7 Occur Num: 2
- Request-Train-Arrive-? Occur Num: 2
- Inform-Train-People-4 Occur Num: 1
- Request-Train-Id-? Occur Num: 1
Bad Inform Dialog Act- inform-train-price Occur Num: 2
- offerbooked-train-price Occur Num: 1
- inform-train-duration Occur Num: 1
- offerbook-train-leaveAt Occur Num: 1
Request But Not Inform Dialog Act- request-train-price Occur Num: 108
- request-train-trainID Occur Num: 3
- request-train-arriveBy Occur Num: 3
- request-train-leaveAt Occur Num: 2
- request-train-duration Occur Num: 1
Inform But Not Request Dialog ActNothing
Domain police
Overall Results
Success Rate: 87.0 %
(Precision, Recall, F1) : (0.913, 0.913, 0.901)
Average Dialog Turn (Succ): 9.700
Average Dialog Turn (All): 13.652
System NLU Failed Dialog Act:- Request-Police-Phone-?
- Occur Num: 30
- NLU Output
- Request-Police-Addr-? Occur Num: 7
- Request-Restaurant-Phone-? Occur Num: 5
- Request-Attraction-Phone-? Occur Num: 4
- Inform-Police-none-none Occur Num: 4
- Request-Attraction-Addr-? Occur Num: 2
- Request-Police-Addr-?
- Occur Num: 19
- NLU Output
- Request-Police-Phone-? Occur Num: 6
- Request-Restaurant-Addr-? Occur Num: 4
- Request-Attraction-Addr-? Occur Num: 2
- Request-Attraction-Phone-? Occur Num: 2
- Request-Restaurant-Phone-? Occur Num: 2
- Inform-Police-none-none
- Occur Num: 6
- NLU Output
- Inform-Police-Name-Parkside Police Station Occur Num: 6
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Police-Addr-? Occur Num: 3
Bad Inform Dialog Act- inform-police-postcode Occur Num: 3
Request But Not Inform Dialog Act- request-police-address Occur Num: 3
- request-police-phone Occur Num: 1
Inform But Not Request Dialog ActNothing
Domain hospital
Overall Results
Success Rate: 93.3 %
(Precision, Recall, F1) : (0.933, 0.933, 0.933)
Average Dialog Turn (Succ): 3.286
Average Dialog Turn (All): 5.733
System NLU Failed Dialog Act:- Request-Hospital-Phone-?
- Occur Num: 5
- NLU Output
- Request-Attraction-Phone-? Occur Num: 2
- Request-Hotel-Phone-? Occur Num: 1
- Request-Police-Phone-? Occur Num: 1
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-oncology
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-plastic and vascular surgery plastics
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-plastic and vascular surgery Occur Num: 1
- Inform-Hospital-Department-surgery
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-children's oncology and haematology
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Hospital-Phone-? Occur Num: 2
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-hospital-phone Occur Num: 2
Inform But Not Request Dialog ActNothing
Domain taxi
Overall Results
Success Rate: 13.6 %
(Precision, Recall, F1) : (0.182, 0.159, 0.167)
Average Dialog Turn (Succ): 8.167
Average Dialog Turn (All): 26.500
System NLU Failed Dialog Act:- Request-Taxi-Phone-?
- Occur Num: 86
- NLU Output
- Request-Taxi-Car-? Occur Num: 69
- Request-Hotel-Phone-? Occur Num: 6
- Request-Police-Phone-? Occur Num: 3
- Request-Restaurant-Phone-? Occur Num: 3
- Request-Hospital-Phone-? Occur Num: 2
- Inform-Taxi-Depart-the lensfield hotel
- Occur Num: 8
- NLU Output
- Inform-Taxi-Depart-the Occur Num: 2
- Inform-Taxi-Depart-lensfield hotel Occur Num: 2
- Inform-Taxi-Dest-thanh binh Occur Num: 1
- Inform-Taxi-Leave-02:00 Occur Num: 1
- Inform-Taxi-Dest-williams art and antiques Occur Num: 1
- Inform-Taxi-Depart-the fitzwilliam museum
- Occur Num: 7
- NLU Output
- Inform-Taxi-Depart-fitzwilliam museum Occur Num: 2
- Inform-Taxi-Dest-golden house Occur Num: 1
- Inform-Taxi-Leave-13:45 Occur Num: 1
- Inform-Taxi-Arrive-13:45 Occur Num: 1
- Inform-Taxi-Depart-the Occur Num: 1
- Request-Taxi-Car-?
- Occur Num: 7
- NLU Output
- Request-Hotel-Phone-? Occur Num: 6
- Request-Hotel-Type-? Occur Num: 1
- Inform-Taxi-Dest-home from home
- Occur Num: 6
- NLU Output
- Inform-Taxi-Depart-home from home Occur Num: 2
- Inform-Taxi-Depart-arbury lodge guesthouse Occur Num: 1
- Inform-Taxi-Leave-02:45 Occur Num: 1
- Inform-Taxi-Depart-thanh binh Occur Num: 1
- Inform-Taxi-Leave-02:30 Occur Num: 1
- Inform-Taxi-Leave-dontcare
- Occur Num: 5
- NLU Output
- Inform-Taxi-Leave-do Occur Num: 3
- Inform-Taxi-Leave-do n't care Occur Num: 1
- Inform-Taxi-Depart-rajmahal Occur Num: 1
- Inform-Taxi-Arrive-19:30
- Occur Num: 4
- NLU Output
- Inform-Train-Arrive-19:30 Occur Num: 2
- Inform-Taxi-Depart-warkworth house Occur Num: 1
- Inform-Taxi-Dest-the gardenia Occur Num: 1
- Inform-Taxi-Dest-great saint mary's church
- Occur Num: 4
- NLU Output
- Inform-Taxi-Depart-broughton house gallery Occur Num: 1
- Inform-Taxi-Arrive-01:15 Occur Num: 1
- Inform-Taxi-none-none Occur Num: 1
- Inform-Taxi-Leave-01:15 Occur Num: 1
- Inform-Taxi-Arrive-13:45
- Occur Num: 4
- NLU Output
- Inform-Taxi-Depart-the Occur Num: 1
- Inform-Taxi-Depart-fitzwilliam museum Occur Num: 1
- Inform-Taxi-Dest-la mimosa Occur Num: 1
- Inform-Train-Arrive-13:45 Occur Num: 1
- Inform-Taxi-Depart-the good luck chinese food takeaway
- Occur Num: 4
- NLU Output
- Inform-Taxi-Dest-nandos Occur Num: 1
- Inform-Taxi-Leave-05:15 Occur Num: 1
- Inform-Taxi-Depart-good luck Occur Num: 1
- Inform-Taxi-Depart-good luck chinese food Occur Num: 1
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Taxi-Car-? Occur Num: 68
- Request-Taxi-Phone-? Occur Num: 4
- Inform-Taxi-Depart-London Occur Num: 1
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-taxi-phone Occur Num: 76
- request-taxi-car type Occur Num: 72
Inform But Not Request Dialog ActNothing