Test Report
Model Name: BERTNLU-RuleDST-HDSA
Dataset: multiwoz
Time: 2020-04-26 07:28:10
Overall Results
Success Rate: 27.5 %
(Precision, Recall, F1) : (0.478, 0.572, 0.488)
Average Dialog Turn (Succ): 12.996
Average Dialog Turn (All): 31.536
Metric
| Total Num | Succ Rate | Precision | Recall | F1 | Dialog Loop Failed Rate | Dialog Turn (Succ) | Dialog Turn (All) |
---|
hotel | 327 | 0.333 | 0.362 | 0.521 | 0.408 | 0.599 | 7.321 | 20.856 |
attraction | 285 | 0.442 | 0.592 | 0.571 | 0.562 | 0.540 | 10.603 | 25.046 |
restaurant | 368 | 0.535 | 0.466 | 0.714 | 0.536 | 0.454 | 6.579 | 16.848 |
train | 314 | 0.459 | 0.617 | 0.596 | 0.593 | 0.525 | 11.472 | 22.656 |
police | 23 | 0.870 | 0.920 | 0.913 | 0.904 | 0.130 | 10.700 | 14.522 |
hospital | 30 | 0.867 | 0.867 | 0.867 | 0.867 | 0.133 | 8.000 | 12.267 |
taxi | 76 | 0.342 | 0.684 | 0.513 | 0.570 | 0.632 | 2.154 | 18.763 |
Domain hotel
Domain attraction
Overall Results
Success Rate: 44.2 %
(Precision, Recall, F1) : (0.592, 0.571, 0.562)
Average Dialog Turn (Succ): 10.603
Average Dialog Turn (All): 25.046
System NLU Failed Dialog Act:- Request-Attraction-Post-?
- Occur Num: 389
- NLU Output
- Request-Restaurant-Post-? Occur Num: 105
- Request-Attraction-Phone-? Occur Num: 47
- Request-Attraction-Addr-? Occur Num: 46
- Request-Restaurant-Addr-? Occur Num: 42
- Request-Attraction-Fee-? Occur Num: 40
- Request-Attraction-Addr-?
- Occur Num: 350
- NLU Output
- Request-Restaurant-Addr-? Occur Num: 89
- Request-Restaurant-Post-? Occur Num: 49
- Request-Attraction-Post-? Occur Num: 49
- Request-Attraction-Phone-? Occur Num: 42
- Request-Attraction-Fee-? Occur Num: 40
- Request-Attraction-Phone-?
- Occur Num: 347
- NLU Output
- Request-Restaurant-Phone-? Occur Num: 87
- Request-Attraction-Post-? Occur Num: 52
- Request-Attraction-Addr-? Occur Num: 49
- Request-Restaurant-Addr-? Occur Num: 40
- Request-Attraction-Fee-? Occur Num: 39
- Request-Attraction-Fee-?
- Occur Num: 120
- NLU Output
- Request-Attraction-Post-? Occur Num: 40
- Request-Attraction-Addr-? Occur Num: 39
- Request-Attraction-Phone-? Occur Num: 31
- Request-Attraction-Type-? Occur Num: 5
- Request-Restaurant-Post-? Occur Num: 3
- Request-Attraction-Area-?
- Occur Num: 53
- NLU Output
- Request-Restaurant-Area-? Occur Num: 14
- Request-Restaurant-Addr-? Occur Num: 9
- Request-Attraction-Phone-? Occur Num: 7
- Request-Restaurant-Phone-? Occur Num: 7
- Request-Attraction-Addr-? Occur Num: 4
- Request-Attraction-Type-?
- Occur Num: 31
- NLU Output
- Request-Attraction-Post-? Occur Num: 8
- Request-Attraction-Phone-? Occur Num: 8
- Request-Attraction-Addr-? Occur Num: 7
- Request-Attraction-Fee-? Occur Num: 6
- Request-Restaurant-Post-? Occur Num: 2
- Inform-Attraction-Area-centre
- Occur Num: 29
- NLU Output
- Inform-Attraction-Type-entertainment Occur Num: 14
- Inform-Attraction-Type-park Occur Num: 2
- Inform-Attraction-Type-museums Occur Num: 2
- Inform-Attraction-Type-boats Occur Num: 2
- Inform-Attraction-Type-museum Occur Num: 2
- Inform-Attraction-Type-museum
- Occur Num: 17
- NLU Output
- Inform-Attraction-Area-centre Occur Num: 7
- Inform-Attraction-Type-museums Occur Num: 5
- Inform-Attraction-Type-entertainment Occur Num: 3
- Inform-Attraction-Area-west Occur Num: 2
- Inform-Attraction-Type-boat
- Occur Num: 16
- NLU Output
- Inform-Attraction-Area-centre Occur Num: 6
- Inform-Attraction-Type-boats Occur Num: 5
- Inform-Attraction-Type-entertainment Occur Num: 3
- Inform-Attraction-Area-west Occur Num: 1
- Inform-Attraction-Area-north Occur Num: 1
- Inform-Attraction-Type-nightclub
- Occur Num: 15
- NLU Output
- Inform-Attraction-none-none Occur Num: 5
- Inform-Attraction-Area-west Occur Num: 4
- Inform-Attraction-Area-centre Occur Num: 4
- Inform-Attraction-Type-entertainment Occur Num: 2
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Attraction-Addr-? Occur Num: 63
- Request-Attraction-Post-? Occur Num: 55
- Request-Attraction-Fee-? Occur Num: 19
- Request-Attraction-Type-? Occur Num: 9
- Request-Attraction-Phone-? Occur Num: 6
- Request-Attraction-Area-? Occur Num: 2
Bad Inform Dialog Act- inform-attraction-area Occur Num: 31
- inform-attraction-phone Occur Num: 6
- inform-attraction-postcode Occur Num: 2
- recommend-attraction-area Occur Num: 1
Request But Not Inform Dialog Act- request-attraction-postcode Occur Num: 92
- request-attraction-address Occur Num: 85
- request-attraction-phone Occur Num: 45
- request-attraction-area Occur Num: 15
- request-attraction-type Occur Num: 13
- request-attraction-entrance fee Occur Num: 4
Inform But Not Request Dialog Act- inform-attraction-phone Occur Num: 36
- inform-attraction-area Occur Num: 6
- inform-attraction-type Occur Num: 5
Domain restaurant
Overall Results
Success Rate: 53.5 %
(Precision, Recall, F1) : (0.466, 0.714, 0.536)
Average Dialog Turn (Succ): 6.579
Average Dialog Turn (All): 16.848
System NLU Failed Dialog Act:- Request-Restaurant-Addr-?
- Occur Num: 190
- NLU Output
- Request-Restaurant-Phone-? Occur Num: 47
- Request-Restaurant-Post-? Occur Num: 39
- Request-Hospital-Addr-? Occur Num: 25
- Request-Attraction-Addr-? Occur Num: 25
- Request-Hotel-Addr-? Occur Num: 14
- Request-Restaurant-Phone-?
- Occur Num: 164
- NLU Output
- Request-Restaurant-Addr-? Occur Num: 47
- Request-Attraction-Phone-? Occur Num: 35
- Request-Restaurant-Post-? Occur Num: 22
- Request-Hotel-Phone-? Occur Num: 15
- Request-Hospital-Phone-? Occur Num: 13
- Request-Restaurant-Post-?
- Occur Num: 120
- NLU Output
- Request-Restaurant-Addr-? Occur Num: 36
- Request-Restaurant-Phone-? Occur Num: 20
- Request-Attraction-Post-? Occur Num: 17
- Request-Hospital-Post-? Occur Num: 16
- Request-Hotel-Post-? Occur Num: 9
- Inform-Restaurant-Price-expensive
- Occur Num: 26
- NLU Output
- Inform-Restaurant-Price-expensively Occur Num: 10
- Inform-Restaurant-Area-centre Occur Num: 4
- Inform-Restaurant-Area-south Occur Num: 2
- Inform-Restaurant-Food-indian Occur Num: 1
- Inform-Restaurant-Food-mexican Occur Num: 1
- Inform-Restaurant-Price-moderate
- Occur Num: 25
- NLU Output
- Inform-Restaurant-Price-moderately Occur Num: 11
- Inform-Restaurant-Area-centre Occur Num: 6
- Inform-Restaurant-Food-british Occur Num: 2
- Inform-Restaurant-Food-mediterranean Occur Num: 1
- Inform-Restaurant-Area-east Occur Num: 1
- Inform-Restaurant-Area-centre
- Occur Num: 16
- NLU Output
- Inform-Restaurant-Price-moderately Occur Num: 6
- Inform-Restaurant-Food-british Occur Num: 3
- Inform-Restaurant-Price-expensively Occur Num: 2
- Inform-Restaurant-Food-modern european Occur Num: 1
- Inform-Restaurant-Food-chinese Occur Num: 1
- Inform-Restaurant-Food-portuguese
- Occur Num: 11
- NLU Output
- Inform-Restaurant-Area-centre Occur Num: 3
- Inform-Restaurant-none-none Occur Num: 3
- Inform-Restaurant-Price-expensive Occur Num: 2
- Inform-Restaurant-Price-moderately Occur Num: 1
- Inform-Restaurant-Price-moderate Occur Num: 1
- Inform-Restaurant-Price-cheap
- Occur Num: 10
- NLU Output
- Inform-Restaurant-Price-cheaply Occur Num: 4
- Inform-Restaurant-Food-italian Occur Num: 3
- Inform-Restaurant-Area-east Occur Num: 1
- Inform-Restaurant-Area-north Occur Num: 1
- Inform-Restaurant-Area-centre Occur Num: 1
- Request-Restaurant-Area-?
- Occur Num: 8
- NLU Output
- Request-Attraction-Area-? Occur Num: 4
- Request-Hotel-Area-? Occur Num: 3
- Request-Restaurant-Addr-? Occur Num: 1
- Inform-Restaurant-Food-italian
- Occur Num: 7
- NLU Output
- Inform-Restaurant-Price-cheaply Occur Num: 3
- Inform-Restaurant-Area-east Occur Num: 1
- Inform-Restaurant-Area-north Occur Num: 1
- Inform-Restaurant-Price-moderately Occur Num: 1
- Inform-Restaurant-Area-west Occur Num: 1
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Restaurant-Phone-? Occur Num: 35
- Request-Restaurant-Post-? Occur Num: 15
- Request-Restaurant-Addr-? Occur Num: 13
- Inform-Restaurant-Day-friday Occur Num: 5
- Inform-Restaurant-Day-sunday Occur Num: 5
- Inform-Restaurant-Day-saturday Occur Num: 5
- Inform-Restaurant-Time-13:45 Occur Num: 4
- Inform-Restaurant-Time-18:30 Occur Num: 4
- Inform-Restaurant-People-2 Occur Num: 4
- Inform-Restaurant-Time-12:30 Occur Num: 3
Bad Inform Dialog Act- inform-restaurant-area Occur Num: 110
- inform-restaurant-pricerange Occur Num: 68
- inform-restaurant-postcode Occur Num: 8
- recommend-restaurant-area Occur Num: 4
- recommend-restaurant-pricerange Occur Num: 1
Request But Not Inform Dialog Act- request-restaurant-phone Occur Num: 33
- request-restaurant-address Occur Num: 29
- request-restaurant-postcode Occur Num: 17
- request-restaurant-area Occur Num: 6
- request-restaurant-pricerange Occur Num: 2
- request-restaurant-food Occur Num: 1
Inform But Not Request Dialog Act- inform-restaurant-phone Occur Num: 84
- inform-restaurant-food Occur Num: 45
- inform-restaurant-area Occur Num: 36
Domain train
Overall Results
Success Rate: 45.9 %
(Precision, Recall, F1) : (0.617, 0.596, 0.593)
Average Dialog Turn (Succ): 11.472
Average Dialog Turn (All): 22.656
System NLU Failed Dialog Act:- Inform-Train-Dest-cambridge
- Occur Num: 210
- NLU Output
- Inform-Train-Depart-cambridge Occur Num: 21
- Inform-Train-Leave-do n't care Occur Num: 19
- Inform-Train-Day-wednesday Occur Num: 9
- Inform-Train-Day-Saturday Occur Num: 8
- Inform-Train-Dest-london liverpool street Occur Num: 7
- Inform-Train-Leave-dontcare
- Occur Num: 144
- NLU Output
- Inform-Train-Leave-do n't care Occur Num: 50
- Inform-Train-Dest-cambridge Occur Num: 19
- Inform-Train-Depart-kings lynn Occur Num: 5
- Inform-Train-Depart-birmingham new street Occur Num: 5
- Inform-Train-Depart-peterborough Occur Num: 5
- Inform-Train-Depart-cambridge
- Occur Num: 135
- NLU Output
- Inform-Train-Dest-cambridge Occur Num: 25
- Inform-Train-Day-Saturday Occur Num: 7
- Inform-Train-Day-thursday Occur Num: 7
- Inform-Train-Day-friday Occur Num: 6
- Inform-Train-Day-monday Occur Num: 6
- Request-Train-Time-?
- Occur Num: 127
- NLU Output
- Inform-Train-none-none Occur Num: 57
- Request-Train-Id-? Occur Num: 33
- Request-Train-Ticket-? Occur Num: 30
- Request-Train-Arrive-? Occur Num: 6
- Request-Train-Leave-? Occur Num: 1
- Request-Train-Id-?
- Occur Num: 98
- NLU Output
- Inform-Train-none-none Occur Num: 41
- Request-Train-Time-? Occur Num: 33
- Request-Train-Ticket-? Occur Num: 20
- Request-Train-Arrive-? Occur Num: 3
- Request-Train-Leave-? Occur Num: 1
- Request-Train-Ticket-?
- Occur Num: 48
- NLU Output
- Request-Train-Time-? Occur Num: 28
- Request-Train-Id-? Occur Num: 16
- Request-Train-Leave-? Occur Num: 2
- Request-Train-Arrive-? Occur Num: 1
- Request-Hotel-Price-? Occur Num: 1
- Inform-Train-Depart-london liverpool street
- Occur Num: 41
- NLU Output
- Inform-Train-Dest-london liverpool street Occur Num: 11
- Inform-Train-Dest-cambridge Occur Num: 6
- Inform-Train-Leave-do n't care Occur Num: 4
- Inform-Train-Day-wednesday Occur Num: 3
- Inform-Train-Depart-cambridge Occur Num: 2
- Inform-Train-Day-wednesday
- Occur Num: 37
- NLU Output
- Inform-Train-Dest-cambridge Occur Num: 8
- Inform-Train-Depart-cambridge Occur Num: 3
- Inform-Train-Dest-london liverpool street Occur Num: 2
- Inform-Train-Depart-broxbourne Occur Num: 2
- Inform-Train-Day-Saturday Occur Num: 2
- Inform-Train-Day-tuesday
- Occur Num: 35
- NLU Output
- Inform-Train-Dest-cambridge Occur Num: 5
- Inform-Train-Day-Saturday Occur Num: 4
- Inform-Train-Depart-cambridge Occur Num: 3
- Inform-Train-Dest-norwich Occur Num: 2
- Inform-Train-Depart-london kings cross Occur Num: 2
- Inform-Train-Day-monday
- Occur Num: 34
- NLU Output
- Inform-Train-Depart-cambridge Occur Num: 5
- Inform-Train-Dest-cambridge Occur Num: 4
- Inform-Train-Dest-norwich Occur Num: 2
- Inform-Train-Day-Saturday Occur Num: 2
- Inform-Train-Leave-13:00 Occur Num: 1
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Train-Time-? Occur Num: 60
- Request-Train-Id-? Occur Num: 15
- Inform-Train-Dest-cambridge Occur Num: 9
- Inform-Train-Leave-dontcare Occur Num: 7
- Request-Train-Leave-? Occur Num: 7
- Inform-Train-People-1 Occur Num: 5
- Inform-Train-People-3 Occur Num: 4
- Inform-Train-Depart-peterborough Occur Num: 4
- Inform-Train-People-4 Occur Num: 3
- Inform-Train-Depart-birmingham new street Occur Num: 3
Bad Inform Dialog Act- inform-train-leaveAt Occur Num: 20
- inform-train-price Occur Num: 14
- offerbooked-train-price Occur Num: 9
- inform-train-duration Occur Num: 8
- inform-train-trainID Occur Num: 4
- inform-train-arriveBy Occur Num: 1
- offerbooked-train-trainID Occur Num: 1
Request But Not Inform Dialog Act- request-train-duration Occur Num: 90
- request-train-trainID Occur Num: 39
- request-train-arriveBy Occur Num: 9
- request-train-leaveAt Occur Num: 9
- request-train-price Occur Num: 1
Inform But Not Request Dialog ActNothing
Domain police
Overall Results
Success Rate: 87.0 %
(Precision, Recall, F1) : (0.920, 0.913, 0.904)
Average Dialog Turn (Succ): 10.700
Average Dialog Turn (All): 14.522
System NLU Failed Dialog Act:- Request-Police-Phone-?
- Occur Num: 34
- NLU Output
- Request-Restaurant-Phone-? Occur Num: 10
- Request-Police-Addr-? Occur Num: 7
- Request-Restaurant-Addr-? Occur Num: 4
- Request-Attraction-Phone-? Occur Num: 4
- Request-Hospital-Phone-? Occur Num: 3
- Request-Police-Addr-?
- Occur Num: 20
- NLU Output
- Request-Restaurant-Addr-? Occur Num: 6
- Request-Restaurant-Phone-? Occur Num: 5
- Request-Police-Phone-? Occur Num: 5
- Request-Attraction-Addr-? Occur Num: 2
- Request-Attraction-Phone-? Occur Num: 2
- Inform-Police-none-none
- Occur Num: 6
- NLU Output
- Inform-Police-Name-Parkside Police Station Occur Num: 6
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Police-Addr-? Occur Num: 3
Bad Inform Dialog Act- inform-police-postcode Occur Num: 2
Request But Not Inform Dialog Act- request-police-address Occur Num: 3
- request-police-phone Occur Num: 1
Inform But Not Request Dialog ActNothing
Domain hospital
Overall Results
Success Rate: 86.7 %
(Precision, Recall, F1) : (0.867, 0.867, 0.867)
Average Dialog Turn (Succ): 8.000
Average Dialog Turn (All): 12.267
System NLU Failed Dialog Act:- Request-Hospital-Phone-?
- Occur Num: 18
- NLU Output
- Request-Restaurant-Phone-? Occur Num: 9
- Request-Attraction-Phone-? Occur Num: 7
- Inform-Hospital-none-none Occur Num: 2
- Inform-Hospital-Department-oncology
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-plastic and vascular surgery plastics
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-plastic and vascular surgery Occur Num: 1
- Inform-Hospital-Department-surgery
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-intermediate dependancy area
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-intermediate dependancy Occur Num: 1
- Inform-Hospital-Department-children's oncology and haematology
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Hospital-Phone-? Occur Num: 4
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-hospital-phone Occur Num: 4
Inform But Not Request Dialog ActNothing
Domain taxi
Overall Results
Success Rate: 34.2 %
(Precision, Recall, F1) : (0.684, 0.513, 0.570)
Average Dialog Turn (Succ): 2.154
Average Dialog Turn (All): 18.763
System NLU Failed Dialog Act:- Request-Taxi-Phone-?
- Occur Num: 90
- NLU Output
- Request-Taxi-Car-? Occur Num: 33
- Request-Attraction-Phone-? Occur Num: 24
- Request-Restaurant-Phone-? Occur Num: 21
- Request-Hotel-Phone-? Occur Num: 7
- Request-Police-Phone-? Occur Num: 3
- Request-Taxi-Car-?
- Occur Num: 10
- NLU Output
- Request-Hotel-Phone-? Occur Num: 7
- Request-Hotel-Type-? Occur Num: 3
- Inform-Taxi-Depart-the lensfield hotel
- Occur Num: 8
- NLU Output
- Inform-Taxi-Depart-the Occur Num: 2
- Inform-Taxi-Depart-lensfield hotel Occur Num: 2
- Inform-Taxi-Dest-thanh binh Occur Num: 1
- Inform-Taxi-Leave-02:00 Occur Num: 1
- Inform-Taxi-Dest-williams art and antiques Occur Num: 1
- Inform-Taxi-Depart-the fitzwilliam museum
- Occur Num: 7
- NLU Output
- Inform-Taxi-Depart-fitzwilliam museum Occur Num: 2
- Inform-Taxi-Dest-golden house Occur Num: 1
- Inform-Taxi-Leave-13:45 Occur Num: 1
- Inform-Taxi-Arrive-13:45 Occur Num: 1
- Inform-Taxi-Depart-the Occur Num: 1
- Inform-Taxi-Dest-home from home
- Occur Num: 6
- NLU Output
- Inform-Taxi-Depart-home from home Occur Num: 2
- Inform-Taxi-Depart-arbury lodge guesthouse Occur Num: 1
- Inform-Taxi-Leave-02:45 Occur Num: 1
- Inform-Taxi-Depart-thanh binh Occur Num: 1
- Inform-Taxi-Leave-02:30 Occur Num: 1
- Inform-Taxi-Arrive-02:00
- Occur Num: 4
- NLU Output
- Inform-Taxi-Dest-thanh binh Occur Num: 1
- Inform-Taxi-Depart-the Occur Num: 1
- Inform-Taxi-Depart-lensfield hotel Occur Num: 1
- Inform-Taxi-Leave-02:00 Occur Num: 1
- Inform-Taxi-Depart-parkside police station
- Occur Num: 3
- NLU Output
- Inform-Taxi-Dest-lynne strover gallery Occur Num: 1
- Inform-Police-Name-parkside police station Occur Num: 1
- Inform-Taxi-Arrive-12:15 Occur Num: 1
- Inform-Taxi-Arrive-13:45
- Occur Num: 3
- NLU Output
- Inform-Taxi-Depart-the Occur Num: 1
- Inform-Taxi-Depart-fitzwilliam museum Occur Num: 1
- Inform-Taxi-Dest-la mimosa Occur Num: 1
- Inform-Taxi-Dest-la mimosa
- Occur Num: 3
- NLU Output
- Inform-Taxi-Arrive-13:45 Occur Num: 1
- Inform-Taxi-Depart-the Occur Num: 1
- Inform-Taxi-Depart-fitzwilliam museum Occur Num: 1
- Inform-Taxi-Dest-efes restaurant
- Occur Num: 3
- NLU Output
- Inform-Taxi-Depart-broxbourne train station Occur Num: 1
- Inform-Taxi-Dest-efes Occur Num: 1
- Inform-Taxi-Arrive-16:15 Occur Num: 1
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Taxi-Car-? Occur Num: 31
- Request-Taxi-Phone-? Occur Num: 17
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-taxi-phone Occur Num: 49
- request-taxi-car type Occur Num: 25
Inform But Not Request Dialog ActNothing