Test Report
Model Name: BERTNLU-RuleDST-MDRG
Dataset: multiwoz
Time: 2020-04-28 06:00:57
Overall Results
Success Rate: 25.2 %
(Precision, Recall, F1) : (0.466, 0.431, 0.420)
Average Dialog Turn (Succ): 13.635
Average Dialog Turn (All): 33.580
Metric
| Total Num | Succ Rate | Precision | Recall | F1 | Dialog Loop Failed Rate | Dialog Turn (Succ) | Dialog Turn (All) |
---|
hotel | 332 | 0.449 | 0.306 | 0.414 | 0.339 | 0.497 | 7.960 | 20.464 |
attraction | 285 | 0.512 | 0.623 | 0.603 | 0.592 | 0.467 | 11.082 | 21.551 |
restaurant | 378 | 0.706 | 0.505 | 0.652 | 0.552 | 0.288 | 6.412 | 15.106 |
train | 329 | 0.377 | 0.515 | 0.411 | 0.445 | 0.614 | 8.000 | 25.088 |
police | 23 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 | 0.000 | 40.000 |
hospital | 30 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 | 0.000 | 40.000 |
taxi | 74 | 0.000 | 0.000 | 0.000 | 0.000 | 0.946 | 0.000 | 30.973 |
Domain hotel
Domain attraction
Overall Results
Success Rate: 51.2 %
(Precision, Recall, F1) : (0.623, 0.603, 0.592)
Average Dialog Turn (Succ): 11.082
Average Dialog Turn (All): 21.551
System NLU Failed Dialog Act:- Request-Attraction-Addr-?
- Occur Num: 263
- NLU Output
- Request-Attraction-Phone-? Occur Num: 65
- Request-Attraction-Post-? Occur Num: 64
- Request-Attraction-Fee-? Occur Num: 45
- Request-Restaurant-Addr-? Occur Num: 33
- Request-Restaurant-Post-? Occur Num: 22
- Request-Attraction-Post-?
- Occur Num: 255
- NLU Output
- Request-Attraction-Addr-? Occur Num: 62
- Request-Attraction-Phone-? Occur Num: 58
- Request-Attraction-Fee-? Occur Num: 44
- Request-Restaurant-Post-? Occur Num: 40
- Request-Restaurant-Phone-? Occur Num: 17
- Request-Attraction-Phone-?
- Occur Num: 245
- NLU Output
- Request-Attraction-Addr-? Occur Num: 67
- Request-Attraction-Post-? Occur Num: 60
- Request-Attraction-Fee-? Occur Num: 38
- Request-Restaurant-Phone-? Occur Num: 35
- Request-Restaurant-Post-? Occur Num: 13
- Request-Attraction-Fee-?
- Occur Num: 136
- NLU Output
- Request-Attraction-Addr-? Occur Num: 45
- Request-Attraction-Post-? Occur Num: 44
- Request-Attraction-Phone-? Occur Num: 37
- Request-Attraction-Type-? Occur Num: 7
- Request-Attraction-Area-? Occur Num: 2
- Request-Attraction-Type-?
- Occur Num: 36
- NLU Output
- Request-Attraction-Phone-? Occur Num: 11
- Request-Attraction-Addr-? Occur Num: 9
- Request-Attraction-Post-? Occur Num: 8
- Request-Attraction-Fee-? Occur Num: 7
- Inform-Attraction-Type-dontcare Occur Num: 1
- Inform-Attraction-Area-centre
- Occur Num: 36
- NLU Output
- Inform-Attraction-Type-entertainment Occur Num: 14
- Inform-Attraction-Type-do n't care Occur Num: 4
- Inform-Attraction-Type-colleges Occur Num: 3
- Inform-Restaurant-Area-centre Occur Num: 2
- Inform-Attraction-Type-museums Occur Num: 2
- Inform-Attraction-Type-dontcare
- Occur Num: 18
- NLU Output
- Inform-Attraction-Type-do n't care Occur Num: 12
- Inform-Attraction-Area-centre Occur Num: 4
- Inform-Attraction-Type-entertainment Occur Num: 1
- Inform-Attraction-Area-south Occur Num: 1
- Inform-Attraction-Type-boat
- Occur Num: 16
- NLU Output
- Inform-Attraction-Area-centre Occur Num: 6
- Inform-Attraction-Type-boats Occur Num: 5
- Inform-Attraction-Type-entertainment Occur Num: 3
- Inform-Attraction-Area-west Occur Num: 1
- Inform-Attraction-Area-north Occur Num: 1
- Inform-Attraction-Type-nightclub
- Occur Num: 16
- NLU Output
- Inform-Attraction-none-none Occur Num: 5
- Inform-Attraction-Area-centre Occur Num: 5
- Inform-Attraction-Area-west Occur Num: 4
- Inform-Attraction-Type-entertainment Occur Num: 2
- Request-Attraction-Area-?
- Occur Num: 16
- NLU Output
- Request-Attraction-Post-? Occur Num: 3
- Request-Attraction-Fee-? Occur Num: 3
- Request-Attraction-Addr-? Occur Num: 3
- Request-Restaurant-Area-? Occur Num: 2
- Request-Restaurant-Post-? Occur Num: 2
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Attraction-Fee-? Occur Num: 74
- Request-Attraction-Post-? Occur Num: 24
- Request-Attraction-Addr-? Occur Num: 19
- Request-Attraction-Type-? Occur Num: 8
- Request-Attraction-Phone-? Occur Num: 5
- Request-Attraction-Area-? Occur Num: 1
- Inform-Attraction-Name-all saints church Occur Num: 1
- Inform-Attraction-Name-broughton house gallery Occur Num: 1
Bad Inform Dialog Act- inform-attraction-area Occur Num: 4
Request But Not Inform Dialog Act- request-attraction-entrance fee Occur Num: 80
- request-attraction-postcode Occur Num: 56
- request-attraction-phone Occur Num: 54
- request-attraction-address Occur Num: 53
- request-attraction-type Occur Num: 15
- request-attraction-area Occur Num: 3
Inform But Not Request Dialog Act- inform-attraction-area Occur Num: 60
Domain restaurant
Overall Results
Success Rate: 70.6 %
(Precision, Recall, F1) : (0.505, 0.652, 0.552)
Average Dialog Turn (Succ): 6.412
Average Dialog Turn (All): 15.106
System NLU Failed Dialog Act:- Request-Restaurant-Addr-?
- Occur Num: 110
- NLU Output
- Request-Restaurant-Phone-? Occur Num: 55
- Request-Restaurant-Post-? Occur Num: 44
- Request-Attraction-Addr-? Occur Num: 5
- Request-Restaurant-Price-? Occur Num: 3
- Request-Attraction-Post-? Occur Num: 2
- Request-Restaurant-Phone-?
- Occur Num: 97
- NLU Output
- Request-Restaurant-Addr-? Occur Num: 57
- Request-Restaurant-Post-? Occur Num: 26
- Request-Attraction-Phone-? Occur Num: 5
- Request-Hospital-Post-? Occur Num: 3
- Request-Restaurant-Price-? Occur Num: 3
- Request-Restaurant-Post-?
- Occur Num: 82
- NLU Output
- Request-Restaurant-Addr-? Occur Num: 44
- Request-Restaurant-Phone-? Occur Num: 25
- Request-Hospital-Post-? Occur Num: 6
- Request-Attraction-Post-? Occur Num: 4
- Request-Restaurant-Food-? Occur Num: 1
- Inform-Restaurant-Price-expensive
- Occur Num: 29
- NLU Output
- Inform-Restaurant-Price-expensively Occur Num: 11
- Inform-Restaurant-Area-centre Occur Num: 4
- Inform-Restaurant-Area-south Occur Num: 2
- Inform-Restaurant-Area-east Occur Num: 2
- Inform-Restaurant-Food-indian Occur Num: 1
- Inform-Restaurant-Price-moderate
- Occur Num: 23
- NLU Output
- Inform-Restaurant-Price-moderately Occur Num: 9
- Inform-Restaurant-Area-centre Occur Num: 7
- Inform-Restaurant-Food-british Occur Num: 2
- Inform-Restaurant-Food-mediterranean Occur Num: 1
- Inform-Restaurant-Food-italian Occur Num: 1
- Inform-Restaurant-Area-centre
- Occur Num: 17
- NLU Output
- Inform-Restaurant-Price-moderately Occur Num: 6
- Inform-Restaurant-Food-british Occur Num: 3
- Inform-Restaurant-Price-expensively Occur Num: 2
- Inform-Restaurant-Price-moderate Occur Num: 1
- Inform-Restaurant-Food-modern european Occur Num: 1
- Inform-Restaurant-Area-dontcare
- Occur Num: 12
- NLU Output
- Inform-Restaurant-Area-do n't care Occur Num: 12
- Inform-Restaurant-Food-portuguese
- Occur Num: 12
- NLU Output
- Inform-Restaurant-Area-centre Occur Num: 3
- Inform-Restaurant-Price-expensive Occur Num: 3
- Inform-Restaurant-Price-moderate Occur Num: 2
- Inform-Restaurant-none-none Occur Num: 2
- Inform-Restaurant-Area-north Occur Num: 1
- Request-Restaurant-Price-?
- Occur Num: 12
- NLU Output
- Request-Hotel-Price-? Occur Num: 3
- Inform-Restaurant-none-none Occur Num: 3
- Request-Restaurant-Phone-? Occur Num: 3
- Request-Restaurant-Addr-? Occur Num: 3
- Inform-Restaurant-Food-dontcare
- Occur Num: 10
- NLU Output
- Inform-Restaurant-Food-do n't care Occur Num: 10
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Restaurant-Phone-? Occur Num: 48
- Request-Restaurant-Addr-? Occur Num: 22
- Request-Restaurant-Post-? Occur Num: 15
- Request-Restaurant-Food-? Occur Num: 4
- Request-Restaurant-Price-? Occur Num: 3
- Inform-Restaurant-Day-sunday Occur Num: 2
- Inform-Restaurant-Time-12:30 Occur Num: 1
- Inform-Restaurant-Time-10:45 Occur Num: 1
- Inform-Restaurant-Day-tuesday Occur Num: 1
- Inform-Restaurant-Time-15:30 Occur Num: 1
Bad Inform Dialog Act- inform-restaurant-area Occur Num: 37
- inform-restaurant-pricerange Occur Num: 10
- recommend-restaurant-area Occur Num: 2
- inform-restaurant-postcode Occur Num: 1
Request But Not Inform Dialog Act- request-restaurant-phone Occur Num: 47
- request-restaurant-address Occur Num: 39
- request-restaurant-postcode Occur Num: 20
- request-restaurant-pricerange Occur Num: 7
- request-restaurant-food Occur Num: 6
- request-restaurant-area Occur Num: 1
Inform But Not Request Dialog Act- inform-restaurant-area Occur Num: 98
- inform-restaurant-food Occur Num: 23
- inform-restaurant-phone Occur Num: 4
Domain train
Overall Results
Success Rate: 37.7 %
(Precision, Recall, F1) : (0.515, 0.411, 0.445)
Average Dialog Turn (Succ): 8.000
Average Dialog Turn (All): 25.088
System NLU Failed Dialog Act:- Inform-Train-Dest-cambridge
- Occur Num: 155
- NLU Output
- Inform-Train-Depart-cambridge Occur Num: 29
- Inform-Train-Day-Saturday Occur Num: 9
- Inform-Train-Day-friday Occur Num: 8
- Inform-Train-Day-thursday Occur Num: 8
- Inform-Train-Day-wednesday Occur Num: 6
- Inform-Train-Depart-cambridge
- Occur Num: 127
- NLU Output
- Inform-Train-Dest-cambridge Occur Num: 24
- Inform-Train-Day-friday Occur Num: 9
- Inform-Train-Day-thursday Occur Num: 9
- Inform-Train-Day-Saturday Occur Num: 6
- Inform-Train-Day-tuesday Occur Num: 6
- Request-Train-Time-?
- Occur Num: 121
- NLU Output
- Inform-Train-none-none Occur Num: 59
- Request-Train-Ticket-? Occur Num: 35
- Request-Train-Id-? Occur Num: 16
- Request-Train-Arrive-? Occur Num: 8
- Request-Train-Leave-? Occur Num: 3
- Request-Train-Ticket-?
- Occur Num: 69
- NLU Output
- Request-Train-Time-? Occur Num: 35
- Request-Train-Id-? Occur Num: 11
- Request-Restaurant-Price-? Occur Num: 7
- Request-Train-Leave-? Occur Num: 7
- Request-Train-Arrive-? Occur Num: 5
- Request-Train-Id-?
- Occur Num: 59
- NLU Output
- Request-Train-Time-? Occur Num: 23
- Request-Train-Ticket-? Occur Num: 15
- Inform-Train-none-none Occur Num: 12
- Request-Train-Arrive-? Occur Num: 5
- Request-Train-Leave-? Occur Num: 4
- Request-Train-Leave-?
- Occur Num: 40
- NLU Output
- Inform-Train-none-none Occur Num: 23
- Request-Train-Time-? Occur Num: 7
- Request-Train-Ticket-? Occur Num: 7
- Request-Train-Id-? Occur Num: 3
- Inform-Train-Day-friday
- Occur Num: 37
- NLU Output
- Inform-Train-Depart-cambridge Occur Num: 6
- Inform-Train-Dest-cambridge Occur Num: 5
- Inform-Train-Day-Saturday Occur Num: 3
- Inform-Train-Arrive-11:15 Occur Num: 2
- Inform-Train-Depart-leicester Occur Num: 2
- Inform-Train-Day-thursday
- Occur Num: 34
- NLU Output
- Inform-Train-Depart-cambridge Occur Num: 7
- Inform-Train-Dest-cambridge Occur Num: 5
- Inform-Train-Leave-15:30 Occur Num: 2
- Inform-Train-Depart-kings lynn Occur Num: 1
- Inform-Train-Arrive-19:15 Occur Num: 1
- Inform-Train-Day-monday
- Occur Num: 29
- NLU Output
- Inform-Train-Dest-cambridge Occur Num: 4
- Inform-Train-Depart-cambridge Occur Num: 4
- Inform-Train-Dest-kings lynn Occur Num: 2
- Inform-Train-Day-Saturday Occur Num: 2
- Inform-Train-Arrive-13:30 Occur Num: 1
- Inform-Train-Day-tuesday
- Occur Num: 27
- NLU Output
- Inform-Train-Dest-cambridge Occur Num: 4
- Inform-Train-Depart-cambridge Occur Num: 3
- Inform-Train-Dest-norwich Occur Num: 2
- Inform-Train-Arrive-09:15 Occur Num: 2
- Inform-Train-Day-Saturday Occur Num: 2
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Train-Time-? Occur Num: 96
- Request-Train-Ticket-? Occur Num: 67
- Request-Train-Leave-? Occur Num: 25
- Request-Train-Arrive-? Occur Num: 14
Bad Inform Dialog Act- inform-train-duration Occur Num: 2
Request But Not Inform Dialog Act- request-train-price Occur Num: 104
- request-train-duration Occur Num: 85
- request-train-leaveAt Occur Num: 23
- request-train-arriveBy Occur Num: 19
Inform But Not Request Dialog ActNothing
Domain police
Overall Results
Success Rate: 0.0 %
(Precision, Recall, F1) : (0.000, 0.000, 0.000)
Average Dialog Turn (Succ): 0.000
Average Dialog Turn (All): 40.000
System NLU Failed Dialog Act:- Request-Police-Phone-?
- Occur Num: 49
- NLU Output
- Request-Restaurant-Phone-? Occur Num: 19
- Request-Police-Addr-? Occur Num: 14
- Request-Restaurant-Addr-? Occur Num: 8
- Inform-Police-none-none Occur Num: 5
- Request-Hospital-Phone-? Occur Num: 3
- Request-Police-Addr-?
- Occur Num: 36
- NLU Output
- Request-Restaurant-Phone-? Occur Num: 13
- Request-Police-Phone-? Occur Num: 12
- Request-Restaurant-Addr-? Occur Num: 11
- Inform-Police-none-none
- Occur Num: 6
- NLU Output
- Inform-Police-Name-Parkside Police Station Occur Num: 6
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Police-Addr-? Occur Num: 17
- Request-Police-Phone-? Occur Num: 6
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-police-phone Occur Num: 20
- request-police-address Occur Num: 17
Inform But Not Request Dialog ActNothing
Domain hospital
Overall Results
Success Rate: 0.0 %
(Precision, Recall, F1) : (0.000, 0.000, 0.000)
Average Dialog Turn (Succ): 0.000
Average Dialog Turn (All): 40.000
System NLU Failed Dialog Act:- Request-Hospital-Phone-?
- Occur Num: 28
- NLU Output
- Request-Restaurant-Phone-? Occur Num: 28
- Inform-Hospital-Department-oncology
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-plastic and vascular surgery plastics
- Occur Num: 1
- NLU Output
- Inform-Hospital-Department-plastic and vascular surgery Occur Num: 1
- Inform-Hospital-Department-surgery
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
- Inform-Hospital-Department-children's oncology and haematology
- Occur Num: 1
- NLU Output
- Inform-Hospital-none-none Occur Num: 1
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Hospital-Phone-? Occur Num: 30
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-hospital-phone Occur Num: 30
Inform But Not Request Dialog ActNothing
Domain taxi
Overall Results
Success Rate: 0.0 %
(Precision, Recall, F1) : (0.000, 0.000, 0.000)
Average Dialog Turn (Succ): 0.000
Average Dialog Turn (All): 30.973
System NLU Failed Dialog Act:- Request-Taxi-Phone-?
- Occur Num: 89
- NLU Output
- Request-Taxi-Car-? Occur Num: 71
- Request-Hotel-Type-? Occur Num: 14
- Request-Hotel-Phone-? Occur Num: 2
- Request-Attraction-Type-? Occur Num: 2
- Request-Taxi-Car-?
- Occur Num: 18
- NLU Output
- Request-Hotel-Type-? Occur Num: 14
- Request-Hotel-Phone-? Occur Num: 2
- Request-Attraction-Type-? Occur Num: 2
- Inform-Taxi-Depart-the lensfield hotel
- Occur Num: 8
- NLU Output
- Inform-Taxi-Depart-the Occur Num: 2
- Inform-Taxi-Depart-lensfield hotel Occur Num: 2
- Inform-Taxi-Dest-thanh binh Occur Num: 1
- Inform-Taxi-Leave-02:00 Occur Num: 1
- Inform-Taxi-Dest-williams art and antiques Occur Num: 1
- Inform-Taxi-Depart-the fitzwilliam museum
- Occur Num: 7
- NLU Output
- Inform-Taxi-Depart-fitzwilliam museum Occur Num: 2
- Inform-Taxi-Dest-golden house Occur Num: 1
- Inform-Taxi-Leave-13:45 Occur Num: 1
- Inform-Taxi-Arrive-13:45 Occur Num: 1
- Inform-Taxi-Depart-the Occur Num: 1
- Inform-Taxi-Dest-home from home
- Occur Num: 6
- NLU Output
- Inform-Taxi-Depart-home from home Occur Num: 2
- Inform-Taxi-Depart-arbury lodge guesthouse Occur Num: 1
- Inform-Taxi-Leave-02:45 Occur Num: 1
- Inform-Taxi-Depart-thanh binh Occur Num: 1
- Inform-Taxi-Leave-02:30 Occur Num: 1
- Inform-Taxi-Arrive-13:45
- Occur Num: 4
- NLU Output
- Inform-Taxi-Depart-the Occur Num: 1
- Inform-Taxi-Depart-fitzwilliam museum Occur Num: 1
- Inform-Taxi-Dest-la mimosa Occur Num: 1
- Inform-Train-Arrive-13:45 Occur Num: 1
- Inform-Taxi-Arrive-02:00
- Occur Num: 4
- NLU Output
- Inform-Taxi-Dest-thanh binh Occur Num: 1
- Inform-Taxi-Depart-the Occur Num: 1
- Inform-Taxi-Depart-lensfield hotel Occur Num: 1
- Inform-Taxi-Leave-02:00 Occur Num: 1
- Inform-Taxi-Depart-parkside police station
- Occur Num: 3
- NLU Output
- Inform-Taxi-Dest-lynne strover gallery Occur Num: 1
- Inform-Police-Name-parkside police station Occur Num: 1
- Inform-Taxi-Arrive-12:15 Occur Num: 1
- Inform-Taxi-Dest-la mimosa
- Occur Num: 3
- NLU Output
- Inform-Taxi-Arrive-13:45 Occur Num: 1
- Inform-Taxi-Depart-the Occur Num: 1
- Inform-Taxi-Depart-fitzwilliam museum Occur Num: 1
- Inform-Taxi-Dest-efes restaurant
- Occur Num: 3
- NLU Output
- Inform-Taxi-Depart-broxbourne train station Occur Num: 1
- Inform-Taxi-Dest-efes Occur Num: 1
- Inform-Taxi-Arrive-16:15 Occur Num: 1
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Taxi-Car-? Occur Num: 70
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-taxi-car type Occur Num: 74
- request-taxi-phone Occur Num: 74
Inform But Not Request Dialog ActNothing