Test Report
Model Name: BERTNLU-RuleDST-LaRL
Dataset: multiwoz
Time: 2020-04-27 19:16:35
Overall Results
Success Rate: 33.1 %
(Precision, Recall, F1) : (0.485, 0.560, 0.488)
Average Dialog Turn (Succ): 15.547
Average Dialog Turn (All): 28.734
Metric
| Total Num | Succ Rate | Precision | Recall | F1 | Dialog Loop Failed Rate | Dialog Turn (Succ) | Dialog Turn (All) |
---|
hotel | 343 | 0.364 | 0.282 | 0.471 | 0.326 | 0.458 | 10.096 | 16.711 |
attraction | 294 | 0.595 | 0.619 | 0.693 | 0.631 | 0.350 | 9.337 | 18.197 |
restaurant | 377 | 0.618 | 0.521 | 0.643 | 0.554 | 0.308 | 10.876 | 16.271 |
train | 328 | 0.634 | 0.746 | 0.676 | 0.696 | 0.354 | 9.490 | 18.470 |
police | 23 | 0.783 | 0.870 | 0.848 | 0.843 | 0.217 | 6.333 | 13.652 |
hospital | 30 | 0.900 | 0.900 | 0.900 | 0.900 | 0.100 | 3.852 | 7.467 |
taxi | 94 | 0.085 | 0.138 | 0.112 | 0.121 | 0.862 | 8.750 | 27.872 |
Domain hotel
Overall Results
Success Rate: 36.4 %
(Precision, Recall, F1) : (0.282, 0.471, 0.326)
Average Dialog Turn (Succ): 10.096
Average Dialog Turn (All): 16.711
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Hotel-Addr-? Occur Num: 24
- Request-Hotel-Post-? Occur Num: 22
- Request-Hotel-Phone-? Occur Num: 13
- Inform-Hotel-Stay-4 Occur Num: 10
- Inform-Hotel-Type-hotel Occur Num: 8
- Inform-Hotel-Stay-2 Occur Num: 7
- Request-Hotel-Area-? Occur Num: 6
- Inform-Hotel-Stay-5 Occur Num: 6
- Inform-Hotel-Stay-3 Occur Num: 5
- Inform-Hotel-Parking-yes Occur Num: 5
Bad Inform Dialog Act- inform-hotel-pricerange Occur Num: 144
- recommend-hotel-pricerange Occur Num: 55
- inform-hotel-parking Occur Num: 43
- inform-hotel-postcode Occur Num: 40
- inform-hotel-internet Occur Num: 29
- inform-hotel-stars Occur Num: 11
- recommend-hotel-stars Occur Num: 7
- recommend-hotel-internet Occur Num: 3
- recommend-hotel-parking Occur Num: 2
Request But Not Inform Dialog Act- request-hotel-postcode Occur Num: 43
- request-hotel-phone Occur Num: 24
- request-hotel-internet Occur Num: 21
- request-hotel-address Occur Num: 21
- request-hotel-parking Occur Num: 19
- request-hotel-pricerange Occur Num: 19
- request-hotel-area Occur Num: 8
- request-hotel-stars Occur Num: 7
- request-hotel-type Occur Num: 3
Inform But Not Request Dialog Act- inform-hotel-type Occur Num: 97
- inform-hotel-area Occur Num: 53
- inform-hotel-stars Occur Num: 16
- inform-hotel-phone Occur Num: 15
Domain attraction
Overall Results
Success Rate: 59.5 %
(Precision, Recall, F1) : (0.619, 0.693, 0.631)
Average Dialog Turn (Succ): 9.337
Average Dialog Turn (All): 18.197
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Attraction-Addr-? Occur Num: 32
- Request-Attraction-Fee-? Occur Num: 27
- Request-Attraction-Post-? Occur Num: 23
- Request-Attraction-Phone-? Occur Num: 14
- Request-Attraction-Type-? Occur Num: 4
- Request-Attraction-Area-? Occur Num: 2
- Inform-Attraction-Type-museum Occur Num: 1
Bad Inform Dialog Act- inform-attraction-postcode Occur Num: 89
- recommend-attraction-postcode Occur Num: 10
Request But Not Inform Dialog Act- request-attraction-postcode Occur Num: 68
- request-attraction-phone Occur Num: 55
- request-attraction-address Occur Num: 37
- request-attraction-entrance fee Occur Num: 27
- request-attraction-area Occur Num: 8
- request-attraction-type Occur Num: 5
Inform But Not Request Dialog Act- inform-attraction-phone Occur Num: 47
- inform-attraction-area Occur Num: 12
- inform-attraction-type Occur Num: 9
Domain restaurant
Overall Results
Success Rate: 61.8 %
(Precision, Recall, F1) : (0.521, 0.643, 0.554)
Average Dialog Turn (Succ): 10.876
Average Dialog Turn (All): 16.271
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Restaurant-Addr-? Occur Num: 23
- Request-Restaurant-Phone-? Occur Num: 15
- Request-Restaurant-Post-? Occur Num: 8
- Inform-Restaurant-Food-indian Occur Num: 6
- Inform-Restaurant-Food-chinese Occur Num: 4
- Inform-Restaurant-Day-friday Occur Num: 3
- Request-Restaurant-Area-? Occur Num: 3
- Inform-Restaurant-Food-international Occur Num: 3
- Inform-Restaurant-Food-european Occur Num: 3
- Inform-Restaurant-Food-british Occur Num: 3
Bad Inform Dialog Act- inform-restaurant-postcode Occur Num: 85
- recommend-restaurant-postcode Occur Num: 30
- inform-restaurant-pricerange Occur Num: 16
- inform-restaurant-area Occur Num: 15
Request But Not Inform Dialog Act- request-restaurant-address Occur Num: 52
- request-restaurant-postcode Occur Num: 50
- request-restaurant-phone Occur Num: 13
- request-restaurant-area Occur Num: 7
- request-restaurant-pricerange Occur Num: 5
Inform But Not Request Dialog Act- inform-restaurant-food Occur Num: 60
- inform-restaurant-phone Occur Num: 20
- inform-restaurant-area Occur Num: 16
Domain train
Overall Results
Success Rate: 63.4 %
(Precision, Recall, F1) : (0.746, 0.676, 0.696)
Average Dialog Turn (Succ): 9.490
Average Dialog Turn (All): 18.470
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Train-Ticket-? Occur Num: 75
- Request-Train-Time-? Occur Num: 28
- Request-Train-Leave-? Occur Num: 3
- Inform-Train-People-1 Occur Num: 2
- Inform-Train-People-7 Occur Num: 2
- Request-Train-Arrive-? Occur Num: 2
- Inform-Train-People-4 Occur Num: 1
- Request-Train-Id-? Occur Num: 1
- Inform-Train-Day-saturday Occur Num: 1
- Inform-Train-People-2 Occur Num: 1
Bad Inform Dialog Act- offerbooked-train-price Occur Num: 6
- inform-train-duration Occur Num: 1
- inform-train-price Occur Num: 1
- offerbook-train-trainID Occur Num: 1
Request But Not Inform Dialog Act- request-train-price Occur Num: 108
- request-train-duration Occur Num: 3
- request-train-leaveAt Occur Num: 3
- request-train-arriveBy Occur Num: 3
- request-train-trainID Occur Num: 2
Inform But Not Request Dialog ActNothing
Domain police
Overall Results
Success Rate: 78.3 %
(Precision, Recall, F1) : (0.870, 0.848, 0.843)
Average Dialog Turn (Succ): 6.333
Average Dialog Turn (All): 13.652
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Police-Addr-? Occur Num: 3
- Request-Police-Phone-? Occur Num: 2
Bad Inform Dialog Act- inform-police-postcode Occur Num: 3
Request But Not Inform Dialog Act- request-police-phone Occur Num: 3
- request-police-address Occur Num: 3
Inform But Not Request Dialog ActNothing
Domain hospital
Overall Results
Success Rate: 90.0 %
(Precision, Recall, F1) : (0.900, 0.900, 0.900)
Average Dialog Turn (Succ): 3.852
Average Dialog Turn (All): 7.467
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Hospital-Phone-? Occur Num: 3
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-hospital-phone Occur Num: 3
Inform But Not Request Dialog ActNothing
Domain taxi
Overall Results
Success Rate: 8.5 %
(Precision, Recall, F1) : (0.138, 0.112, 0.121)
Average Dialog Turn (Succ): 8.750
Average Dialog Turn (All): 27.872
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Taxi-Car-? Occur Num: 77
- Request-Taxi-Phone-? Occur Num: 4
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-taxi-phone Occur Num: 86
- request-taxi-car type Occur Num: 81
Inform But Not Request Dialog ActNothing