Test Report
Model Name: Sequicity
Dataset: multiwoz
Time: 2020-04-26 00:56:51
Overall Results
Success Rate: 10.5 %
(Precision, Recall, F1) : (0.414, 0.308, 0.313)
Average Dialog Turn (Succ): 12.876
Average Dialog Turn (All): 38.320
Metric
| Total Num | Succ Rate | Precision | Recall | F1 | Dialog Loop Failed Rate | Dialog Turn (Succ) | Dialog Turn (All) |
---|
hotel | 244 | 0.164 | 0.272 | 0.328 | 0.275 | 0.803 | 10.550 | 31.336 |
restaurant | 332 | 0.265 | 0.494 | 0.464 | 0.456 | 0.720 | 6.409 | 29.229 |
train | 251 | 0.191 | 0.345 | 0.272 | 0.295 | 0.789 | 15.500 | 32.430 |
attraction | 234 | 0.162 | 0.635 | 0.430 | 0.488 | 0.833 | 5.842 | 33.103 |
police | 23 | 0.087 | 0.087 | 0.087 | 0.087 | 0.913 | 16.000 | 37.913 |
hospital | 30 | 0.867 | 0.867 | 0.867 | 0.867 | 0.133 | 7.692 | 12.000 |
taxi | 44 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 | 0.000 | 40.000 |
Domain hotel
Overall Results
Success Rate: 16.4 %
(Precision, Recall, F1) : (0.272, 0.328, 0.275)
Average Dialog Turn (Succ): 10.550
Average Dialog Turn (All): 31.336
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Hotel-Addr-? Occur Num: 75
- Inform-Hotel-Stay-4 Occur Num: 14
- Request-Hotel-Post-? Occur Num: 12
- Inform-Hotel-Stay-2 Occur Num: 10
- Inform-Hotel-Stay-3 Occur Num: 8
- Request-Hotel-Stars-? Occur Num: 8
- Inform-Hotel-Stay-5 Occur Num: 6
- Request-Hotel-Price-? Occur Num: 6
- Inform-Hotel-Type-guesthouse Occur Num: 5
- Request-Hotel-Phone-? Occur Num: 5
Bad Inform Dialog Act- inform-hotel-pricerange Occur Num: 34
- inform-hotel-area Occur Num: 19
- inform-hotel-postcode Occur Num: 7
- inform-hotel-internet Occur Num: 6
- inform-hotel-parking Occur Num: 5
- recommend-hotel-area Occur Num: 1
Request But Not Inform Dialog Act- request-hotel-address Occur Num: 92
- request-hotel-internet Occur Num: 11
- request-hotel-stars Occur Num: 11
- request-hotel-postcode Occur Num: 11
- request-hotel-pricerange Occur Num: 11
- request-hotel-parking Occur Num: 9
- request-hotel-area Occur Num: 8
- request-hotel-phone Occur Num: 5
- request-hotel-type Occur Num: 3
Inform But Not Request Dialog Act- inform-hotel-type Occur Num: 86
- inform-hotel-area Occur Num: 43
- inform-hotel-phone Occur Num: 15
- inform-hotel-stars Occur Num: 5
Domain restaurant
Overall Results
Success Rate: 26.5 %
(Precision, Recall, F1) : (0.494, 0.464, 0.456)
Average Dialog Turn (Succ): 6.409
Average Dialog Turn (All): 29.229
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Restaurant-Addr-? Occur Num: 129
- Request-Restaurant-Phone-? Occur Num: 38
- Request-Restaurant-Post-? Occur Num: 8
- Inform-Restaurant-Day-sunday Occur Num: 4
- Request-Restaurant-Price-? Occur Num: 4
- Inform-Restaurant-Time-16:45 Occur Num: 3
- Inform-Restaurant-Day-saturday Occur Num: 3
- Inform-Restaurant-Time-18:30 Occur Num: 3
- Inform-Restaurant-People-1 Occur Num: 2
- Inform-Restaurant-Day-wednesday Occur Num: 2
Bad Inform Dialog Act- inform-restaurant-area Occur Num: 33
- inform-restaurant-pricerange Occur Num: 19
- inform-restaurant-postcode Occur Num: 3
- recommend-restaurant-pricerange Occur Num: 1
Request But Not Inform Dialog Act- request-restaurant-address Occur Num: 166
- request-restaurant-phone Occur Num: 11
- request-restaurant-pricerange Occur Num: 6
- request-restaurant-postcode Occur Num: 5
- request-restaurant-area Occur Num: 2
Inform But Not Request Dialog Act- inform-restaurant-area Occur Num: 54
- inform-restaurant-phone Occur Num: 11
- inform-restaurant-food Occur Num: 11
Domain train
Overall Results
Success Rate: 19.1 %
(Precision, Recall, F1) : (0.345, 0.272, 0.295)
Average Dialog Turn (Succ): 15.500
Average Dialog Turn (All): 32.430
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Train-Time-? Occur Num: 62
- Request-Train-Id-? Occur Num: 52
- Request-Train-Ticket-? Occur Num: 42
- Request-Train-Leave-? Occur Num: 13
- Request-Train-Arrive-? Occur Num: 9
- Inform-Train-People-6 Occur Num: 3
- Inform-Train-People-5 Occur Num: 3
- Inform-Train-People-3 Occur Num: 3
- Inform-Train-People-1 Occur Num: 3
- Inform-Train-People-8 Occur Num: 2
Bad Inform Dialog Act- inform-train-trainID Occur Num: 2
- offerbooked-train-price Occur Num: 2
- inform-train-day Occur Num: 2
- inform-train-arriveBy Occur Num: 1
- inform-train-leaveAt Occur Num: 1
Request But Not Inform Dialog Act- request-train-trainID Occur Num: 100
- request-train-price Occur Num: 57
- request-train-duration Occur Num: 46
- request-train-arriveBy Occur Num: 20
- request-train-leaveAt Occur Num: 15
Inform But Not Request Dialog ActNothing
Domain attraction
Overall Results
Success Rate: 16.2 %
(Precision, Recall, F1) : (0.635, 0.430, 0.488)
Average Dialog Turn (Succ): 5.842
Average Dialog Turn (All): 33.103
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Attraction-Fee-? Occur Num: 109
- Request-Attraction-Addr-? Occur Num: 71
- Request-Attraction-Post-? Occur Num: 6
- Request-Attraction-Type-? Occur Num: 6
- Request-Attraction-Phone-? Occur Num: 2
- Request-Attraction-Area-? Occur Num: 1
Bad Inform Dialog Act- inform-attraction-area Occur Num: 13
Request But Not Inform Dialog Act- request-attraction-address Occur Num: 132
- request-attraction-entrance fee Occur Num: 109
- request-attraction-phone Occur Num: 17
- request-attraction-postcode Occur Num: 16
- request-attraction-area Occur Num: 11
- request-attraction-type Occur Num: 8
Inform But Not Request Dialog Act- inform-attraction-phone Occur Num: 24
- inform-attraction-type Occur Num: 15
- inform-attraction-area Occur Num: 5
Domain police
Overall Results
Success Rate: 8.7 %
(Precision, Recall, F1) : (0.087, 0.087, 0.087)
Average Dialog Turn (Succ): 16.000
Average Dialog Turn (All): 37.913
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Police-Addr-? Occur Num: 17
- Request-Police-Phone-? Occur Num: 4
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-police-phone Occur Num: 18
- request-police-address Occur Num: 17
Inform But Not Request Dialog ActNothing
Domain hospital
Overall Results
Success Rate: 86.7 %
(Precision, Recall, F1) : (0.867, 0.867, 0.867)
Average Dialog Turn (Succ): 7.692
Average Dialog Turn (All): 12.000
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Hospital-Phone-? Occur Num: 4
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-hospital-phone Occur Num: 4
Inform But Not Request Dialog ActNothing
Domain taxi
Overall Results
Success Rate: 0.0 %
(Precision, Recall, F1) : (0.000, 0.000, 0.000)
Average Dialog Turn (Succ): 0.000
Average Dialog Turn (All): 40.000
System NLU Failed Dialog Act:Nothing
User NLU Failed Dialog Act:Nothing
Dialog Loop- Request-Taxi-Car-? Occur Num: 44
Bad Inform Dialog ActNothing
Request But Not Inform Dialog Act- request-taxi-phone Occur Num: 44
- request-taxi-car type Occur Num: 44
Inform But Not Request Dialog ActNothing