Sustainability 2019 , 11, x doi: FOR PEER REVIEW www.mdpi.comjournal sustainability [616996]

Sustainability 2019 , 11, x; doi: FOR PEER REVIEW www.mdpi.com/journal/ sustainability
Article 1
Evaluation of Recommendation System for 2
Sustainable e -commerce : Accuracy, Diversity and 3
Customer Satisfaction 4
Qinglong Li 1, Ilyoung Choi 2 and Jaekyeong Kim 3,* 5
1 Department of Social Network Science , Kyung Hee University, 26, Kyungheedae -ro, Dongdaemun -gu, 6
Seoul, 02453, Republic of Korea ; [anonimizat] 7
2 Graduate School of Business Administration , KyungHee University, 26, Kyungheedae -ro, Dongdaemun – 8
gu, Seoul, 02453, Republic of Korea ; choice102 @khu.ac.kr 9
3 School of Management , KyungHee University, 26, Kyungheedae -ro, Dongdaemun -gu, Seoul, 02453, 10
Republic of Korea ; [anonimizat] 11
* Correspondence: [anonimizat] ; Tel.: +82-2-961-9355 (F.L.) 12
Received: date; Accepted: date; Published: date 13
Abstract: With the development of information technology and the popularization of mobile 14
devices, collecting various types of customer data such as purchase history or behavior patterns 15
became possible. As the customer data being accumulated, there is a growing demand for 16
personalized recommendation services that provide customized services to customers. Currently, 17
global e -commerce companies offer personalized recommendation services to gain a sustainable 18
competitive advantage. However, previous research on recommendation s ystems has consistently 19
raised the issue that the accuracy of recommendation algorithms does not necessarily lead to the 20
satisfaction of recommended service users. It also claims that customers are highly satisfied when 21
the recommendation system recommends diverse items to them. In this study, we want to identify 22
the factors that determine customer satisfaction when using the recommendation system which 23
provides personalized services. To this end, we developed a recommendation system based on Deep 24
Neural Ne tworks (DNN) and measured the accuracy of recommendation service, the diversity of 25
recommended items and customer satisfaction with the recommendation service. The experimental 26
results of is the study showed that both recommendation system accuracy and div ersity would have 27
a positive effect on customer satisfaction. These results can further improve customer satisfaction 28
with the recommendation system and promote the sustainable development of e -commerce. 29
Keywords: expectancy disconfirmation theory ; custome r satisfaction ; e-commerce personalized 30
service ; recommendation system ; deep neural network 31
32
1. Introduction 33
With the development of information technology and the popularization of mobile devices, the 34
e-commerce market continues to grow. Although many new products are released to satisfy 35
consumer’s needs, customers are spending too much time selecting their pref erred products. 36
Therefore, the importance of personalized recommendation services has emerged . Global companies 37
such as Amazon [1], Netflix [2] and Google [3] are offerin g various services using the 38
recommendation system to pursue the sustainable development of e -commerce [4]. Recommended 39
the item s or services that suit customers' interests not only can improve customer satisfaction by 40
reducing customers' exploring efforts but also increase item sales [5, 6] . Notably, the recommendation 41
system that recommended items or services using customer's purchase history data helps them make 42
decisions among their various alternatives [7]. 43

Sustainability 2019 , 11, x FOR PEER REVIEW 2 of 14
The previous stud y’s recommendation systems focused on enhancing the perf ormance of the 44
recommendation system using customers' purchasing history or preference [8, 9] . The performance 45
of the recommendation system was mainly measured by recommend ation accuracy and the diversity 46
of recommended items . Recommend ation accuracy shows how well the customer's actual preference 47
and predicted preference fit and recommended diversity shows if the customers are willing to 48
recommend the p roducts that recommended by the system and have not previously been purchased 49
to others [8]. A general r ecommendation system study aims to increase the predictive accuracy of the 50
recommendation system [10-15]. However, a study suggests that even if the recommendation system 51
is accurate, when customers were recommended with the same item every time, their satisfaction or 52
reliability is likely to decrease [16]. Other studies suggest that pursuing diversity while maintaining 53
a certain level of recommend ation accuracy in the recommenda tion system increase customer 54
satisfaction [17]. Thus, although the study of recommendation systems focuses on enhancing the 55
performance of the model, customer satisfaction with the recommendation system is just as 56
important as the improvement of the recommended system performance. Nonetheless, not many 57
studies have considered the performance of the recommendation systems and their relation with 58
customer satisfaction. 59
Among the existing studies of the reco mmendation systems, one of the most representative 60
analysis technique used is Collaborative Filtering [18]. Collaborative Filtering is a technique t hat 61
recommends item s that are suitable for customers based on their similar neighbors' preferences and 62
purchasing history [19, 20] . However, Collaboration Filtering has the issue of Cold Start, which is 63
unpredictable for new customers due to a lack of historical data on their past purchases, and the issue 64
of the First Start, which cannot be recommended until someone reflects their preferences [21]. In 65
addition, the scalability of the model could also become an issue as customers' purchasing data are 66
accumulating continually . Collaborative Filtering techniques may cause problems with poor 67
accuracy of models if they use all customer data [21]. Many studies were conducted to supplement 68
such issues as data sparsity and scalability [13, 21] . Recently, deep learning techniques show high 69
performance in image processing or natural language processing areas, has been received attention 70
[22-24]. Recently, many researched has been proposed to apply deep learning techniques to 71
recommendation systems [25-29]. For example, recommendation was proposed using the Recurrent 72
Neural Network technique in an environment where there is no customer preference information 73
available for recommendation [30]. In other studies shown that in environments with customer 74
preference data, it is possible to use Embedding methods and De ep Neural Networks to improve 75
recommended accuracy over Collaborative Filtering methods [31]. 76
Therefore, in this study , in order to overcome data sparsity and scalability issues in 77
recommendation systems , we developed a Deep Neural Network -based recommendation system 78
and identified factors that could affect the level of customer satisfaction. To explore this question, we 79
first employ the expectancy disconfirmation theory (EDT) [32]. According to expectancy 80
disconfirmation theory, the customer's satisfaction depends on the expected level before the purchase 81
of the item and the quality difference after the purchase [33-36]. In other words, if the quality of the 82
goods after purchas e is higher than the expected level before purchase, the customer will be satisfied. 83
Conversely, if the quality of the goods after purchase is lower than the expected level before purchase, 84
the customer will not be satisfied. Second, for the experiment of this study, we develop a Deep Neural 85
Network -based recommendation system and measured the accuracy, diversity and customer 86
satisfaction through a series of experiments with real data set. Finally , we statistically analyzed the 87
experiment output data to iden tify which factors could affect the level of customer satisfaction. 88
Our experiment results indicate that accuracy and diversity positively could have an effect on 89
customer satisfaction. Thus, we claim that the accuracy of recommendation systems and the div ersity 90
of recommended items are important for improving customer satisfaction. 91
2. Literature Review 92
2.1. Recommendation System 93

Sustainability 2019 , 11, x FOR PEER REVIEW 3 of 14
A recommendation system provides the customer with proper item s or services based on the 94
customer's purchasing history and their deduced preference [37]. Previous research on most 95
recommendation systems mainly are based on Collaborat ive Filtering and Content -based Filtering as 96
shown in Figure 1 [38]. 97
98
99
Figure 1. The types of Recommend ation Systems 100
The Collaborative Filtering -based recommendation system is to predict preferences by 101
calculating similarities between customers or item s [18]. This recommended method basically 102
recommends item s based on customers' past purchasing history and preferences. However, his 103
method has a Cold -Star issue where there is not enough data available to measure similarity and the 104
customer's preferences [21]. Furthermore, there is also a First -Star issue where customers' preferred 105
item s are not recommended because they have not yet been purchased [21]. 106
The Content -based Filtering recommendation system is a method to analyze the contents of a item 107
and analyze the similarity between item s and customer preferences, and to recommend suitable item s 108
to the customer [39]. This metho d does not cause the First -Rate issue such as the Collaborative Filtering 109
method, because customers are recommended for items of a similar category to the attributes of the 110
preferred items [21, 40] . However, it fails to reflect the taste or preferences of other customers as it 111
recommends items with high similarity based on the customers' purchasing history. So, this method 112
has an Over Specialization issue that items are similar to those purchased prev iously is recommende d 113
[41]. 114
2.2. Deep Neural Network 115
Deep Neural Network refers to a network of two or more hidden layers between the input and 116
output layers as shown in Figure 2 [42, 43] . This method uses sophisticated mathematical modeling 117
to solve complex problems. Traditional machine learning algorithms are made up of one input and 118
output layer, but the Deep Neural Network co ntains multiple hidden layers, so it can learn various 119
nonlinear relationships [44]. Thus, Deep Neural Network has the advantage of being able to identify 120
the potential structure of data. However, the existing deep neural network has occurred Vanishing 121
Gradient issues if layers increase, and excessive learning of models leads to problems such as 122
overfitting or low learning speed. 123
124

Sustainability 2019 , 11, x FOR PEER REVIEW 4 of 14
125
Figure 2. Deep Neural Network Basic Structure 126
To solve the Vanishing Gradient in Deep Neural Network, the University of Toronto, professor 127
Jeffrey Everest Hinton proposed a ReLU (Rectified Linear Units ) activated function [45]. This function 128
is currently the most commonly used activation function, emerging as an alternative to the existing 129
Sigmoid activation function. Then , to s olve the Over Fitting issue in the learning process Methods 130
were proposed such as these Mini Batch, Dropou t [46]. Furthermore, GPU specialized in vector 131
operations can process large sizes of data quickly, greatly improving the overall learning speed of the 132
model [43]. Currently, the issue that Deep Neural Networks had is solved and are used in different 133
areas such as image processing and natural language processing [47-49]. 134
2.3. Expec tancy Disconfirmation Theory 135
Expectancy disconfirmation theory (EDT) is known as the theory that describes the process of 136
determining customer satisfaction with item s and services [32]. According to the theory, if the 137
performance for item s and services is higher than expected the customer becomes satisfaction, and if 138
the performance is lower than expected, the customer becomes dissatisfaction [32]. In other words, 139
higher performance than a customer's expectations for a special item or service is positive 140
disconfirmed, but a lower performance than the customer's expectations is negative disconfirmed. If 141
the positive disconfirmed is increasing that may increase customer satisfaction and the negative 142
disconfirmed is increasing may increase in customer dissatisfaction [50]. Especially, nega tive 143
disconfirmed are important factors to consider, not just dissatisfaction, but also the customers may 144
have negative results such as moving to other item s or services or discontinuing purchases [51, 52] . 145
146
Figure 3. Expectancy Disconfirmation Theory model 147
Expectancy disconfirmation theory is used in studies to identify the impact on the intention of 148
continuous use of information systems (IS) in the late st technology or online environment [53, 54] . As 149
shown Figure 3 IS continuance intention is influenced by customer satisfaction, which is determined 150
by the difference between perceived quality and expectatio n levels. In conclusion, customer 151
satisfaction has a positive effect on repurchase intentions and word -of-mouth. 152
This Expectancy disconfirmation theory can also be applied to the recommendation system. At 153
this time, the expectation of item or service quali ty that the customer had is compared to the actual 154
item or service quality perceived by the customer through the recommendation system. In the online 155
store, many customers post star rating to the item s that they have purchased. Star ratings are 156

Sustainability 2019 , 11, x FOR PEER REVIEW 5 of 14
important f or predicting initial expectation levels for recommended item s because the 157
recommendation system predicts customer’s purchase scores based on star ratings. Additionally, star 158
ratings are important in measuring performance after purchase because high and lo w ratings indicate 159
positive and negative views of item s [55]. Therefore, we compute disconfirmation as the average of 160
the differences between predicted ratings and actual ratings. The d isconfirmation is computed as 161
follows: 162
𝐷𝑖𝑠𝑐𝑜𝑛𝑓𝑖𝑟𝑚 𝑎𝑡𝑖𝑜𝑛 = 1
𝑚∑(𝑦𝑖−𝑓𝑖)𝑛
𝑖=1 (1)
where 𝑚 is the total number of the recommended item s, and 𝑦𝑖 and 𝑓𝑖, are the actual star ratings 163
and the predicted star ratings , respectively. 164
2.4. Accuracy and Diversity Metrics of recommendation s ystem 165
The performance of recommendation systems can be measured with accuracy and diversity. To 166
evaluate the rating prediction model accuracy, the difference between prediction rating and the 167
actual rating is compared. In this study, the model accuracy was measured based on the mean 168
absolute error (MAE). The mean absolute error (MAE) is computed as follows [8, 56] : 169
𝑀𝐴𝐸 = ∑|𝑝𝑖,𝑗−𝑟𝑗,𝑗|
𝑁 (2)
where 𝑁 is the total number of recommended item s, 𝑝𝑖,𝑗 and 𝑟𝑗,𝑗 are the predicted star rating 170
and the actual star rating by customer 𝑖 for item 𝑗. In other words, an absolute value is taken for the 171
difference between the predicted star rating and the actual star rating and then the sum of them is 172
divided by the total number of recommended item s. The mean absolute error (MAE) is the mean of the 173
absol ute value of the error, and regardless of the magnitude of the error, it will be given the same weight. 174
Recently, most studies have suggested measuring diversity as it has been identified that there is 175
a limit to focusing only on improving the accuracy of the recommendation system. Lathia et al [17] 176
proposed to measure the diversity of recommended items as follows: 177
𝐷𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦 (𝐿1,𝐿2,N)= |𝐿2−𝐿1|
𝑁 (3)
Where recommended list 𝐿1 is recommended at one point and 𝐿2 is recommended at the next 178
point to customer s. Next, in the 𝐿2 recommended list, divide the number of items not on the 𝐿1 179
recommended list by 𝑁. 180
3. Hypotheses Development 181
Customer satisfacti on includes the customer's assessment after purchasing the item or service 182
[57-59]. It is important for companies to satisfy customers because it is likely that customers who are 183
satisfied with their items and services will repurchase and recommend those to the people around 184
them . 185
The previous studies of the recommendation systems have been done mainly in a way to 186
improve the ac curacy [10-14]. In fact, most studies have suggested algorithms that improve the 187
recommendation system accuracy. Research shows that customer satisfa ction with a 188
recommendation system can change depending on how accurately recommend item s and services 189
for customers [60-62]. Moreover, if recommendations are made to suit the customer's preference, the 190
customer can form a positive attitude toward the recommendation system. In other words, accurate 191
recommendations increase the likelihood that customers will find item s that suit their preferences, 192
which in theory increases custom er satisfaction. Therefore, the hypothesis is as follows: 193
194

Sustainability 2019 , 11, x FOR PEER REVIEW 6 of 14
Hypothesis 1 (H1) : Accurate recommendations as a function of the size of customer dataset positively 195
influence customer satisfaction 196
197
It is known that if the recommendation system is highly accurate the customer satisfaction level 198
is high [37]. However, even if the recommendation system is highly accurate, the satisfaction or 199
reliability of the recommendation system will decrease if the customer received the recommendation 200
service composed of the same items every time. More studies have claimed that it is very important 201
to provide a variety of items or services to customers while maintaining a certain level of accuracy 202
[17]. Other studies argue that a more diverse list of recommendations increases t he probability that a 203
customer will purchase the items. [63-66]. As a result, more various recommended items improve 204
customer satisfaction. Therefore, the hypothesis is as follows: 205
206
Hypothesis 2 (H2) : Diverse recommendations as a function of the size of customer dataset positively influence 207
customer satisfaction 208
4. Experiments 209
To test the hypothesis, we developed a Deep Neural Network -based recommendation system, 210
which is free of sparsity and scalability issues of traditional recommendation systems. And we 211
measured the accuracy, diversity and customer satisfaction of the system through a series of 212
experiments with a real dataset. Then, we statistically analyze the experiment result data to identify 213
which factors could affect customer satisfaction. The overall flow chart is shown in Figure 4. 214
215
216
Figure 4. Flow chart of method steps 217
4.1. Dataset s Description 218
For experiments, we use three datasets: MovieLens 100K, MovieLens 1M, MovieLens 10M from 219
GroupLens . All datasets contain star ratings on items (1 to 5 stars). The first dataset, MovieLens 100k 220
dataset contains 100,000 ratings from 943 users on 1,682 movies and the density is 6.30% . The second 221
dataset, MovieLens 1M, with 1 million ratings from 6,040 users on 3,706 items and the density is 222
4.47%. The third dataset, MovieLens 10M dataset contains 10,000 ,054 ratings by 69,878 users on 1 0,677 223
items and the density is 1.34%. In this study, experiment s were conducted by selecting items with 224
more than 50 ratings and users with more than 25 ratings . A summary of data set statistics is shown 225
in Table 1. 226

Sustainability 2019 , 11, x FOR PEER REVIEW 7 of 14
Table 1. Descriptive statistics of the MovieLens datasets . 227
Dataset #users #items #ratings #density
MovieLens 100K 943 1,682 100,000 6.30%
MovieLens 1M 6,040 3,706 1,000,209 4.47%
MovieLens 10M 69,878 10,677 10,000,054 1.34%
4.2. Implementation Details 228
To test our research hypothesis and in order to overcome data sparsity and scalability issues, we 229
developed a Deep Neural Network (DNN) -based recommendation system as shown in figure5. Then 230
we measured the accuracy and diversity by MAE and diversity metrics proposed b y Lathia et al [17] 231
, respectively, which are shown in equation (2) and (3). A previous study used questionnaires to 232
measure customer satisfaction with the recommendation system [67, 68] . In this study, we measured 233
disconfirmation -based customer satisfaction through a series of experiments. Here, disconfirmation 234
is computed as the average of the differences between predicted ratings and actual ratings. 235
Experiments used 80% as a training dataset and 20% as a test dataset. A training dataset used to train 236
the recommendation system, and the test dataset is used to predict the ratings and recommends Top – 237
5 lists with the highest ratings for the customer . 238
239
240
Figure 5. Deep Neural Network based -Recommendation System 241
To training the Deep Neural Network -based recommendation system, the dataset consisting of 242
user ID, movie ID, and ratings are converted to user embedding vector, movie embedding vector, 243
and ratings to form a training dataset. The Deep Neural Network -based recommendation system 244
used in this study is shown in Figure 5. We develop the system based on Keras 2.2.4 and test it on an 245
NVIDIA GEFORCE RTX 2080 Ti GPU. This recommendation system is trained by estimating the 246
relationship between the latent vector of th e user and the latent vector of the item. Therefore, we 247
separately construct a 20 -dimensional embedding vector model for the user ID and a 10 -dimensional 248
embedding vector model for the movie ID. Then, the Deep Neural Network model was constructed 249
that comb ines the two models in the input phase of the fully connected layer. 250

Sustainability 2019 , 11, x FOR PEER REVIEW 8 of 14
251
Figure 6. Comparison Training Loss Graph 252
The input layer receives the training dataset and outputs it without any conversion process. The 253
user models and item models output data in 10 -dimensional vectors and 20 -dimensional vectors 254
through the embedding layer. The user model and item model output vector are connected, and it 255
combined into a 30 -dimensional vector and entered a fully connected layer. The hidden layer of the 256
fully connected layer consists of three nodes and converts the data from a 30 -dimensional vector into 257
a 10 -dimensional vector to output. As a hidden layer activation function, it is used the RELU 258
(Rectified Linear Unit). The output layer consists of one node and outputs 1 0-dimensional vector data 259
as a 1 -dimensional value. The experiment was repeated up to 250 times until the loss value remained 260
unchanged as shown in Figure 6. Then the optimization function is used Adam and as the loss 261
function, it is used MAE. After traini ng the Deep Neural Network model, the user embedding vector 262
and item embedding vector of the test dataset are inputted to predict the rating of the item and 263
recommends Top -5 lists with the highest ratings for the customer . 264
4.4. Results 265
The mean and standard deviation for accuracy, diversity, and customer satisfaction at 266
MovieLens 100K, MovieLens 1M, MovieLens 10M are listed in Table 2. The mean value for accuracy 267
and diversity was between 0.655 and 0.722 and between 0.477 and 0.570, resp ectively. Furthermore, 268
the mean value of customer satisfaction was between 0.144 and 0.204. The highest value of accuracy 269
is at a MovieLens 10M (0.655) and the lowest value of accuracy at a MovieLens 100K (0.722). The 270
highest value of diversity is at a Mov ieLens 1M (0.570) and the lowest value of diversity is at a 271
MovieLens 100K (0.477). The highest value of customer satisfaction is at a MovieLens 1M (0.204) and 272
the lowest value of customer satisfaction is at a MovieLens 100K (0.165). Especially, item s with highly 273
predicted ratings are recommended regardless of the actual purchase. So, customer satisfaction is 274
positive because it is defined as the average of the differences between predicted ratings and actual 275
ratings. 276
Table 2. Descriptive statistics of accu racy, diversity, and customer satisfaction . 277
Dataset s Variables Mean Standard Deviation
MovieLens 100K Accuracy 0.722 0.415
Diversity 0.477 0.296
Customer Satisfaction 0.165 0.620
MovieLens 1M Accuracy 0.673 0.399
Diversity 0.570 0.292
Customer Satisfaction 0.204 0.555
MovieLens 10M Accuracy 0.655 0.368
Diversity 0.522 0.295
Customer Satisfaction 0.144 0.525
278
To test the research hypothesis, we performed multiple regression analyses under the three 279
Datasets. Table 3 summarizes the results of multiple linear regressions for hypotheses H1 and H2. 280

Sustainability 2019 , 11, x FOR PEER REVIEW 9 of 14
The table shows the unstandardized regression coefficient, the stan dardized regression coefficient, t – 281
value, tolerance, and variance inflation factor (VIF) of each predictor and R2, adjusted R2, F, and 282
Durbin -Watson of each model in linear regression analysis. Under the three Datasets, each model is 283
statistically signific ant ((MovieLens 100K) F=80.798, p<0.01, (MovieLens 1M) F=1140.951, p<0.01 and 284
(MovieLens 10M) F=13030.16, p<0.01 for three Dataset predicting 16.8%, 29.1% and 29.7% of the 285
variance in customer satisfaction). Moreover, there is no multicollinearity between accuracy and 286
diversity under the three Datasets ((MovieLens 100K) tolerance=.999 and VIF=1.001, (MovieLens 1M) 287
tolerance=.999 and VIF=1.001, and (MovieLens 10M) tolerance=.999 and VIF=1.001). The results show 288
that accuracy and diversity positively and sign ificantly affect the disconfirmation (p<0.01), therefore 289
supporting hypothesis 1 and hypothesis 2. These results suggest that both recommended accuracy 290
and diversity are important for customer satisfaction. 291
Table 3. Results of multiple regression analysis . 292
Dataset s Dependent
Variable Unstandardized
Beta Standardized
Beta t-value Tolerance VIF
MovieLens
100K
Accuracy .391 .262 9.979** .999 1.001
Diversity .674 .321 8.121** .999 1.001
𝑅2 = 0.1 68, Adjusted 𝑅2= 0.165, F = 80. 798**, Durbin -Watson=2.00 1

MovieLens
1M Accuracy .597 .430 38.036 ** .999 1.001
Diversity .603 .317 28.017 ** .999 1.001
𝑅2 = 0.292, Adjusted 𝑅2= 0.291, F = 1140 .951**, Durbin -Watson=2.0 16

MovieLens
10M Accuracy .581 .408 120.673** .999 1.001
Diversity .622 .350 103.430** .999 1.001
𝑅2 = 0.297, Adjusted 𝑅2= 0.297, F = 13010 .396**, Durbin -Watson= 1.992

** p < 0.01 , * p < 0.05 . 293
Also, a one -way analysis of variance (ANOVA) was conducted to determine whether there was 294
a significant difference in customer satisfaction under the three datasets. The Scheffe Post Hoc Test 295
was used to identify multiple comparisons of group means. The re sults presented in Table 4 indicated 296
that there was a significant customer satisfaction difference between the dataset. (F = 34.088, Sig. = 297
0.000). The significant mean difference was found between MovieLens 1M and MovieLens 10M 298
(mean difference = -0.609, Sig. = 0.000), which indicates that customer satisfaction is related to 299
diversity when the accuracy is somewhat constant. 300
Table 4. One -way ANOVA of customer satisfaction . 301
One -Way
ANOVA Sum of
Squares Df Mean Square F Sig.
Between Group 19.077 2 9.539 34.088 .000
Within
Group 19022.218 67978 0.280
Total 19041.296 67980
Feature Mean
Difference Std.Error Sig.
MovieLens 100K MovieLens 1M
MovieLens 10M -0.039
0.218 0.199
0.188 0.148
0.507
MovieLens 1M MovieLens 100K **
MovieLens 10M 0.039
0.061 0.199
0.007 0.148
0.000

Sustainability 2019 , 11, x FOR PEER REVIEW 10 of 14
MovieLens 10M MovieLens 100K
MovieLens 1M** -0.218
-0.061 0.188
0.007 0.507
0.000
5. Discussion and Conclusions 302
A recommendation system is a critical tool for e -commerce companies to pursue sustainable 303
growth. Therefore the global companies such as Amazon [1], Netflix [2] and Google [3]are offering 304
recommendation services to their customers to gain a competitive advantage. However, there are not 305
only trade -offs between recommendation accuracy and diversity of recommended items but also 306
continuing debates over which factor, accuracy or diversity, have a more significant impact on 307
customer satisfaction. Thus, we investigated which factors affect customer satisfaction through 308
statistical analyses of a series of experiments performed with a Deep Neural Network -based 309
recommendation system . 310
This study provides the following implications for the performance aspects of the 311
recommendation systems and customer satisfaction. First, we used the expectancy disconfirmation 312
theory (EDT) to measure customer satisfaction with Deep Neural Network -based recommendation 313
systems for the first time. In particular, the recommended accuracy and recommended diversity of 314
the recommendation system were measured using a real dataset, so disconfirmation was calculated 315
as an indicator for measuring customer satisfac tion. Second, we identified the impact of dataset size 316
on recommended accuracy and diversity. The larger the dataset size, the recommended accuracy 317
increase, but it is not sure about the diversity metric because it increases and decreased according to 318
the increase of dataset size. Third, we identified factors that affect customer satisfaction. While 319
recommendation accuracy has continued to increase, and diversity has decreased, then results that 320
all have a positive impact on customer satisfaction. This is consistent with the results of previous 321
studies in which the customer was satisfied by various item s recommended while maintaining a 322
certain level of accuracy [17]. 323
As a result, these results provide e -commerce companies with the following insights. First, it is 324
possible to increase customer satisfaction by recommendin g various item s while maintaining a 325
certain level of constant accuracy. Through, suppliers can increase sales by offering a variety of item s 326
that meet customer preferences. Second, the more customer datasets have high recommendation 327
accuracy, but the diver sity and satisfaction levels are rather reduced. Therefore, when customer data 328
is accumulated to some extent, it should be managed regularly . 329
There are several limitations to this study. First, the experiment with the existing dataset that 330
was different fr om finding information in a real environment. Therefore, more work must be 331
performed to know whether the results hold true in the real world. Second, this study was conducted 332
using a movie dataset. For a general of research results, further study is requir ed using the dataset 333
from various domains. Third, the algorithm used in the experiment is a deep neural network -based 334
algorithm that is commonly used in the study of recommendation systems. We are not sure whether 335
other algorithms, such as Recurrent Neural Network (RNN) or Convolutional Neural Network 336
(CNN) would result in the same findings. Therefore, further research is needed on how the results of 337
this study will appear when various algorithms are used. 338
339
Author Contributions: Conceptualization, I.C. and Q.L.; Methodology, I.C. and Q.L.; Data curation, Q.L. and 340
J.K.;Writing, Q.L.; Supervision, J.K . 341
Funding: This research received no external funding. 342
Conflicts of Interest: The authors declare no conflict of interest. 343
References 344
1. Linden, G.; Smith, B.; York, J., Amazon. com recommendations: Item -to-item collaborative filtering. 345
IEEE Internet computing 2003 , (1), 76 -80. 346

Sustainability 2019 , 11, x FOR PEER REVIEW 11 of 14
2. Bennett, J.; Lanning, S. In The n etflix prize , Proceedings of KDD cup and workshop, 2007; New York, 347
NY, USA.: 2007; p 35. 348
3. Das, A. S.; Datar, M.; Garg, A.; Rajaram, S. In Google news personalization: scalable online collaborative 349
filtering , Proceedings of the 16th international conferen ce on World Wide Web, 2007; ACM: 2007; pp 350
271-280. 351
4. Guo, Y.; Yin, C.; Li, M.; Ren, X.; Liu, P., Mobile e -commerce recommendation system based on multi – 352
source information fusion for sustainable e -business. Sustainability 2018, 10, (1), 147. 353
5. Kumar, N.; Benbasat, I., Research note: the influence of recommendations and consumer reviews on 354
evaluations of websites. Information Systems Research 2006, 17, (4), 425 -439. 355
6. Xiao, B.; Benbasat, I., E -commerce product recommendation agents: use, characteristics, and impact. 356
MIS quarterly 2007, 31, (1), 137 -209. 357
7. Thirumalai, S.; Sinha, K. K., Customization strategies in electronic retailing: Implications of customer 358
purchase b ehavior. Decision Sciences 2009, 40, (1), 5 -36. 359
8. Herlocker, J. L.; Konstan, J. A.; Terveen, L. G.; Riedl, J. T., Evaluating collaborative filtering 360
recommender systems. ACM Transactions on Information Systems (TOIS) 2004, 22, (1), 5 -53. 361
9. Pu, P.; Chen, L.; Hu, R. In A user -centric evaluation framework for recommender systems , Proceedings of the 362
fifth ACM conference on Recommender systems, 2011; ACM: 2011; pp 157 -164. 363
10. Cho, Y. H.; Kim, J. K., Application of Web usage mining and product taxonomy to coll aborative 364
recommendations in e -commerce. Expert systems with Applications 2004, 26, (2), 233 -246. 365
11. Cho, Y. H.; Kim, J. K.; Kim, S. H., A personalized recommender system based on web usage mining and 366
decision tree induction. Expert systems with Applicati ons 2002, 23, (3), 329 -342. 367
12. Herlocker, J. L.; Konstan, J. A.; Riedl, J. In Explaining collaborative filtering recommendations , Proceedings 368
of the 2000 ACM conference on Computer supported cooperative work, 2000; ACM: 2000; pp 241 -250. 369
13. Lee, J.; Beng io, S.; Kim, S.; Lebanon, G.; Singer, Y. In Local collaborative ranking , Proceedings of the 23rd 370
international conference on World wide web, 2014; ACM: 2014; pp 85 -96. 371
14. Shardanand, U.; Maes, P. In Social information filtering: algorithms for automating" word of mouth" , Chi, 372
1995; Citeseer: 1995; pp 210 -217. 373
15. Zhu, W.; Lu, J.; Li, Y.; Yang, Y., A Pick -Up Points Recommendation System for Ridesourcing Service. 374
Sustainability 2019, 11, (4), 1097. 375
16. Son, J.; Kim, S. B.; Kim, H.; Cho, S., Review and analys is of recommender systems. Journal of Korean 376
Institute of Industrial Engineers 2015, 41, (2), 185 -208. 377
17. Lathia, N.; Hailes, S.; Capra, L.; Amatriain, X. In Temporal diversity in recommender systems , Proceedings 378
of the 33rd international ACM SIGIR confer ence on Research and development in information retrieval, 379
2010; ACM: 2010; pp 210 -217. 380
18. Goldberg, D.; Nichols, D.; Oki, B. M.; Terry, D., Using collaborative filtering to weave an information 381
tapestry. Communications of the ACM 1992, 35, (12), 61 -71. 382
19. Lekakos, G.; Giaglis, G. M., Improving the prediction accuracy of recommendation algorithms: 383
Approaches anchored on human factors. Interacting with computers 2006, 18, (3), 410 -431. 384
20. Resnick, P.; Iacovou, N.; Suchak, M.; Bergstrom, P.; Riedl, J. In GroupLens: an open architecture for 385
collaborative filtering of netnews , Proceedings of the 1994 ACM conference on Computer supported 386
cooperative work, 1994; ACM: 1994; pp 175 -186. 387
21. Su, X.; Khoshgoftaar, T. M., A survey of collaborative filtering techniqu es. Advances in artificial 388
intelligence 2009, 2009. 389

Sustainability 2019 , 11, x FOR PEER REVIEW 12 of 14
22. Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; Bengio, Y., 390
Learning phrase representations using RNN encoder -decoder for statistical machine translation. arXiv 391
preprint arXiv:1406.1078 2014 . 392
23. Girshick, R. In Fast r -cnn, Proceedings of the IEEE international conference on computer vision, 2015; 393
2015; pp 1440 -1448. 394
24. Ren, S.; He, K.; Girshick, R.; Sun, J. In Faster r -cnn: Towards real -time object detection with region proposal 395
networks , Advances in neural information processing systems, 2015; 2015; pp 91 -99. 396
25. Cheng, H. -T.; Koc, L.; Harmsen, J.; Shaked, T.; Chandra, T.; Aradhye, H.; Anderson, G.; Corrado, G.; 397
Chai, W.; Ispir, M. In Wide & deep learning for reco mmender systems , Proceedings of the 1st workshop on 398
deep learning for recommender systems, 2016; ACM: 2016; pp 7 -10. 399
26. Covington, P.; Adams, J.; Sargin, E. In Deep neural networks for youtube recommendations , Proceedings of 400
the 10th ACM conference on recommender systems, 2016; ACM: 2016; pp 191 -198. 401
27. Cremonesi, P.; Garzotto, F.; Negro, S.; Papadopoulos, A. V.; Turrin, R. In Looking for “good” 402
recommendations: A comparative evaluation of recommender systems , IFIP Conference on Human -Com puter 403
Interaction, 2011; Springer: 2011; pp 152 -168. 404
28. Hidasi, B.; Karatzoglou, A.; Baltrunas, L.; Tikk, D., Session -based recommendations with recurrent 405
neural networks. arXiv preprint arXiv:1511.06939 2015 . 406
29. Van den Oord, A.; Dieleman, S.; Schrauwen , B. In Deep content -based music recommendation , Advances 407
in neural information processing systems, 2013; 2013; pp 2643 -2651. 408
30. Devooght, R.; Bersini, H., Collaborative filtering with recurrent neural networks. arXiv preprint 409
arXiv:1608.07400 2016 . 410
31. He, X.; Liao, L.; Zhang, H.; Nie, L.; Hu, X.; Chua, T. -S. In Neural collaborative filtering , Proceedings of the 411
26th international conference on world wide web, 2017; International World Wide Web Conferences 412
Steering Committee: 2017; pp 173 -182. 413
32. Oliver, R. L., Effect of expectation and disconfirmation on postexposure product evaluations: An 414
alternative interpretation. Journal of applied psychology 1977, 62, (4), 480. 415
33. Athiyaman, A., Linking student satisfaction and service quality perceptions: the cas e of university 416
education. European journal of marketing 1997, 31, (7), 528 -540. 417
34. Bitner, M. J., Evaluating service encounters: the effects of physical surroundings and employee 418
responses. Journal of marketing 1990, 54, (2), 69 -82. 419
35. Chong, B.; Wong, M., Crafting an effective customer retention strategy: a review of halo effect on 420
customer satisfaction in online auctions. International Journal of Management and Enterprise Development 421
2005, 2, (1), 12 -26. 422
36. Maxham III, J. G., Service recovery's influe nce on consumer satisfaction, positive word -of-mouth, and 423
purchase intentions. Journal of business research 2001, 54, (1), 11 -24. 424
37. Liang, T. -P.; Lai, H. -J.; Ku, Y. -C., Personalized content recommendation and user satisfaction: 425
Theoretical synthesis and empirical findings. Journal of Management Information Systems 2006, 23, (3), 45 – 426
70. 427
38. Herlocker, J. L.; Konstan, J. A.; Borchers, A.; Riedl, J. In An algorithmic framework for performing 428
collaborative filtering , 22nd Annual International ACM SIGIR Confer ence on Research and Development 429
in Information Retrieval, SIGIR 1999, 1999; Association for Computing Machinery, Inc: 1999; pp 230 – 430
237. 431

Sustainability 2019 , 11, x FOR PEER REVIEW 13 of 14
39. Wu, Y. -H.; Chen, A. L. In Index structures of user profiles for efficient web page filtering services , Proceedings 432
20th IEEE International Conference on Distributed Computing Systems, 2000; IEEE: 2000; pp 644 -651. 433
40. Wartena, C.; Slakhorst, W.; Wibbels, M.; Gantner, Z.; Freudenthaler, C.; Newell, C.; Schmidt -Thieme, 434
L. In Keyword -Based TV Program Recommendation , ITWP@ IJCAI, 2011; 2011. 435
41. Balabanović, M.; Shoham, Y., Fab: content -based, collaborative recommendation. Communications of the 436
ACM 1997, 40, (3), 66 -72. 437
42. Bengio, Y.; Courville, A.; Vincent, P., Representation learning: A review and new perspectives. IEEE 438
transactions on pattern analysis and machine intelligence 2013, 35, (8), 1798 -1828. 439
43. Schmidhuber, J., Deep learning in neural networks: An overview. Neural networks 2015, 61, 85 -117. 440
44. Szegedy, C.; Toshev, A.; Erhan, D. In Deep neural networks for object detection , Advances in neural 441
information processing systems, 2013; 2013; pp 2553 -2561. 442
45. Nair, V.; Hinton, G. E. In Rectified linear units improve r estricted boltzmann machines , Proceedings of the 443
27th international conference on machine learning (ICML -10), 2010; 2010; pp 807 -814. 444
46. Hinton, G. E., A practical guide to training restricted Boltzmann machines. In Neural networks: Tricks of 445
the trade , Springer: 2012; pp 599 -619. 446
47. LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P., Gradient -based learning applied to document recognition. 447
Proceedings of the IEEE 1998, 86, (11), 2278 -2324. 448
48. Mikolov, T.; Karafiát, M.; Burget, L.; Černocký, J.; Khudanpur, S . In Recurrent neural network based 449
language model , Eleventh annual conference of the international speech communication association, 450
2010; 2010. 451
49. Sainath, T. N.; Mohamed, A. -r.; Kingsbury, B.; Ramabhadran, B. In Deep convolutional neural networks 452
for L VCSR , 2013 IEEE international conference on acoustics, speech and signal processing, 2013; IEEE: 453
2013; pp 8614 -8618. 454
50. McCollough, M. A.; Berry, L. L.; Yadav, M. S., An empirical investigation of customer satisfaction after 455
service failure and recovery. Journal of service research 2000, 3, (2), 121 -137. 456
51. Audrain -Pontevia, A. -F.; Balague, C., The relationships between dissatisfaction, complaints and 457
subsequent behavior in electronic marketplace. ACR North American Advances 2008 . 458
52. Lu, Y.; Lu, Y.; Wang , B., Effects of dissatisfaction on customer repurchase decisions in e -commerce -an 459
emotion -based perspective. Journal of Electronic Commerce Research 2012, 13, (3), 224. 460
53. Bhattacherjee, A., Understanding information systems continuance: an expectation -confirmation 461
model. MIS quarterly 2001 , 351 -370. 462
54. Roca, J. C.; Chiu, C. -M.; Martí nez, F. J., Understanding e -learning continuance intention: An extension 463
of the Technology Acceptance Model. International Journal of human -computer studies 2006, 64, (8), 683 – 464
696. 465
55. Mudambi, S. M.; Schuff, D., What makes a helpful review? A study of customer reviews on Amazon. 466
com. MIS quarterly 2010, 34, (1), 185 -200. 467
56. Goldberg, K.; Roeder, T.; Gupta, D.; Perkins, C., Eigentaste: A constant time collaborative filtering 468
algorithm. information retrieval 2001, 4, (2), 133 -151. 469
57. Calvo -Porral, C.; Lévy -Mangin, J. -P., Switching behavior and customer satisfaction in mobile services: 470
Analyzing virtual and traditional operators. Computers in Human Behavior 2015, 49, 532 -540. 471
58. Deng, Z.; Lu, Y.; Wei, K. K.; Zhang, J., Underst anding customer satisfaction and loyalty: An empirical 472
study of mobile instant messages in China. International journal of information management 2010, 30, (4), 473
289-300. 474

Sustainability 2019 , 11, x FOR PEER REVIEW 14 of 14
59. Gerpott, T. J.; Rams, W.; Schindler, A., Customer retention, loyalty, and satisfac tion in the German 475
mobile cellular telecommunications market. Telecommunications policy 2001, 25, (4), 249 -269. 476
60. Abdel -Hafez, A.; Tang, X.; Tian, N.; Xu, Y. In A reputation -enhanced recommender system , International 477
Conference on Advanced Data Mining an d Applications, 2014; Springer: 2014; pp 185 -198. 478
61. Christoffel, F.; Paudel, B.; Newell, C.; Bernstein, A. In Blockbusters and wallflowers: Accurate, diverse, and 479
scalable recommendations with random walks , Proceedings of the 9th ACM Conference on Recomm ender 480
Systems, 2015; ACM: 2015; pp 163 -170. 481
62. Zhou, T.; Kuscsik, Z.; Liu, J. -G.; Medo, M.; Wakeling, J. R.; Zhang, Y. -C., Solving the apparent diversity – 482
accuracy dilemma of recommender systems. Proceedings of the National Academy of Sciences 2010, 107, 483
(10), 4511 -4515. 484
63. McGinty, L.; Smyth, B. In On the role of diversity in conversational recommender systems , International 485
Conference on Case -Based Reasoning, 2003; Springer: 2003; pp 276 -290. 486
64. McNee, S. M.; Riedl, J.; Konstan, J. A. In Being accurate is not enough: how accuracy metrics have hurt 487
recommender systems , CHI'06 extended abstracts on Human factors in computing systems, 2006; ACM: 488
2006; pp 1097 -1101. 489
65. Smyth, B.; McClave, P. In Similarity vs. diversity , International conference on case -base d reasoning, 2001; 490
Springer: 2001; pp 347 -361. 491
66. Ziegler, C. -N.; McNee, S. M.; Konstan, J. A.; Lausen, G. In Improving recommendation lists through topic 492
diversification , Proceedings of the 14th international conference on World Wide Web, 2005; ACM: 2005 ; 493
pp 22 -32. 494
67. Zins, A. H.; Bauernfeind, U.; Del Missier, F.; Venturini, A.; Rumetshofer, H., An experimental usability 495
test for different destination recommender systems . na: 2004. 496
68. Ekstrand, M. D.; Harper, F. M.; Willemsen, M. C.; Konstan, J. A. In User perception of differences in 497
recommender algorithms , Proceedings of the 8th ACM Conference on Recommender systems, 2014; ACM: 498
2014; pp 161 -168. 499
500
© 2019 by the authors. Submitted for possible open access publication under the terms
and conditions of the Creative Commons Attribution (CC BY) license
(http://creativecommons.org/licenses/by/4.0/).
501