Paper: Using psychophysiological techniques to measure user experience with entertainment technologies

Written by Milad Ghaznavi

Citation

Mandryk, R. L., Inkpen, K. M., & Calvert, T. W. (2006). Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour & Information Technology, 25(2), 141-158.

Image from http://www.jingyif.com/dux/darpa-web-experience-for-kids/

Image from http://www.jingyif.com/dux/darpa-web-experience-for-kids/

Introduction

This paper describes two experiments designed to test the efficacy of physiological measures as evaluators of user experience with entertainment technologies. The authors described two experiments that were designed to test the two main conjectures:

  • Physiological measures can be used to measure a player’s experience with entertainment technology.
  • Normalized physiological measures of experience with entertainment technology will correspond to subjective reports.

The First Experiment

Nhl03ps2
The aim of Experiment One was understanding how psychology can be used to measure user experience with entertainment technology. Eight male participants in this study played NHL 2003 in the four different levels of difficulty, beginner, easy, medium, and difficult. The authors collected two categories of measurements. The first one was a questionary which asked participants about boredom, challenge, frustration, and fun in each difficulty level. Participants rated each subjective measure on a scale of one (low) to five (high). The physiological measures were the second category consisting of:

  • EKG (Cardiovascular measures)
  • HR (Heart Rate)
  • GSR (Galvanic Skin Response)
  • EMG (Electromyography)
  • Resp Rate (Respiratory Rate)
  • RespAmp (Respiratory Amplitude)

The authors’ hypothesis for this experiment was that participants would prefer playing in the condition that was best matched to their level of expertise, experiencing the most enjoyment, satisfaction, and engrossment in this condition.
Based on subjective responses, the mean ratings for challenge were significantly different amongst different difficulty levels in spite of boredom, frustration and fun, which were no significantly different.
Based on physiological measures, authors found out that there were no main effects of difficulty level on any of the physiological measures.
Regarding the correlation of physiological measures to subjective results, they correlated the mean of each physiological measure to the subjective ratings for each participant individually because unlike subjective ratings, there are large individual variations in physiological data. They said that a relationship between a physiological measure and a subjective rating would be evidenced by a significant number of the participants showing the correlation between the physiological measure and the subjective rating. They found out that although there were correlation for most individuals, these correlations were not consistent across participants.

The Second Experiment

The authors’ aim in Experiment Two was further understanding how body responses can be used to create an objective evaluation methodology. In this experiment Ten male participants age 19 to 23 played the same game as Experiment One in two different conditions:

  • Against the computer
  • Against another co-located player

Their main suppositions for Experiment Two were that participants would be more excited, and prefer playing against a friend than against a computer.
Like Experiment One, they collected two categories of measurement. The first one was questionary asking about Boring, Challenging, Easy, Engaging, Exciting, Frustrating, and Fun. The participants again rated these measures on a five-point scale. The second category measurements were psychological measurements similar to Experiment One.
Based on subjective responses, they found out significant differences in Boring, Engaging, Exciting, and Fun between two play conditions.
Based on physiological measures, the authors reported that mean GSR and mean EMG were significantly higher when playing against a co-located player. However, the MANOVA, the technique authors used to analyze physiological measures, showed no significant differences in HR, RespAmp, or RespRate between the two play conditions.
Regarding the correlation of physiological measures to subjective results,they normalized both categories. They reported significant correlations between subjective responses and physiological measures as follows:

  • Fun significantly correlated with GSR.
  • Boredom significantly correlated with EMG.
  • Challenge significantly correlated with RespAmp and EMG.
  • Ease significantly correlated with RespAmp and EMG.
  • Frustration significantly correlated with GSR and RespRate.

Finally authors reported that all of participants except one participant prefer to play against their friend than playing against computer in the future.

Discussion

All in all, this paper is really amazing. The authors work was really interesting and they explained it very well. They clearly explained their works and stated what they wanted and how they measured, evaluated them, and how concluded the results. In this part of my discussion about the paper, at first I explain the good points of this paper and then enumerate the negative points.

Positive points

First of all, I should mention that the paper is really well structured and clear. Every thing is understandable and explicit. This paper is really easy to read and follow, very well structured. They used the same structure for explaining both experiments. The experience design, procedure, every details were mentioned, so for novice researchers in this field, it is really useful. The level of details in describing procedures, data collection, and data analyzing was really amazing. With this level of explanation, this study can be used as a template for other similar studies.
But the most important strength of this paper in my opinion is its results. Based on their results in Experiment Two, the other researcher can identify which physiological measure is suitable for which subjective measurement in studying collaborative situations. For example, based on their results showed in the Table 1, it is better to measure GSR, and RespRate in order to studying Frustration instead of measuring HR and EMG.

Table 1: Correlations between normalized subjective measures and normalized physiological measures. Significant correlations (2-tailed) are shaded in grey. r values are Pearson Correlation.
GSR HR RespAmp RespRate EMG
Fun r
p
.694
.026
-.156
.667
-.173
.633
.086
.812
-.608
.062
Boredom r
p
-.508
.134
-.218
.545
.249
.487
-.182
.614
.823
.003
Challenge r
p
-.383
.275
-.173
.632
.697
.025
Ease r
p
.489
.151
-.068
.852
-.684
.029
-.534
.112
Engagement r
p
.160
.659
.248
.489
.259
.470
-.006
.988
.056
.878
Excitement
Frustration r
p

Negative Points

Besides all good points, the paper has some negative points that I discuss them in this section. The paper in the section 2 described all of the physiology measurements that the authors used in studying experiments. They described them in details, I think it is not necessary to talking about them in this level of details. They could easily reference these measures instead of describing them again in details.

Regarding Experiment One the number of participants was really low. They recruited only eight male participants that one of them did not completed the task.

Regarding Experiment Two, they just studied ten male participants that were really low for the experiment. Another issue about this experiment was that in playing against co-located partner, each participant played against her friend. The authors mentioned that “We recruited the participants individually, but asked them to bring their own partner. We didn’t want the participants playing against strangers, which may have discouraged people who prefer playing alone from signing up.”, but we can criticize that playing against a friend may not be equal to playing against a stranger, and you cannot generalize your findings about playing against friends to a wider subjects, playing against co-located partner. May playing the game against your friend be enjoyable because they have fun and enjoy being in collaboration, not because of playing against a co-located partner. Therefore, this condition might had some effects on responses and physiological measures. The authors could eliminate this effect in that way that half of players play the game against their friends, and half of them play the game against strangers.

Conclusion

As I mentioned before, this paper is interesting and in explaining every thing was really clear and in details. However, the paper had some negative points, but at last we can say that the positive points do compensate them.

One thought on “Paper: Using psychophysiological techniques to measure user experience with entertainment technologies

  1. Erik Paluka

    I disagree with Milad’s criticism about how the paper describes the physiological measurements in too much detail. I believe that this actually adds more to the strength of this paper. It allows novices to become very familiar with the process of taking these measurements, as well as what one can infer based on the collected data. This information is further backed up by a multitude of citations. As a result, I will be referring to this paper in the future whenever I am doing work involving physiological measures. Also, it is a journal paper, which allows for the inclusion of more in-depth discussions and information.

    For both studies, the participants were all male. The authors state that this is to eliminate any gender bias. It would be interesting to see if the results would be the same if only female participants were tested. Concerning the choice of game in both studies, the authors do not explain or justify why they chose to use NHL 2003™ by EA Sports™. They needed a game with varying levels of difficulty, but why did the authors choose a sports game? Selecting the most appropriate video game for a study is important, and justification should be included in the written work, as this allows the reader to understand the thought process behind the decision. Other researchers would then be able to conduct a similar study with a better choice of game if they do not agree with the authors’ justification and game choice.

Comments are closed.