Authors: Gabriel Dzodom, Akshay Kulkarni, Catherine C. Marshall, Frank M. Shipman

Abstract

Educational prediction games use the popularity and engagement of fantasy sports as a success model to promote learning in other domains. Fantasy sports motivate players to stay up-to-date with relevant news and explore large statistical data sets, thereby deepening their domain understanding while potentially honing their data analysis skills. We conducted a study of fantasy sports players, and discovered that while some participants performed sophisticated data analysis to support their gameplay, far more relied on news and published commentary. We used results from this study to design a prototype prediction game, Fantasy Climate, which helps players move from intuitions and advice to consuming news and analyzing data by supporting a variety of activities essential to gameplay. Because news is a key component of Fantasy Climate, we evaluated two link-based interfaces to domain-related news, one geospatial and the other organized as a list. The evaluation revealed that news presentation has a strong effect on players’ engagement and performance: players using the geospatial interface not only were more engaged in the game; they also made better predictions than players who used the list-based presentation.

CCS CONCEPTS

Human-centered computing~Human computer interaction (HCI);

Empirical studies in HCI; User studies

KEYWORDS: Prediction games; fantasy sports; serious games;

player engagement; educational games; data analysis skills.

ACM Reference format: Gabriel Dzodom, Akshay Kulkarni, Catherine C. Marshall, and Frank M. Shipman. 2020. Keeping People Playing: The Effects of Domain News Presentation on Player Engagement in Educational Prediction Games. In Proceedings of the 31[st] ACM Conference on Hypertext and Social Media (HT’20), July 13-15, 2020, Virtual Event, USA. 6 pages. ACM, New York, NY, USA. https://doi.org/10.1145/3372923.3404813

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. HT ’20, July 13–15, 2020, Virtual Event, USA. © 2020 Association of Computing Machinery. ACM ISBN 978-1-4503-7098-1/20/07...$15.00. https://doi.org/10.1145/3372923.3404813

1 INTRODUCTION

Online data provides new opportunities for people to become better informed about the world around them. For example, government agencies (e.g. NOAA and NASA) collect, maintain, and publish data documenting aspects of the Earth’s current and historic climate. However, it is unrealistic to expect most individuals to seek out and analyze detailed climate data without a reason to do so. Games have been shown to be an effective means of engaging learners in both formal and informal educational environments [8, 9, 15]. We have been investigating an approach to motivate people to engage with open datasets by bringing them together with domain-related information resources and analysis tools in the context of prediction games.

Our exploration of prediction games is motivated by the popularity of fantasy sports [13]. Fantasy sports have players act as if they are the manager of an athletic team [18]. Players may compete with each other directly or in a league by deciding which athletes to draft, start, and trade with other fantasy players. Players’ scores are based on the performance of their athletes during actual sporting events, which motivates players to stay upto-date with relevant news. Furthermore, studies show that fantasy sports competition spurs players to explore and interpret large data sets to improve their predictions [2, 3, 11, 16]. As a result, players learn more about the sport and may gain data analysis experience.

The success of fantasy sports sparked interest in applying its game model to other domains. One example, FantasySCOTUS, encourages students to learn about constitutional law and current Supreme Court decisions [1]. Such efforts have focused on the development of playable leagues and have not explored how alternative designs affect player engagement and activity.

This paper begins with results of a survey of fantasy sports players, focusing on their data gathering and analysis practices. Next, we describe a sample prediction game, Fantasy Climate. Finally, we report on an evaluation of two news presentation modes, since the survey identified news as an important feature of prediction games that can affect engagement, encourage data literacy, and improve prediction accuracy.

2 FANTASY SPORTS NEWS AND DATA USE

Many studies of fantasy sports have explored players’ motivations for participating in leagues, especially the social aspects of the game, e.g. [7, 12, 16]. Fewer have characterized the actual practice of gameplay, especially how players gather and analyze news and

data [10, 17, 19, 21]. To more fully understand how players inform their team selections and gameplay strategies, we conducted a survey of self-identified fantasy sports players on Amazon Mechanical Turk [4]. Participants answered 57 questions, including 11 open-ended questions about their motivations, gameplay practices, and strategy. All 160 completed questionnaires were kept and analyzed.

In this section, we selectively discuss two research questions that the survey addressed:

Q1: Which information do players use when making decisions and where does it come from? Q2: How do players collect and analyze data?

Other portions of the study were used to design components of the game, such as its communications infrastructure. Study results confirmed the Fantasy Sports literature’s reports about the importance of players’ social motivations. For many, gameplay is a way to maintain bonds with family and friends: “it is a way to connect with my father who loves sports and also plays” (P008), or cement workplace relations, “To take part in an office activity with co-workers and not look like someone who doesn't want to participate in social activities in the workplace.” (P125) This data was coded and analyzed, and will be covered elsewhere.

For this bottom-up analysis, we triangulated among 4 openended, 13 7-point Likert scale questions, and a multiple choice response to code how participants implemented their fantasy sports information-seeking and data-use strategies, evaluating whether their gameplay primarily used:

Intuition or favorites (participant seeks little information

to support gameplay, instead trusting gut reactions)

Advice or secondary sources (participant consults

trusted experts, forums, or friends)

Primary news sources (participant seeks news coverage,

sports websites, and may examine player statistics without manipulating them)

Data resources (participant obtains or constructs

dataset(s) to systematically track player attributes)

Statistical resources (participant uses models or more

complex analytic tools)

These categories were created to reflect different patterns in participant responses. Triangulation among a participant’s responses helped us determine how various sources and tools are used together. E.g. A participant may report using Excel, but describe primarily managing ranked lists of available players and positions, rather than using the tool’s statistical analysis or projection capabilities. A combination of open-ended responses, supplemented by other information from the questionnaire (e.g. the participant’s mathematical skills, level of involvement with fantasy sports, and reasons for playing) provide us a basis for assessing whether and where they seek information and data, and the degree to which they perform additional analysis. Most fantasy sports enthusiasts use more than one technique for judging players. In our analysis, they are coded to reflect the most detailed strategy they described. E.g., if participants used both expert advice, and data resources, they are coded as using data resources,

since we are focusing on improving the extent of their analytic capabilities.

Table 1. Coding of gameplay resource use

Table 1 shows the results of this analysis. A small number, about 7% (11/160), referred to using predictive models or projections. P127, who rated himself as having a strong statistical background and who developed his own models (based on Sabermetrics), said “I buy a magazine and check the top rated players at each position. I also upload the previous seasons’ stats into an Access database and play around with these figures trying to identify trends that are not normally in a 5x5 league that will help my draft choices.”

Almost half of the participants, 47.5% (76/160), described looking at news and data, including other websites’ player statistics. In other words, they consulted statistics, but they didn’t gather them, nor attempt to perform systematic analysis. Instead they relied on list management, in tandem with their own gestalt of the information, and experts’ interpretations of the numbers. E.g., P044 reported, “I focus on their statistics and how well they played in past games/seasons. I also watch sportscenter and other tv shows and make decisions based on the opinions of the announcers and hosts.” This type of participant may be amenable to applying more sophisticated analytic techniques.

Some participants—those we coded as relying on intuition and favorites—described gameplay habits like those of P080, “I never simulated a thing, I just played for fun, pretending to be a coach from the old days when I was growing up. Trying to rely on instinct and not simulations and data analysis like happens now in sports. For me that stuff ruins the fun of the game.” 22 (13.8%) participants were coded in this “Intuition/favorites” category. The same number (22, 13.8%) relied on expert advice or recommendations from friends. The availability of news may help these two categories of players move into a more information-based choice process.

We added a transitional category (“4. Data resources”) as we performed the bottom-up coding. 29 participants (18.1%) used data and seemed to do extensive list management and data comparison (sometimes using in-game tools, sometimes in Excel, and sometimes on paper), but didn’t have confidence in their understanding of statistics. E.g. P052 reported, “I decide which stats I want to win in (pts, rbds, stls, 3pt, etc.) and then select players that do well in those categories then place role players which excel in 2-3 categories around my keeper players.” However, P052 also rated himself as a stats novice, a beginning fantasy sports player, and he appeared to use relatively few player characteristics and in-game tools to judge his team choices. It seemed necessary to distinguish these participants from those who used statistics to perform complex modeling and prediction.

Resources for gameplay choices	#	%
1. Intuition/favorites	22	13.8%
2. Advice/secondary sources	22	13.8%
3. Primary news sources	76	47.5%
4. Data resources	29	18.1%
5. Statistical resources	11	6.9%
Total	160	100%

The study results reveal some movement and malleability between modes of data and news analysis as participants gain more familiarity with the domain, their interpretations become deeper, and in-game tools become more readily available. These results suggest a design strategy in which prediction games help move learners into more information- and data-based techniques over time. For users of secondary sources (like expert advice), exposure to news may lead them into learning the value of primary sources. If they already consult statistics and manipulate ranked lists of players using individual player characteristics, users may be ready to learn other statistical tools (as long as we can provide the scaffolding necessary to understand these tools).

Our initial evaluation focuses on modes of providing news and primary sources because this seems to be where we can intercept players at a realistic level; almost half of the study participants sought out primary sources to do their own research. Prediction games need to facilitate use at a range of levels of analytic sophistication: almost all players are open to reading news and taking expert advice (or advice from their peers); some are also going to be able to take advantage of quantitative resources (e.g. by using data visualizations to identify patterns).

willingness to apply them, but most players kept informed about relevant events like game results, player injuries, and standout performances. Hence, Fantasy Climate provides four ways players can get up-to-date information and analyze data: a list view of news items (NewsBoard), a geospatial presentation of news items (GeoNews), a temperature-based visualization tool (ThermoVizz), and a map displaying current high and low temperatures. The two competing interfaces for presenting climate and weather-related news, GeoNews and NewsBoard, the focus of our evaluation in this paper, rely on the same underlying processing mechanisms.

Fantasy Climate gathers prospective articles from news services, and caches them and their metadata (including the URL, date published, summary, title, and author) in a local database. Stanford’s Named Entity Recognizer extracts locations mentioned in each article; frequently-mentioned locations are assumed to be the article’s geographic focus. If the news source doesn’t provide a summary, one is generated using a text summarization engine.

3. FANTASY CLIMATE

Fantasy Climate is a prediction game that encourages players to look for long-term changes in weather data [5]. During each scoring period, players make two selections from a list of U.S. cities: one location that will be hotter than the historic norm and a second location that will be cooler. These selections are scored according to a preset formula that uses the observed difference between the daily high/low temperature for the prediction date and the historical average for that day.

Fantasy Climate integrates core prediction-making functionality (including a means of tracking due dates and current and past scores) with player-toplayer and player-to-league communication venues (to address the game’s social aspect). The survey revealed that players had varying familiarity with statistical techniques and

The cache of articles is updated periodically to keep the news current.

GeoNews (Fig. 1) groups news articles by their focal locations, and represents them on a US map using a marker. Clicking on the marker shows a list of titles and summaries for the most recent articles at that location; links provide access to full versions. Filters allow users to specify the time period and location for the news displayed. Because weather discussions often refer to larger geographic regions, players can toggle between seeing all available news and news salient to the cities in the current prediction activity.

NewsBoard (Fig. 2) presents news articles as a list, most recent first, with the item summaries visible by default. Again, filters allow players to limit the time period of the news display, and to focus on locations and topics salient to the current activity.

4. EVALUATION

The popularity of fantasy sports owes in part to widespread interest in sports. The larger question of Fantasy Climate is, can this type of prediction game be as engaging in other non-sports domains, particularly in an educational setting where alternative presentations may be less interactive. Exploring the effect of important elements of fantasy sports such as competition, social interaction, news presentation, and data analysis support (e.g. visualization) individually will provide guidance in the design of prediction games. As a first step, we report on an evaluation of Fantasy Climate’s competing domain news interfaces, since our survey showed news to be central to prediction-making.

4.1 Experimental Setup

29 participants, ages between 18 and 27, with no prior knowledge of the prediction games project were recruited via word of mouth from the university community. Most of the participants were students, which reflects the intended audience of Fantasy Climate. Participants were randomly assigned to two independent leagues with identical prediction activities except for the news presentation interface. The two leagues were run concurrently with the same subsets of locations available in each round, so the prediction tasks would be comparable. Participants used a video tutorial to familiarize themselves with the game. They accessed the game via a web browser on their own computers.

The overall activity lasted 14 days and included three prediction rounds. The first two days were a pre-activity stage to let the players familiarize themselves with the game goals and interface. During the next 9 days, a prediction submission was due every three days. Each round included 7 locations to learn about and select from, for a total of 21 unique U.S. locations. Each prediction was for the day after the submission due date, which was also when predictions were scored. The final two days of the activity allowed participants to review their scores and to socialize.

For each round, players were asked to make two predictions: the warmest location compared to historic norms and the coolest location compared to historic norms. The following formula was used to score the players’ predictions:

Warming location score = 50 + Observed high – Historical avg. high Cooling location score = 50 + Historical avg. low – Observed low

Participants were also provided with 70 years of historical weather data (Weather Underground’s dataset of average high/low temperatures), which they could visualize using Thermovizz.

4.2 Results and Discussion

The two in-game news presentations were compared using player feedback, interaction telemetry (e.g. session length), and an analysis of prediction quality in the two leagues.

Did the two news interfaces influence player perceptions? Participants in both groups reacted positively to the availability of in-game weather and climate news. 71% in the GeoNews league (Group 1) and 66% in the NewsBoard league (Group 2) agreed with the statement, “Exploring the news articles kept you engrossed in the game.” Similarly, 78% in GeoNews users and 73% of NewsBoard users agreed with the statement “Exploring the news articles was part of what made the game fun.” Thus, there was little difference in the two groups’ reactions to news availability.

Perceptions differed between the groups when they were asked about their engagement with the game as a whole. More than half (56%) of GeoNews league members agreed that they played Fantasy Climate longer than they meant to compared with only one-fifth (20%) of NewsBoard league members. Similarly, when asked whether they really got involved in the game, 85% of those in the GeoNews group agreed compared to only 53% of those in the NewsBoard group.

How did the two news interfaces affect time spent with news and with the game? We collected interaction data using Google Web Analytics [6, 14, 20], and used the following metrics to assess engagement: page views, number of sessions, average session duration, and average time spent. The results across all of these metrics indicate a consistent pattern at the overall game level and at the news interface level. First, the GeoNews group accessed more pages with greater frequency than the NewsBoard group. The GeoNews group had 32% more unique page views overall, 23% more web sessions, and 68% more unique news page views.

But page view counts can be inflated if navigational cues are poor: users may visit more pages if they don’t find what they are looking for. Session duration and time spent in the game are more direct indicators of player engagement.

Not only did the GeoNews group visit the game more often, but its members spent more time per visit. The average session duration of GeoNews players was 9 minutes and 29 seconds compared to 6 minutes and 28 seconds for the NewsBoard players. Players in the GeoNews group were active in the game for more than 115 minutes across the two weeks, compared with slightly less than 60 minutes of activity for players in the NewsBoard group (Fig. 3, right). GeoNews users logged over 34 minutes using the news interface compared to less than 12 minutes for those using NewsBoard (Fig. 3, middle). The GeoNews group spent an average of 4 minutes and 44 seconds per individual news page view compared to 3 minutes and 15 seconds for NewsBoard (Fig. 3, left).

page view (in minutes)

GeoNews users logged over 34 minutes using the news interface compared to less than 12 minutes for those using NewsBoard (Fig. 3, middle). The GeoNews group spent an average of 4 minutes and 44 seconds per individual news page view compared to 3 minutes and 15 seconds for NewsBoard (Fig. 3, left).

In general, the analytics indicate that GeoNews participants interacted more often and longer with Fantasy Climate than NewsBoard participants did. This was also true when considering the news interfaces alone. We could potentially interpret the difference as players using external sources to find additional information in the NewsBoard case, but not all of the extra time spent over the course of the league was in the news interface. Only about 22 of the 55 minutes (on average) over the activity was spent in the news interface.

Does the news interface influence the quality of player predictions? The above results provide evidence that the news presentation influences player engagement with Fantasy Climate. Does the extra time players spend result in better predictions?

To compute the players’ performance in each prediction round, we ranked the round’s 7 cities based on the deviation of their high/low temperature from the historic average. Each city had two ranks, one for warming, the other for cooling. Rank 1 meant that the city was the best of the cities available for selection; Rank 7 was the worst. We calculated players’ performance in a prediction round by the average rank of their selected cities. Since rank 1 is best, a lower score signifies better predictions. Over the game’s three rounds, the average performance of the players in the GeoNews league was better (average rank 3.28) than the performance of the players in the NewsBoard league (average rank

shows better predictions for GeoNews users than

NewsBoard users.

shows improving performance with GeoNews.

3.96). This difference is significant p=0.015 (2 tailed t-test).

Figure 4. Average rank per prediction (lower is better)

Figure 4 shows this result held for predicting warming and cooling locations. This figure also reveals that the warm predictions were more accurate than the cool predictions for both groups and there was a relatively consistent gap between performance on warming predictions and cooling predictions. The reason may be that cooling predictions are more difficult to make because low temperatures are rising. We expect our players to develop this type of insight as they play Fantasy Climate over a period of time.

Finally, how do player predictions change over the course of an activity? Our hope is that, by engaging with data and news, players will develop a better mental model over time;

Figure 5. Average rank per round (decrease is desirable)

i.e., their predictions should improve. Unfortunately, Figure 5 shows this is not always the case. The predictions from the two groups were similar for the activity’s first round. But, while the GeoNews group’s predictions improved over time, the NewsBoard group’s predictions worsened. A possibility in activities like this is that early engagement due to novelty will subside, resulting in decreased prediction quality. Features that increase engagement, including effective news presentations, are meant to reduce such drop-offs in effort.

In summary, our experimental results indicate that news presentation influenced players' performance in Fantasy Climate. The results suggest that not only the information available, but also its presentation, helped players make better predictions over time. From the web analytics, we observed that news presentation increased players' activity in Fantasy Climate. Participants in the GeoNews group visited the game and its news interface more often and longer than their NewsBoard counterparts. Taken together, it appears that the GeoNews interface engaged players more and the extra time they spent in the game improved the participants’ understanding of the domain, resulting in better predictions.

Overall, prediction games have the potential to encourage domain and data analysis skills and learning, especially in a pedagogical setting, where many alternatives are less interactive. Unlike fantasy sports, prediction games in other domains cannot rely on players’ inherent interest. Gamification can provide motivation to players, but other features of gaming environments play an important role in keeping players interested and active. Here we have identified domain-related news as an important component of prediction games and have evaluated its effects.

ACKNOWLEDGMENTS

This material is based upon work supported by the National Science Foundation under Grant No. 1816923.

Figure 1. The GeoNews interface provides spatialized links to news

Figure 2. The NewsBoard interface displays a chronological list view of news

Figure 3. Average time spent on the game, on news, and per

REFERENCES

[1] Josh Blackman, Adam Aft, and Corey Carpenter. 2012. FantasySCOTUS:

Crowdsourcing a Prediction Market for the Supreme Court. NorthWestern Journal of Technology and Intellectual Property. 10, (2012), 125.

[2] Brendan Dwyer, Stephen L. Shapiro, and Joris Drayer. 2011. Segmenting

Motivation: An Analysis of Fantasy Baseball Motives and Mediated Sport Consumption. Sport Marketing Quarterly. 20, 3 (2011), 129–137.

[3] Brendan Dwyer and Yongjae Kim. 2011. For Love or Money: Developing and

Validating a Motivational Scale for Fantasy Football Participation. Journal of Sport Management. 25, 1 (Jan. 2011), 70–83.

[4] Gabriel S. Dzodom and Frank M. Shipman. 2014. Data-driven web entertainment:

the data collection and analysis practices of fantasy sports players. In Proceedings of the 2014 ACM conference on Web science (WebSci ’14). Association for Computing Machinery, New York, NY, USA, 293–294.

[5] Gabriel Dzodom and Frank Shipman. 2016. Data-driven Prediction Games. In

Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’16). Association for Computing Machinery, New York, NY, USA, 1857–1864.

[6] Wei Fang. 2007. Using Google Analytics for Improving Library Website Content

and Design: A Case Study. Library Philosophy and Practice. 2, 9 (2007).

[7] Lee K. Farquhar and Robert Meeds. 2007. Types of Fantasy Sports Users and

Their Motivations. Journal of Computer-Mediated Communication. 12, 4 (Jul. 2007), 1208–1228.

[8] Rosemary Garris, Robert Ahlers, and James Driskell. 2002. Games, Motivation,

and Learning: A Research and Practice Model. Simulation & Gaming. 33, 4 (Dec. 2002), 441–467.

[9] Jacob Habgood and Shaaron Ainsworth. 2011. Motivating Children to Learn

Effectively: Exploring the Value of Intrinsic Integration in Educational Games. Journal of the Learning Sciences. 20, 2 (2011), 169–206.

[10] Sandra Hirsh, Christine Anderson, and Matthew Caselli. 2012. The reality of

fantasy: uncovering information-seeking behaviors and needs in online fantasy

sports. In CHI ’12 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’12). Association for Computing Machinery, New York, NY, USA, 849– 864.

[11] Young Ik Suh, Lim Choonghoon, Dae Hee Kwak, and Paul M. Pedersen. 2010.

Examining the psychological factors associated with involvement in fantasy sports: An analysis of participants’ motivations and constraints. International Journal of Sport Management, Recreation and Tourism. 5, (Jun. 2010), 1–28.

[12] Seunghwan Lee, Won Jae Seo, and Christine Green. 2013. Understanding why

people play fantasy sport: development of the Fantasy Sport Motivation Inventory (FanSMI). European Sport Management Quarterly. 13, 2 (2013), 166–199.

[13] Nielsen Company. Fantasy is Reality: A Look at the Growing Engagement in

Fantasy Sports. https://www.nielsen.com/us/en/insights/article/2018/fantasy-is- reality-a-look-at-growing-engagement-in-fantasy-sports/. Accessed: 2020-06-08.

[14] Beatriz Plaza. 2011. Google Analytics for measuring website performance.

Tourism Management. 32, 3 (Jun. 2011), 477–481.

[15] Marc Prensky. 2005. Digital game-based learning. Handbook of Computer Game

Studies. MIT Press. 97–122.

[16] Brody J. Ruihley and Andrew C. Billings. 2013. Infiltrating the boys’ club:

Motivations for women’s fantasy sport participation. International Review for the Sociology of Sport. 48, 4 (Aug. 2013), 435–452.

[17] Brody J. Ruihley and Robin Hardin. 2013. Meeting the informational needs of the

fantasy sport user. Journal of Sports Media. 8, 2 (2013), 53–80.

[18] Frank M. Shipman. 2009. Blending the real and virtual in games: the model of

fantasy sports. In Proceedings of the 4th International Conference on Foundations of Digital Games (FDG ’09). Association for Computing Machinery, New York, NY, USA, 169–174.

[19] Brian K. Smith, Priya Sharma, and Paula Hooper. 2006. Decision making in

online fantasy sports communities. Interactive Technology and Smart Education. 3, 4 (2006), 347–360.

[20] Katrien Verbert, Erik Duval, Joris Klerkx, Sten Govaerts, and Jose Luis Santos.

Learning Analytics Dashboard Applications. American Behavioral Scientist.

(Feb. 2013), 2764213479363.

[21] Donghee Yvette Wohn, Emma J. Freeman, and Katherine J. Quehl. 2017. A Game

of Research: Information Management and Decision-making in Daily Fantasy Sports. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY ’17). Association for Computing Machinery, New York, NY, USA, 355–366.

---

This full-text Seed edition preserves attribution to the ACM Hypertext 2020 version of record. Formatting was converted from that version under the supplied ACM authorization.