Abstract
Recent advances on the web have made social media a significant platform for disseminating scientific information. However, posts referencing peer-reviewed research often lack sufficient context for non-experts to assess their credibility. In this study, we investigate how different strategies for enriching social media posts referencing scientific publications affect users’ trust perceptions and sharing behavior. We developed a web-based platform that simulates a social media feed containing posts with links to scientific publications and conducted a user study (N=160), comparing four conditions: a baseline with original posts, and three enriched variants containing (1) metadata from the publication (title, abstract, authors), (2) a direct quote from the publication, and (3) an AI-generated summary of the publication. Our results show that enriched posts were shared more frequently than baseline posts, though trustworthiness ratings did not significantly differ. Furthermore, the AI-generated summaries were perceived as the most understandable form of enrichment. Interaction data showed that users were more likely to engage with the enriched content than with posts containing only links to the publications.
Introduction
1
In recent years, social media platforms have evolved beyond their original function as social hubs to become prominent channels for the dissemination of scientific information [13,27]. Approximately one-third of scientific publications are shared on social media [17], and over half of these shared publications receive some form of user interaction, such as likes, reposts, or replies [18]. Notably, about half of the posts linking to or citing scientific publications are not posted by the publication authors themselves, but by others without an academic background [43]. In such cases, scientific information is often reduced to a hyperlink or a sensationalized headline, lacking the necessary context to support informed interpretation. While these posts can help raise public awareness of scientific developments, they frequently fall short in enabling non-experts to assess the credibility of the information presented [29]. This poses a significant challenge. When posts citing peer-reviewed research provide minimal content, users may find it difficult to judge their trustworthiness, which could lead to either unwarranted skepticism or misplaced trust. Ensuring trust in science communication is especially crucial in today’s information landscape, where misinformation spreads rapidly [40], and the understanding of science plays a vital role in public opinion formation [14,36].
In our research, we examine how enriching social media posts citing scientific publications with additional content influences users’ perceptions of trust. Specifically, we investigate three strategies for enhancing posts that refer to scientific publications: (1) adding the original title, abstract, and author names (Metadata), (2) presenting a direct quote from the publication that supports the content of the post (Quote), and (3) replacing the abstract with a simplified, AI-generated summary (AIsummary).
To assess the impact of these strategies, we conducted a user study evaluating how each type of additional information affects users’ perceived trustworthiness and sharing behavior regarding scientific social media posts.
Our findings show that participants were more likely to share posts with scientific content when they were enriched with AI-generated summaries or metadata, and that AI-generated summaries were rated as significantly easier to understand than metadata. We also found a strong correlation between perceived trustworthiness and sharing behavior. Interestingly, interaction data showed that users often shared scientific content without accessing the original publication. However, when additional information was provided, it tended to generate more interactions with the post, suggesting that such information influences whether the post is shared.
Related Work
2
In this section, we review prior work on communicating scientific information on social media and factors influencing user trust in such content. We first examine how scientific content is shared and perceived on social media platforms. We then discuss the role of trust in such environments, before turning to recent advances in text simplification and summarization techniques aimed at improving the accessibility of scientific texts for non-expert audiences. At the end of this section, review approaches to building trust in social media platforms.
Engagement With Scientific Content on Social Media
2.1
In recent years, social media platforms have become increasingly significant for the dissemination of scientific information [13,27]. As a result, research has intensified around how scientific research is shared, consumed, and engaged with in these environments [1]. Studies have shown that scientists frequently use social media to promote their scholarly work, and that there is a positive correlation between social media activity around a publication and its citation count [27,33]. One common practice among scientists is embedding URLs to their publications in social media posts. However, these links are often shortened, providing limited cues about the actual content of the linked page, as well as being accessible temporarily [4]. Fang et al. [16] analyzed the click-through rates of shortened links to scientific publications and found that approximately 50% of these links are never clicked. A similar pattern was observed by Sundar et al. [38] in the context of news sharing on Facebook, where users share most of the posts with news article links (75%) without actually clicking on the links. Additionally, Holmström et al. [22] found a correlation between the number of clicks on links to scientific content and social media engagement metrics such as likes and retweets, meaning that users engage more with the posts when they click more on their embedded links. The way scientific information is framed also plays a critical role in how it spreads. Milkman and Berger [31] found that summaries of scientific findings written in a more interesting and emotionally engaging way are more likely to be shared. Furthermore, Bowman [8] noted that scholars are more likely to include URLs in their professional tweets than in personal ones, reinforcing the role of audience and intent in science communication on social media.
Trust in Scientific Information on Social Media
2.2
Trustworthiness refers to the degree of confidence in a source’s "intent to communicate the assertions they consider the most valid and true" [23]. However, trust in the context of social media remains poorly defined in the literature. Existing research often broadly addresses trust, focusing on general information, social media users, and the platforms themselves [20]. When trust is discussed specifically in relation to information, its specific dimensions, such as reliability, accuracy, and trustworthiness, are usually investigated [44]. In addition, the absence of standardized quality control mechanisms on social media raises concerns about the credibility of scientific information [41]. To address this, Imran and Ahmad [24] proposed an information credibility framework, which incorporates credibility indicators designed to improve trust in content shared on social media platforms.
Empirical research has investigated how to enhance the credibility of social media posts communicating scientific findings. For example, in a user study conducted by Boothby et al. [7], participants evaluated the credibility of science content on Twitter compared to the same content presented through other media. The study also explored how different features of tweets, such as the addition of visual elements (e.g., abstract screenshots, charts), text sentiment (neutral vs. emotional tone), and engagement metrics (likes/retweets), affected perceived credibility. Results showed that scientific content on Twitter was generally perceived as less credible than on other platforms. However, credibility increased when tweets included visuals, used neutral or positive language, or had high levels of engagement. Similarly, a recent user study by Millet et al. [32] investigated the effect of source credibility and social endorsement (i.e., likes, retweets) on perceived credibility of posts with scientific news onX(formerly Twitter). They found that the credibility of the source had a significant effect on users’ perceptions of the credibility of the post, while social endorsement had no measurable effect on credibility.
Simplification and Summarization of Scientific Texts
2.3
Non-expert readers often struggle to understand scientific texts due to their inherent complexity, characterized by domain-specific jargon, abbreviations, and technical language [26]. Text simplification aims to address these challenges by reducing linguistic complexity (e.g., vocabulary difficulty and sentence structure) while preserving the original meaning of the content [2]. This task has become increasingly important in light of the exponential growth in scientific publications [19]. A recent user study demonstrated that text simplification significantly improves the comprehensibility of technical, particularly medical, texts for general audiences [25]. In recognition of this challenge, dedicated evaluation initiatives, such as shared tasks at CLEF, have emerged to support scientific text simplification [15]. These efforts frequently employ large language models (LLMs), which have shown notable success in simplifying scientific discourse.
In addition to simplification, summarization also plays an important role in helping readers comprehend scientific publications [3,37]. Scientific document summarization models aim to condense entire documents or abstracts into concise summaries [11,30,39]. Some systems go further by generating summaries of multiple documents, integrating information from several related publications [28]. More recently, research has focused on creating simplified summaries of scientific documents to improve accessibility for broader audiences [42]. These models aim to optimize both content fidelity and readability, evaluated across dimensions such as grammar, coherence, consistency, fluency, and simplicity.
Considering that both simplification and summarization tasks increasingly rely on generative AI, questions have emerged regarding their perceived trustworthiness. In a user study by Cabrero-Daniel and Sanagustín Cabrero [10], participants were found to struggle to distinguish between machine-generated and human-written summaries. While participants generally acknowledged the limitations and potential ethical concerns surrounding generative AI, they also demonstrated a degree of trust in its outputs, especially when the content appeared fluent and coherent.
Designing for Trust in Online Information
2.4
Both platform providers and researchers have acknowledged the challenges surrounding trust in online information and have sought to address them through design interventions. For example, a user study by Henke et al. [21] demonstrated that the inclusion of scientific sources, statistical data, and their visualization in online news articles increases perceived credibility. In recent years, many news organizations have begun incorporating these trust-enhancing elements into their article layouts, through footnotes, authors’ notes, or dedicated ’transparency boxes’ to provide readers with insight into the editorial processes and the sourcing of information [34].
Summary
2.5
Prior research has highlighted the challenges of communicating scientific information on social media, particularly in terms of accessibility, credibility, and user trust. While simplification and summarization techniques have emerged as promising tools for making complex information more understandable, trust in AI-generated content and online sources remains a critical concern. Building on these insights, our study explores how different strategies for enriching scientific posts can influence users’ trust perceptions and sharing behavior on social media. We conducted our user study on a simulated social media platform to ensure that the results were not dependent on a specific platform or influenced by certain credibility indicators such as user profiles and engagement metrics (e.g., likes or shares).
Enhancing scientific post content with additional information
3
Due to the nature of social media platforms, scientific posts are typically user-generated. These posts often provide little contextual information about the cited publications and, at most, contain a hyperlink. The typical structure of such a post is illustrated in Figure1.
Illustration of a post referring to a scientific publication with a shortened URL.
The hyperlinks provided are frequently shortened URLs, which can reduce their perceived accessibility [16]. Moreover, users rarely click on links before deciding to share a post [38]. As a result, trust and sharing behavior are often based primarily on the visible content of the post rather than the cited publication itself.
To address this challenge, we introduce three enrichment strategies designed to provide users with better contextual information about cited scientific publications. In all three approaches, an additional content box is appended below the original post. The strategies are as follows:
(1) Metadata: Displays key metadata of the cited publication, including its title, abstract, and author names (see Fig.2).
(2) Quote: Shows a single sentence directly excerpted from the cited publication that supports or corresponds to the post content (see Fig.3).
(3) AIsummary: Provides a concise, AI-generated summary of the cited publication in easy language, tailored to align with the topic of the post (see Fig.4).
Illustration of the concept Metadata. Title, Authors and Abstract (collapsed) are added to the post.
Illustration of the concept Quote. The relevant quote that either supports or contradicts the statement made in the post is extracted from the publication and added to it.
Illustration of the concept AIsummary. A short AI-generated summary of the publication is added to the post.
Each enrichment strategy is motivated by distinct assumptions about how users process and evaluate scientific information on social media. TheMetadatacondition draws inspiration from how publications are presented in academic digital libraries and repositories, where metadata supports informed evaluation. A similar approach is used in [7]. TheQuotecondition is intended to support rapid fact-checking. Including a direct quote from the source can help substantiate claims and provide users with an anchor for verifying the accuracy of the post content. TheAIsummarycondition addresses the gap between formal academic language and the potential lack of domain knowledge among general users. By providing a simplified summary in plain language, this approach aims to enhance comprehension and trust [25].
User Study
4
Our research is driven by the following research question:RQ:How does additional information influence users’ trust and sharing behavior regarding scientific content on social media?
To answer this research question, we conducted an online user study with 160 participants using a between-subjects design with the conditions:Baselinewithout any additional information,Metadata,Quote, andAIsummaryas introduced before.
Figure 6: The second page of the timeline. It additionally contains a trustworthiness question for each post and a reasoning question for the shared posts.
The first page of the timeline. It starts with a reminder of the scenario and it contains the posts with corresponding additional information.
Material
4.1
To compile a set of posts for our study, we utilized Altmetric1, a platform that assigns an attention score to scientific publications based on their online mentions, and categorizes them by topic. Using the Altmetric API, we identified the four most popular topics from the last three years: Biomedical and Clinical Sciences, Biological Sciences, Health Sciences, and Psychology. For each topic, we selected the two most popular publications, resulting in a total of eight publications. Details of these topics and selected publications are shown in Table1. We then usedX’s (formerly Twitter) advanced search to manually locate posts citing these publications by searching for their URLs. To ensure relevance and engagement, we applied the following inclusion criteria: posts had to (1) be written in English, (2) contain scientific content citing the linked publication, and (3) have received at least three replies. Additionally, we ensured that the replies did not contradict the main post, maintaining consistency with the cited publication.
From the search results, we selected the first post for each publication that met these criteria. This process yielded a final set of eight posts, which were used in our user study.
The two most popular publications (DOIs) from the four most popular scientific topics based on Altmetric scores as of September 2024.
Topic | Publication |
|---|---|
Biomedical and Clinical Sciences | "Physical interventions to interrupt or reduce the spread of respiratory viruses" (DOI: 10.1002/14651858.CD006207.pub6) |
"The Incidence of Myocarditis and Pericarditis in Post COVID-19 Unvaccinated Patients—A Large Population-Based Study" (DOI: 10.3390/jcm11082219) | |
Biological Sciences | "Intracellular Reverse Transcription of Pfizer BioNTech COVID-19 mRNA Vaccine BNT162b2 In Vitro in Human Liver Cell Line" (DOI: 10.3390/cimb44030073) |
"Long COVID: major findings, mechanisms and recommendations" (DOI:10.1038/s41579-022-00846-2) | |
Health Sciences | "Myocarditis Cases Reported After mRNA-Based COVID-19 Vaccination in the US From December 2020 to August 2021" (DOI: 10.1001/jama.2021.24110) |
"Cannabinoids Block Cellular Entry of SARS-CoV-2 and the Emerging Variants" (DOI: 10.1021/acs.jnatprod.1c00946) | |
Psychology | "SARS-CoV-2 is associated with changes in brain structure in UK Biobank" (DOI: 10.1038/s41586-022-04569-5) |
"The serotonin theory of depression: a systematic umbrella review of the evidence" (DOI: 10.1038/s41380-022-01661-0) |
Setup
4.2
We developed a web application featuring four distinct interfaces, each corresponding to one of the experimental conditions. Each interface presents a timeline that simulates a social media feed, displaying the same eight posts (in randomized order) but with different types of additional information depending on the assigned condition. To maintain poster anonymity and realism, all posts include fake usernames. Profile pictures were assigned using the 10k US Adult Faces Database2, which is annotated with various psychological attributes, including perceived trustworthiness [5]. We selected images with trustworthiness scores of 4, 5, or 6 (representing neutral values on the scale) to assign them to the posters’ profiles. Each post includes a collapsible comment section displaying three replies, as well as a prominent "Share" button. To prevent accidental engagement, the share button is reversible, ensuring that only participants’ final sharing choice is recorded.
As shown in Figure5, the first page of the application starts with instructions about the study scenario, which is further explained in Section4.3.
Before deciding whether to share each post, participants could review the eight posts by exploring the comments and interacting with the embedded links. Once they completed their review, they could proceed to the second page by clicking the "Next Page" button3at the bottom of the first page.
All interactions on the first page are logged to analyze user engagement. We track several interaction types, including clicks on URLs to assess whether participants attempted to access the original publication, and clicks on the "Comments" button to determine whether participants viewed replies before making a sharing decision. To measure engagement with the additional content, only a preview of the metadata, quote, or AI-generated summary is shown by default. Participants must click a "Read more" button to view the full content. These interactions allow us to observe whether participants actively engaged with the additional information provided under each post.
The second page, illustrated in Figure6, presents a similar timeline with the same posts, but this time each post is accompanied by a follow-up question to assess participants’ perceptions of trustworthiness. Participants were not allowed to change their sharing decisions on the second page.
We created the additional content in each condition as follows:
Metadata:We checked the publisher page to collect metadata, including authors, publication date, and abstract for each cited publication.
Quote:We manually searched each cited publication to find the relevant text snippet for the corresponding post content.
AIsummary:For each post, we provided the following prompt together with the post content and the link of the publisher page of the cited publications to ChatGPT (4o mini):"Kindly provide a brief (3-4 sentence) summary of the article on [link of the publisher page] in clear, straightforward language, highlighting the following content: [post content]"4.
Regardless of condition, each post was accompanied by the reference to the original publication to provide the correct scientific citation.
Procedure and Task
4.3
For the online study, we created a questionnaire using the SoSci Survey5platform. Before starting the study, we made sure that participants were using desktop devices, with an internal check by SoSci. If they were not, we asked them to switch to a desktop device. Participants first provided informed consent and completed a series of closed questions regarding their demographic background. After completing the pre-questionnaire, participants were directed to the user study application, where they were randomly assigned to one of the four interface conditions:Baseline, Metadata, Quote, andAIsummary. Each condition started with a scenario description designed to contextualize the task and simulate a realistic evaluation setting:
Imagine you are an active social media user, i.e., you use social media platforms in your daily life interactively and you like to share content with your social circle. Your social circle consists of many people who are interested in various topics, especially in Biomedical and Clinical Sciences, Biological Sciences, Health Sciences, and Psychology. Below you will find your timeline.
Participants then scrolled through the timeline consisting of the eight posts we selected, examining the posts and deciding whether to share a post or not.
After completing this task, participants were directed to the second page, which displays the same timeline and posts. Here, they were asked to rate the trustworthiness of each post. Finally, participants were redirected to the questionnaire, where they answered both open and closed questions regarding their experience with the timeline interface and interacting with the posts. To ensure data quality, we added an attention check question to the questionnaire.
Measurements
4.4
To evaluate the effects of our experimental conditions, we collected both behavioral and self-reported data.
Sharing Behavior: Participants were asked to decide whether they would share each post shown on the timeline. For each of the eight posts, we recorded their sharing decision (shared or not shared). Since the share button was reversible, we only considered the final state at the time of transition to the second page. We consider sharing behavior an indirect, observable measure of trust, based on the assumption that users are more likely to share posts they perceive as credible and trustworthy.
Perceived Trustworthiness: On the second page, participants rated their perceived trustworthiness of each post using a 5-point Likert scale, ranging from 1 (not at all trustworthy) to 5 (totally trustworthy). These subjective ratings provide a direct, self-reported assessment of trust and allow for comparison with behavioral patterns.
Interaction Types (Engagement Metrics): To better understand participants’ engagement with the posts and their additional content, we logged two types of user interactions on the first page:
Link Clicks:Clicks on the shortened URL provided in a post, indicating an attempt to access the original scientific publication.
Content Expansions:Clicks on the “Read more” button to view the full abstract (Metadata condition), quote (Quote condition), or AI-generated summary (AIsummary condition). These actions serve as indicators of engagement with the additional content.
Overall AssessmentsIn addition, we measured the perceived understandability ("How understandable did you find the additional information?") of the additional content types across conditions (excluding the Baseline) on a 5-point Likert scale. These questions aimed to assess participants’ general perceptions of the usefulness and clarity of the enrichment strategies.
Furthermore, because quantitative data alone may not fully capture participants’ reasoning and subjective experiences, we also included open-ended questions ("What further kind of information would make you trust the posts more?" and "What helped you decide whether or not to share the posts?") in the final questionnaire. These prompted participants to reflect on their preferences and experiences on the social media timeline, offering richer, individualized insights into their decision-making and trust evaluations.
Together, these measures enable us to analyze how our content enrichment strategies influence both perceived trust and actual engagement behavior, offering a multifaceted view of trust in scientific posts on social media.
Hypotheses
4.5
To address our research question, we formulated the following hypotheses:H1:Posts with enriched content (metadata, quote, or AI summary) will result in higher perceived trustworthiness than the baseline condition.H2:Sharing behavior will be more frequent in conditions with additional information compared to the baseline.H3:Sharing behavior correlates with perceived trustworthiness.H4:AI-generated summaries will be perceived as more understandable than other enrichment types.H5:Content expansions (clicking "read more") will be positively correlated with perceived trustworthiness and sharing behavior.
HypothesisH1andH2are informed by findings from prior research [6,12], which suggest that increasing transparency by providing more contextual information alongside a URL can enhance perceived trust in online content. Following this line of reasoning, we expect that adding contextual information will lead to greater trust in the scientific content and an increased likelihood of sharing (H3).
H4is based on the strong performance of large language models (LLMs) in various natural language processing tasks, including text simplification [9]. Given that our AI-generated summaries were explicitly prompted to be "easy to understand," we anticipate that participants will perceive these summaries as more understandable than the scientific language used in the abstract and direct quote conditions.
H5reflects the assumption that the opportunity to explore and engage with additional information can foster trust. When users actively expand content to learn more about the underlying scientific publication, this behavior may indicate deeper cognitive engagement and may be positively associated with both perceived trustworthiness and willingness to share.
Participants
4.6
We recruited participants via the online crowd-sourcing platform Prolific6. During the study, 10 participants either voluntarily withdrew or had their participation automatically terminated by Prolific after exceeding the time limit of 56 minutes. In these cases, Prolific automatically recruited replacements until the target sample size of 160 was reached. Participants were randomly and evenly assigned to one of our four conditions: (Baseline=40,Metadata=40,Quote=40, andAIsummary=40). To be eligible for the study, participants were required to be residents of either the UK or the USA, speak English as their primary language, and identify themselves as active social media users. Participation in the study took approximately 14 minutes, and each participant received a compensation of 2.25 GBP, corresponding to an hourly rate of 9.43 GBP.
Of our 160 participants, 83 identified themselves as male, 72 as female and 5 as non-binary. Participants ranged in age from 18 to 67 years (M=34, SD=11).
Results - Quantitative Analysis
5
This section outlines the data collected in our user study and our findings. We begin by detailing the results of our statistical analyses and evaluating our hypotheses. Following this, we present the qualitative insights derived from participants’ responses to the open-ended questions.
In order to assess statistical differences between conditions, we used two different tests: 1) the Chi-Square test for independence for categorical data and 2) the Kruskal-Wallis tests with post-hoc Dunn’s tests, applying the Bonferroni correction, for data that was not normally distributed.
Perceived Trustworthiness Across Conditions
5.1
To assess perceived trust, we used participants’ trustworthiness ratings provided on the second page of the timeline interface. For analysis, we calculated the average trustworthiness score per participant across all eight posts. Descriptive statistics for trust ratings across conditions are shown in Table2.
We conducted Kruskal-Wallis tests to compare conditions, given that the trustworthiness data did not meet the parametric assumption of normality. The results showed no statistically significant differences in any of the comparisons (p>0.05), indicating that the presence of additional information, regardless of type, did not significantly influence perceived trustworthiness. H1 is rejected.
Descriptive statistics for our dependent variables. Perceived trust and understandability were measured on a scale from 1 to 5 (low to high), while sharing behavior is indicated by the percentage of share decisions in each condition.
Dep. Var. | Perc. Trust | Sharing | Understandability |
|---|---|---|---|
Metric | Mean (SD) | Ratio | Mean (SD) |
Baseline | 2.82 (0.41) | 25.93% | - |
Metadata | 2.93 (0.36) | 32.81% | 3.05 (1.11) |
Quote | 2.64 (0.50) | 31.25% | 3.43 (1.06) |
AIsummary | 2.78 (0.35) | 34.38% | 3.75 (0.95) |
Sharing Behavior Across Conditions
5.2
To assess sharing behavior, we analyzed participants’ (binary) decision to share or not to share each post. We calculated the overall sharing rate (i.e., the percentage of posts shared) within each condition. The cumulative sharing rates were: Baseline 25.93%, Metadata 32.81%, Quote 31.25%, AIsummary 34.38% (see also Table2).
We conducted a Chi-square test of independence to determine whether the observed differences in sharing decisions across conditions were statistically significant. The results revealed statistically significant differences between theBaselineand both theMetadata(p<0.05), andAIsummary(p<0.05) conditions. However, no significant difference was found betweenBaselineandQuote. H2 is partially accepted.
Correlation Between Perceived Trustworthiness and Sharing Behavior
5.3
We calculated Pearson’s correlation coefficient to assess the correlation between trustworthiness ratings and sharing behavior within each condition. It yielded significant positive correlations (all p<0.01) of r=0.93 forBaseline, r=0.93 forMetadata, r=0.89 forQuote, and r=0.82 for theAIsummarycondition. H3 is accepted.
Understandability of Additional Information
5.4
To assess the perceived understandability of the enriched content, we analyzed participants’ responses from the post-task questionnaire. Participants in theMetadata,Quote, andAIsummaryconditions rated how understandable they found the additional information on a 5-point Likert scale (low to high).
A Kruskal-Wallis test revealed a statistically significant difference between conditions (p<0.05). Post-hoc Dunn’s tests showed thatAIsummarywas rated significantly more understandable thanMetadata(p<0.05), but no significant difference was found between other conditions (p>0.05). H4 is partially accepted.
Correlation Between Engagement and Trust and Sharing Behavior
5.5
We further analyzed users’ interaction with the additional information and other interactive features on the simulated social media timeline to gain a better understanding of participants’ engagement with scientific content.
We examined whether users tend to view or read the publications mentioned in the posts they shared. To do this, we analyzed the clicks on the shortened URLs of all shared posts. The observed URL click ratios for each condition were 10% forBaseline, 12% forMetadata), 10% forQuote, and 4% forAIsummary. These results indicate generally low URL click ratios for shared posts.
We also investigated the correlations between the frequency of clicks to expand the additional content and 1) perceived trustworthiness and 2) sharing behavior per condition. We found that content expansion and trustworthiness ratings are significantly positively correlated (r=0.72; p<0.05) in theAIsummarycondition. No other significant correlations were found for the remaining conditions. H5 is partially accepted.
Results - Qualitative Analysis
6
For our open-ended questions, one of the authors open-coded the participants’ answers.
Information That Would Increase Trust
6.1
Regarding the open question "What further kind of information would make you trust the posts more?" , several patterns emerged across experimental conditions: Participants in theQuote(mentions=10) andAIsummary(m=9) conditions requested "More sources" (either strengthening evidence or providing opposing viewpoints) more than twice as frequently as those in theBaseline(m=4) condition.
The desire for links or references to sources (despite participants being provided short URLs in all conditions) was most prominent in theBaselinecondition (m=10), with moderate frequency in theMetadata(m=5),Quote(m=7) conditions, but notably low frequency in theAIsummarycondition (m=2). Similarly, the need for more context (regarding authors, sources, methodology, and/or results) appeared least frequently in theAIsummarycondition (m=1), while occurring with similar frequency across the other three conditions (m=4 to 5). TheBaselinecondition, on the other hand, also had the highest frequency for the desire for more neutral, formal, or unbiased language (m=3), with no mentions in theQuotecondition.
Independent fact-checkers, whether organizations or individual experts, were requested most frequently in theMetadatacondition (m=7), followed byQuote(m=6),AIsummary(m=5), andBaseline(m=3), showing a clear descending pattern across conditions. A similar descending pattern emerged for requests for post/poster reliability or truthfulness scores/ratings, withMetadatashowing the highest frequency (m=4), followed byQuoteandAIsummary(m=2 each), andBaseline(m=1).
Factors Influencing Sharing Decisions
6.2
For the question "What helped you decide whether or not to share the posts?" , we observed the following patterns: Personal interest or perceived relevance for self and others was mentioned almost twice as often in theQuotecondition (m=17) compared to theAIsummaryandBaselineconditions (m=9 each), withMetadatafalling between these extremes (m=14).
Confirmation bias (i.e., congruence of post content with existing knowledge, beliefs, opinions, and life experience) was most prevalent in theAIsummarycondition (m=12), appearing twice as frequently as in theBaselinecondition (m=6) and more than twice as often as in theMetadata(m=5) andQuote(m=3) conditions.
Participants mentioned the posts’ language style and tone as a decision factor least in theMetadatacondition (m=2), compared to at least three times more mentions in the other conditions (m=6-8). Concern about preventing harm or misinformation spread, on the other hand, was highest in theMetadatacondition (m=4), with fewer mentions across the other three conditions (m=1-2).
Comments under the posts barely influenced sharing decisions in theAIsummarycondition (1 mention), but were more influential in all other conditions (m=6-8). Similarly, source credibility, reputation, and trustworthiness were mentioned approximately three times less frequently in theAIsummarycondition (m=2) than in theMetadata(m=6),Quote(m=7), andBaseline(m=6) conditions.
Perceived truthfulness or unbiasedness of posts (m=7-10), source checking practices (m=5-7), and intellectual humility (declining to share due to insufficient topic knowledge) (m=2-3) showed similar frequencies across conditions.
Discussion
7
The results show that adding additional information to social media posts containing scientific content can enhance users’ sharing behavior, depending on the type of information provided. Specifically, posts enriched with metadata and AI-generated summaries were shared more frequently than posts in the baseline condition. However, our findings imply that additional information does not influence perceived trustworthiness, which partially contradicts previous findings showing a significant impact of an abstract as a visual entity on the credibility of the corresponding post [7]. This suggests a more nuanced relationship between trust and sharing. While we observed a strong positive correlation between sharing behavior and perceived trustworthiness across all conditions, qualitative feedback revealed that users’ sharing decisions are also influenced by additional factors, such as personal interest and confirmation bias. This finding underscores that sharing behavior cannot be considered a direct or objective measure of perceived trust alone.
Despite the lack of significant differences in trust ratings across conditions, our findings highlight factors that may foster perceived trust in scientific content. In particular, we observed a positive correlation between content expansions and perceived trustworthiness in the AIsummary condition, suggesting that active engagement with enriched content may enhance trust. This insight is supported by the fact that the AI summary was perceived as the most understandable among the additional information types, aligning with previous work [25], and by the qualitative feedback implying that the AI summary seems to have enabled users to grasp the context of the publication better. Notably, providing metadata seems to have reduced the influence of the posts’ language style on participants’ sharing decisions, seemingly balancing biased-sounding posts with neutral information. Qualitative responses suggested that providing multiple sources, and verification by independent fact-checkers, could further enhance trust in scientific posts.
Across all conditions, we observed low click-through rates on URLs to the original publications (10% for Baseline, 12% for Metadata, 10% for Quote, and 4% for AIsummary). This supports previous findings by Fang et al. [16] that approximately half of shortened scientific links are never clicked. In our case, the rates are even lower. Notably, the AIsummary condition, despite showing the highest sharing rate, had the lowest URL click rate. Our qualitative insights suggest that users may feel less compelled to verify the source when a summary appears comprehensive. While this indicates that summaries can enhance understanding, it also presents a potential risk of over-trust. If simplified summaries omit nuance or contain errors, they could inadvertently contribute to misinformation, especially when users bypass the original source.
Our findings have several implications for the design of social media platforms and science communication strategies. By providing additional information to scientific posts, especially AI-generated summaries in easy-to-understand language, social media platforms can help users better understand and evaluate scientific content. To mitigate over-reliance on simplified content, summaries should be accurate and ideally linked to multiple sources, with expert verification where possible. Design interventions, such as fact-checking prompts, viewpoint diversity indicators, or attention cues (e.g., alerts or warnings), could further encourage critical engagement with the content [35].
While our study provides valuable insights into the effects of additional information on trust and sharing behavior, several limitations should be considered. First, the simulated timeline study may not fully reflect real-time social media behaviors. Second, users’ interest in the topic domains, especially given the prominence of COVID-19-related publications, may have influenced their responses. Third, while we measured content expansions (e.g., clicks to read full AI summaries), these interactions do not necessarily indicate comprehension, and future work could benefit from more in-depth assessments of understanding.
Conclusion & Future Work
8
In this study, we examined how the inclusion of additional contextual information influences users’ trust and sharing behavior with scientific content on social media. We conceptualized three types of additional information: (1) metadata from the publication (title, abstract, authors), (2) a supporting quote from the publication, and (3) an AI-generated summary. Our findings indicate that while metadata and AI-generated summaries significantly increase sharing behavior, they do not necessarily enhance perceived trustworthiness. Through qualitative analysis, we identified various additional factors, such as personal interest, confirmation bias, and the language style of the post, that influence trust and sharing decisions. Notably, the AI-generated summaries were perceived as the most understandable, suggesting their potential in making scientific content more accessible to non-experts.
In future research, we plan to address several limitations and expand the scope of this research. First, to better understand long-term effects, we will design a longitudinal study examining how enriched scientific posts influence users’ trust and engagement over time. Additionally, we plan to investigate the impact of providing multiple sources for comparison, including AI-recommended references, and to explore the potential of overtrust. Our current and future research contributes to the trustworthy communication of scientific information on social media.
[^fn1]: <sup>1</sup>https://www.altmetric.com/about-us/our-data/donut-and-altmetric-attention-score/
[^fn2]: <sup>2</sup>https://wilmabainbridge.com/facememorability2.html
[^fn3]: <sup>3</sup>The "Next Page" button becomes active after 30 seconds, ensuring participants have a certain amount of time to interact with the first page.
[^fn4]: <sup>4</sup>All publications were open access and we provided links to their source pages with full text.
[^fn5]: <sup>5</sup>https://www.soscisurvey.de/
[^fn6]: <sup>6</sup>https://www.prolific.com/
References
[1]EuanAdieandWilliamRoe.2013.Altmetric: enriching scholarly content with article-level discussion and metrics.Learned Publishing26,1(2013),11–17.
[2]Suha S.Al-ThanyyanandAqil M.Azmi.2021.Automated Text Simplification: A Survey.ACM Comput. Surv.54,2, Article43(March2021),36pages.10.1145/3442695
[3]Nouf IbrahimAltmamiandMohamed El BachirMenai.2022.Automatic summarization of scientific articles: A survey.Journal of King Saud University-Computer and Information Sciences34,4(2022),1011–1028.
[4]DemetrisAntoniades,IasonasPolakis,GeorgiosKontaxis,EliasAthanasopoulos,SotirisIoannidis,Evangelos P.Markatos, andThomasKaragiannis.2011.we.b: the web of short urls. InProceedings of the 20th International Conference on World Wide Web(Hyderabad, India)(WWW ’11).Association for Computing Machinery,New York, NY, USA,715–724.10.1145/1963405.1963505
[5]Wilma ABainbridge,PhillipIsola, andAudeOliva.2013.The intrinsic memorability of face photographs.Journal of Experimental Psychology: General142,4(2013),1323.
[6]GraemeBaxter,RitaMarcella, andAgnieszkaWalicka.2019.Scottish citizens’ perceptions of the credibility of online political “facts” in the “fake news” era: an exploratory study.Journal of documentation75,5(2019),1100–1123.
[7]ClaraBoothby,Dakota S.Murray,Anna PolovickWaggy,AndrewTsou, andCassidy R.Sugimoto.2020.Credibility of scientific information on social media: Variation by platform, genre and presence of formal credibility cues.Quantitative Science Studies2(2020),845–863.https://api.semanticscholar.org/CorpusID:238654370
[8]Timothy DBowman.2015.Differences in personal and professional tweets of scholars.Aslib Journal of Information Management67,3(2015),356–371.
[9]TomBrown,BenjaminMann,NickRyder,MelanieSubbiah,Jared DKaplan,PrafullaDhariwal,ArvindNeelakantan,PranavShyam,GirishSastry,AmandaAskell, et al.2020.Language models are few-shot learners.Advances in neural information processing systems33(2020),1877–1901.
[10]BeatrizCabrero-DanielandAndreaSanagustín Cabrero.2023.Perceived Trustworthiness of Natural Language Generators. InProceedings of the First International Symposium on Trustworthy Autonomous Systems(Edinburgh, United Kingdom)(TAS ’23).Association for Computing Machinery,New York, NY, USA, Article23, 9 pages.10.1145/3597512.3599715
[11]IsabelCachola,KyleLo,ArmanCohan, andDaniel SWeld.2020.TLDR: Extreme summarization of scientific documents.arXiv preprint arXiv:https://arXiv.org/abs/2004.15011(2020).
[12]ApoorvaChauhanandAmanda LeeHughes.2020.Trustworthiness perceptions of social media resources named after a crisis event.Proceedings of the ACM on Human-Computer Interaction4,CSCW1(2020),1–23.
[13]MonicaDéchène,KaleyLesperance,LisaZiernwald, andDorisHolzberger.2024.From research to retweets—exploring the role of educational twitter (X) communities in promoting science communication and evidence-based teaching.Education Sciences14,2(2024),196.
[14]John RDurant,Geoffrey AEvans, andGeoffrey PThomas.1989.The public understanding of science.Nature340,6228(1989),11–14.
[15]LianaErmakova,ValentinLaimé,HelenMcCombie, andJaapKamps.2024.Overview of the CLEF 2024 SimpleText task 3: Simplify scientific text. InWorking Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024).3147–3162.
[16]ZFang,RCostas,WTian, andXWang.2021.How is science clicked on Twitter? Click metrics for Bitly short links to scientific papers. (2021).
[17]ZhichaoFang,RodrigoCostas,WencanTian,XianwenWang, andPaulWouters.2020.An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics.Scientometrics124,3(2020),2519–2549.
[18]ZhichaoFang,RodrigoCostas, andPaulWouters.2022.User engagement with scholarly tweets of scientific papers: A large-scale and cross-disciplinary analysis.Scientometrics127,8(2022),4523–4546.
[19]SantoFortunato,Carl TBergstrom,KatyBörner,James AEvans,DirkHelbing,StašaMilojević,Alexander MPetersen,FilippoRadicchi,RobertaSinatra,BrianUzzi, et al.2018.Science of science.Science359,6379(2018),eaao0185.
[20]PeterHåkanssonandHopeWitmer.2015.Social media and trust: A systematic literature review.Journal of business and economics6,3(2015),517–524.
[21]JakobHenke,LauraLeissner, andWiebkeMöhring.2020.How can journalists promote news credibility? Effects of evidences on trust and credibility.Journalism Practice14,3(2020),299–318.
[22]JesperHolmström,DanielJonsson,FilipPolbratt,OlavNilsson,LinneaLundström,SebastianRagnarsson,AntonForsberg,KarlAndersson, andNiklasCarlsson.2019.Do we Read what we Share? Analyzing the Click Dynamic of News Articles Shared on Twitter. In2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).420–425.10.1145/3341161.3342933
[23]Carl IverHovland,Irving LesterJanis, andHarold HKelley.1953.Communication and persuasion.(1953).
[24]MuhammadImranandAdnanAhmad.2021.Enhancing data quality to mine credible patterns.Journal of Information Science49(2021),544 – 564.https://api.semanticscholar.org/CorpusID:236259276
[25]OksanaIvchenkoandNataliaGrabar.2022.Impact of the text simplification on understanding. InChallenges of Trustable AI and Added-Value on Health.IOS Press,634–638.
[26]Loene M Howes K PaulKirkbrideandSally FKelty.2013.The Readability of Expert Reports for Non-Scientist.Forensic Science International230(2013),8–15.
[27]SamaraKlar,YannaKrupnikov,John BarryRyan,KathleenSearles, andYotamShmargad.2020.Using social media to promote academic research: Identifying the benefits of twitter for sharing academic work.PloS one15,4(2020),e0229446.
[28]YaoLu,YueDong, andLaurentCharlin.2020.Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles.arXiv preprint arXiv:https://arXiv.org/abs/2010.14235(2020).
[29]TeunLucassen,RiencoMuilwijk,Matthijs LNoordzij, andJan MaartenSchraagen.2013.Topic familiarity and information skills in online credibility evaluation.Journal of the American Society for Information Science and Technology64,2(2013),254–264.
[30]YuningMao,MingZhong, andJiaweiHan.2022.CiteSum: Citation text-guided scientific extreme summarization and domain adaptation with limited supervision.arXiv preprint arXiv:https://arXiv.org/abs/2205.06207(2022).
[31]Katherine LMilkmanandJonahBerger.2014.The science of sharing and the sharing of science.Proceedings of the National Academy of Sciences111,supplement_4(2014),13642–13649.
[32]BarbaraMillet,JiajingTang,MichelleSeelig,JohnPetit, andRuoyuSun.2024.In Twitter we trust(ed): How perceptions of Twitter’s helpfulness influence news post credibility perceptions and news engagement.Computers in Human Behavior155(2024),108185.10.1016/j.chb.2024.108185
[33]YaseminÖzkent.2022.Social media usage to share information in communication journals: An analysis of social media activity and article citations.Plos one17,2(2022),e0263725.
[34]Jason TPeiferandJaredMeisinger.2021.The value of explaining the process: How journalistic transparency and perceptions of news media importance can (sometimes) foster message credibility and engagement intentions.Journalism & Mass Communication Quarterly98,3(2021),828–853.
[35]AlisaRieger,TimDraws,MariëtTheune, andNavaTintarev.2021.This Item Might Reinforce Your Opinion: Obfuscation and Labeling of Search Results to Mitigate Confirmation Bias. InProceedings of the 32nd ACM Conference on Hypertext and Social Media(Virtual Event, USA)(HT ’21).Association for Computing Machinery,New York, NY, USA,189–199.10.1145/3465336.3475101
[36]Stephen GSapp,Peter FKorsching,CharlieArnot, andJannette JHWilson.2013.Science communication and the rationality of public opinion formation.Science Communication35,6(2013),734–757.
[37]GrishmaSharmaandDeepakSharma.2022.Automatic text summarization methods: A comprehensive review.SN Computer Science4,1(2022),33.
[38]S ShyamSundar,Eugene ChoSnyder,MengqiLiao,JunjunYin,JinpingWang, andGuangqingChi.2024.Sharing without clicking on news in social media.Nature Human Behaviour(2024),1–13.
[39]SotaroTakeshita,TommasoGreen,NiklasFriedrich,KaiEckert, andSimone PaoloPonzetto.2022.X-scitldr: cross-lingual extreme summarization of scholarly documents. InProceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries.1–12.
[40]SoroushVosoughi,DebRoy, andSinanAral.2018.The spread of true and false news online.science359,6380(2018),1146–1151.
[41]PeterWeingartandLarsGuenther.2016.Science communication and the issue of trust.Journal of Science communication15,5(2016),C01.
[42]FarooqZaman,FaisalKamiran,MatthewShardlow,Saeed-UlHassan,AsimKarim, andNaif RadiAljohani.2024.SATS: simplification aware text summarization of scientific documents.Frontiers in Artificial Intelligence7(2024),1375419.
[43]LinZhang,ZhenyuGou,ZhichaoFang,GunnarSivertsen, andYingHuang.2023.Who tweets scientific publications? A large-scale study of tweeting audiences in all areas of research.Journal of the Association for Information Science and Technology74,13(2023),1485–1497.
[44]YixuanZhang,Joseph DGaggiano,NutchanonYongsatianchot,Nurul MSuhaimi,MisoKim,YifanSun,JacquelineGriffin, andAndrea GParker.2023.What Do We Mean When We Talk about Trust in Social Media? A Systematic Review. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems(CHI ’23).ACM,1–22.10.1145/3544548.3581019
Do you like what you are reading? Subscribe to receive updates.
Unsubscribe anytime