With their widespread popularity, social media platforms are particularly vulnerable to the spread of disinformation, misinformation, and conspiracy theories, collectively referred as problematic information [14]. Several studies have named YouTube as one of the major platforms prone to problematic information. For example, [26] describes YouTube as a favorite source for prominent online spreaders of disinformation on Twitter. Research suggests that individuals who use YouTube for news are more likely to believe conspiracy theories [36] and less likely to be fully vaccinated against COVID-19 [5]. [3] found that YouTube users who believe in conspiracy theories related to the alleged 2020 US presidential election fraud had videos endorsing such claims in their recommended videos.

Past research has tracked the prevalence of conspiratorial videos after a major YouTube policy was implemented to remove content from the platform [6], understanding disinformation campaigns on YouTube [12], and examining how user activity differs for conspiratorial content [25]. Furthermore, researchers have focused on misinformation present for well-known topics like climate change [1], COVID-19 [5, 22, 26], and the 2020 US presidential election [3].

The listed examples indicate that research on problematic information on YouTube has focused on a few popular topics that capture everyone's attention. In this study, our goal is to go beyond popular topics, taking a human-centered approach, and looking at all conspiracy theory topics people show an interest in, both popular and obscure, to determine the extent to which YouTube has implemented IPs to address them. While we agree that some conspiracy theories are more harmful than others, when taken all together, they contribute in polluting the online information environment.

Auditing algorithms has many uses [2], but one of the ways auditing has been used is as a tool for researchers to learn more about content labeling and misinformation-deterring strategies across many platforms. Some examples are Google's Top Stories feature [39], and news publishers’ search results for reputation check [21].

In the past, researchers have audited YouTube to expose which factors, such as watch history and demographics, have the most influence on whether misinformation is recommended to users [13] as well as identifying user radicalization pathways [30]. YouTube has also been audited to understand how filter bubbles form [37] and what it takes to undo them [35]. [17] conducts a crowd-sourced audit to assess YouTube's stated intention to remove conspiratorial content regarding the 2020 US presidential election from recommendations, while [3] audits the YouTube recommendation algorithm to determine if it played a role in radicalizing users about the 2020 US presidential election. Because of the ever-changing nature of social media and misinformation, [34] advocates for continuous automatic audits.

Many audit studies gather queries using automated resources, such as Google Trends or search auto-complete [13] and videos collected by fact-check sites [32]. While these methods are a great way to focus on the high-volume queries of the moment or target only videos with misinformation, they do not capture the variation of search queries formulated by individuals. Additionally, it may not be the case that Google Trends reflects true population behavior [38]. The two studies mentioned previously, [3, 17], are the only studies that we know of that use crowd-sourced data for their audit of YouTube (in the context of the 2020 US presidential election). We aim to fill this gap in the literature, by crowd-sourcing queries about conspiracy topics, which we then compare against queries and topics gathered without the direct input of search engine users.

Platforms like YouTube, Facebook, Twitter, and Instagram have implemented several forms of content moderation to fight problematic content on their respective platforms. One form of content moderation is content labeling, essentially information attached to online posts and websites that is intended to help individuals understand the intent and credibility of the content. In response to the rise in content moderation strategies across platforms, researchers have begun detailing the different forms of content labeling and the implications for both platforms and users [20, 23]. Prior research has analyzed the effect of fact-checking on users from a psychological perspective [24], investigated political bias in content moderation [16], detailed the fact-checking of news publishers via Reviewed Claims on Google search results pages [21], and considered the effect of source-level labeling of political bias on assessing credibility [28]. [33] provides an overview of the strategies used by Facebook and YouTube to prevent the spread of COVID-19 misinformation, including a brief mention of IPs. More recent work has evaluated democratic forms of content labeling that are not centralized to companies, including browser extensions with crowd-sourced assessments of credibility [15] and citations [11]. The latter study includes a survey of YouTube users on which features of a video they rely on to determine credibility.

Similarly to our auditing work, [18] studied IPs in the context of their effect on the viewership of anti-vaccination content on YouTube. [32] investigates the frequency of IPs in relation to queries for watch page IPs, the links for which were selected for containing misinformation as rated by a fact-checking site. To the best of our knowledge, [32] and [18] are the only other works to date that have focused on YouTube's IPs. While [18] investigates the effect of IPs on viewership and [32] conducts a brief study on watch page IPs, our work explores the extent to which SERP IPs are implemented on YouTube by using a broad range of conspiratorial themes as well as understanding the consistency, extent, and coverage of IPs.

In the following, we give details about the survey that we used to gather query phrases from the crowd.

3.1.1 Survey Composition. Our survey, which was composed on Qualtrics, had two questions and a demographics section. The first question asked respondents whether they had ever searched about conspiracy theory topics. 219 participants answered “Yes” and 90 survey participants responded with “No”. We asked this question to get a sense of how “knowledgeable” people are about the topic. The second question provided them with the following prompt: “Please think of any conspiracy theory topics, then enter below three phrases that you have used or would use to search for those topics.” There were no additional instructions to report three different subjects or queries related to specific topics. This was intentional to ensure some ecological validity, by not putting any additional constraints.

After this question, we collected demographic information such as political affiliation, education, age, gender, and the self-reported frequency with which the participant received their news from YouTube. We collected this information in order to ensure that our sample is somewhat representative of the general YouTube audience and that the results capture diversity across various dimensions. In this paper, we do not use this demographic information for any other purposes.

3.1.2 Survey Distribution and Results. The survey was distributed using AMT after gaining approval from our institution's IRB (Protocol #22092R-E, effective December 20, 2021). Subjects were limited to residing in the United States when taking the survey, having a human intelligence task (HIT) approval rate of over 85 percent, having 50 or more approved HITs, and not having taken the survey before. The workers were compensated $0.50 for their participation. A total of 325 respondents took the survey between December 2021 and March 2022. After the removal of duplicate responses, we were left with 309 participants. From these participants, we collected 927 queries. We considered some responses as nonsense and therefore removed them from our analysis. Removing repeated queries and nonsense queries resulted in 620 unique queries. Adding together the number of times each unique query repeated in the dataset amounts to 892 total queries, indicating some consensus in popular conspiracy topics.

The AMT survey participants described themselves politically as 50% on the left, 27% on the right, and 22% moderate. 43% have bachelor's degrees and 42% are between the ages of 30-39. Gender was roughly equal, with 52% of respondents identifying as men. Of our survey participants, 30% percent reported using YouTube for news frequently, and 65% percent that they did so infrequently.

After collecting the search phrases through our survey, we preprocessed them by lowercasing and removing the ’/’ character.4 We identified 620 unique search phrases, which then were used as inputs for our audit. We aimed to study the extent of IPs across a variety of topics produced by real people without external influence like the autosuggestion feature in the search bar. Such a feature may not be used extensively by YouTube searchers interested in conspiracy theories given that YouTube is likely to filter out problematic queries. Additionally, this method of crowd-sourcing queries for a controlled audit protects participants from research methodology that may be intrusive, such as harvesting real-time user data.

To study the effect of geolocation (that is, whether computers located in different geographical areas in the United States will see the same results or not), we set up virtual machine instances on Google Cloud representing multiple geographical locations in the United States.5 Using Google Cloud servers is a method used by other YouTube audit studies as well, such as [13]. To make sure that the time (morning, afternoon, evening) did not affect the results, the audits were conducted synchronously in all locations. That is, the script that searched YouTube started at the same time and the queries were in the same order in all machines.

Then, to study whether the presence of IPs on search pages changes over time, we repeated our data collection longitudinally on 16 different dates, roughly weekly between March to July 2023.

The setup of the audit was the following: the same Python script and set of queries were stored in each virtual machine instance. On a given date, all instances started running the script at the same exact time. The script created a new instance of the Chrome browser in incognito mode, controlled by Selenium, which then opened YouTube on the browser, searched for the query phrase, stored the SERP as an HTML page, and then closed the browser. This process was repeated for each query. We want to make it explicit that this audit did not use any Google accounts, so the results are not personalized to any particular account. While the lack of personalization might affect how YouTube algorithms work, we did not want to add more variability to our auditing setup. Our goal was to only focus on location, time, and query.

For three locations, some dates did not collect all 620 expected SERPs on all dates. These three locations were removed from the data completely to ensure all locations collected the same amount of data for all dates. Before removal, we had 161,139 SERPs. We were left with 138,880 SERPs from 14 locations on 16 dates using 620 queries that were used for the analysis within this paper.

After the completion of the data collection, the HTML pages of SERPs were scraped to check for the presence of SERP IPs and collect the content of the IP.

For the purposes of understanding the kinds of conspiracy theories that YouTube is prioritizing for content labeling, the unique queries gathered by the survey were grouped in two ways: using Wikipedia-based categorization and handpicked topics (i.e. manually formulated). We analyze the proportion of Wikipedia categories and handpicked topics with IPs. Further, we use the groupings to determine how YouTube covers topics that are popular among the survey participants using context IPs.

Table 1: Handpicked topics with examples of queries grouped under that topic. They show the variation in query formulation by our survey participants.

Table 2: Left subtable: Top 10 queries (out of 620 unique) generated by crowdworkers with the number of times they were repeated. Middle subtable: Top 10 handpicked topics (out of 92) and the total number of queries in each group. Left subtable: Wikipedia categories and the total number of queries in each group. For all query groupings refer to the GitHub repository for this project, linked in Section A.

We determined the Wikipedia categories by using the Wikipedia article titled “List of conspiracy theories,”6 which outlines broad categories of conspiracy theories with examples under each category. Then, we manually sorted queries into 11 broad categories using examples given in the article as guidelines for sorting.

All unique queries were grouped again into 92 topics handpicked by the first author based on the queries in the dataset. We will refer to this grouping as handpicked topics. Examples of handpicked topics with queries are shown in Table  1.

We acknowledge that these groupings are subjective and may differ depending on how a conspiracy theory is interpreted. For example, area 51 was assigned the Wikipedia category outer space but could also fit under government, politics, and conflict considering the theory revolves around both aliens and the US government's involvement. To make our groupings more objective, queries were categorized by multiple researchers, and any discrepancies were resolved through conversations.

To provide an aggregated overview of the queries and the topics, we show them grouped in three ways in Table  2. In the left subtable, we show the top 10 of the unique query phrases, sorted by the number of times they were repeated in the survey response data. In the middle subtable, we show our most popular handpicked topics sorted by the number of queries included in that topic, and in the final subtable, we do the same for the Wikipedia categories. These groups help to give a sense of what conspiracy theories are most popular in our sample, a proxy of the American public in general.

Table 3: Examples of context information panels and some of the queries that invoked the IP at least once along with similar queries (in syntax and/or semantics) that did not invoke any IPs.

We analyzed 138,880 SERPs collected using 620 input queries across 14 locations on 16 different dates. Of these, 23,100 SERPs (16.63% of the dataset) had an IP present. 114 queries triggered a total of 49 unique IPs. On average, 103 out of 620 SERPs showed IPs on any given day at any given location (SD = 1.89).

Three different types of IPs were found in our dataset: context (21,162 SERPs, 28 unique), fact-check (1,892 SERPs, 17 unique), and news (46 SERPs; 4 unique). For any given day at any given location, context IPs were the most prevalent type (mean = 95, SD = 0.87). The next most prevalent type was the fact-check IP (mean = 8, SD = 1.16). News IPs were the least prevalent (mean = 0.2, SD = 0.57).

Given that context IPs are intended to cover a range of queries within a topic (while fact-check and news IPs are invoked in response to specific statements or queries searching for news), we were expecting that the context IPs that are shown in the dataset will cover all queries that are relevant to the IP. For handpicked topics containing at least one query invoking a context IP, there are 62,944 collected SERPs expected to have an IP. The actual number of SERPs with a context IP (21,162) covers 33.6% of what we expected.

Figure 3: Proportion of all Wikipedia categories with any type of IP. Blue represents the total number of queries, orange represents the number of queries that invoked an IP at least once. The mean number of IP-invoking queries across categories is 23 with a standard deviation of 23.

Additionally, out of the 114 queries that showed IPs, 22 queries showed IPs only partially (i.e., showed up for at least one day or location, but not the rest). We go more in-depth about these queries in Section  4.2. Because YouTube has shown an IP for these queries at least once, this indicates that it recognizes these queries as related to problematic content. For these 22 queries, we calculated the number of times the IP did not occur over the 16 dates and 14 locations: 2,436. This corresponds to a 49.4% absence rate for queries that are recognized as problematic.

Further, context IPs in particular would be expected to cover popular topics prone to conspiracy theories (i.e. topics for which our participants formulated many queries). Four of the top 10 handpicked topics (911, covid, election, vaccine) did not have context IPs associated with them (see example queries for some of these topics in Table  1).

Similarly, when analyzing the prevalence of all IPs as the proportion of queries in Wikipedia categories, see Figure 3, we find that the number of queries with IPs within Wikipedia categories is not related to how popular the category is, as determined by the total number of queries in the category. Concretely, Wikipedia categories have an average of 23 invoking queries (SD = 23). However, the majority of queries for the two main categories: government, politics, and conflict and outer space don't appear to have triggered any IP.

To summarize the findings of this section, we can say that in the context of our audit, IPs do not appear to be very prevalent on YouTube among queries formulated with the intent of finding conspiratorial content. This claim is supported by the fact that a relatively small percentage of our dataset of SERPs displayed IPs, IP coverage did not follow patterns of query and topic popularity, and even queries that were marked as related to problematic content by YouTube's system did not show IPs across all dates and locations.

Table 4: Examples of queries that invoked fact-check information panels.

Heatmap of IP distribution across date and location.

Figure 4: Information panel (IP) distribution across dates and locations. Colorbar values represent the number of SERPs with an IP present from 620 input queries. X-axis values are the dates of data collection in 2023 from March to July.

This investigation examines the effect of date and location on IP occurrence. We plotted the number of IPs for each round of data collection and observed that the highest occurrence of SERPs with IPs on one day in one location was 108 and the lowest was 101 (Figure  4). The date of data collection had more influence on the number of SERPs with a present IP than location, though there was a slight variation in the number of IPs across locations within a small subset of dates. The heatmap in Figure  4 makes this clear: there is little variation vertically (each column is mostly mono-chromatic - meaning the IP number was constant across all locations), meanwhile, there is large variation horizontall, showing that the number of IPs changed from one date to the next.

Because the number of queries with an IP on any given date and location varies, this means that a subset of queries are invoking IPs inconsistently. We became interested in investigating if there is a pattern associated with these inconsistencies. Figure  5 depicts all 22 queries that inconsistently showed IPs across dates. Eight of these 22 queries are inconsistent across both date and location. Qualitatively, there is no clear pattern as to why these queries are inconsistently showing IPs.

Additionally, the date influenced the type of IP shown. The vast majority of queries (112) showed consistent IP types across all SERPs collected using that query. For example, the IP associated with jfk assassination was always a context IP. However, 2 queries did not follow this pattern and invoked either a fact-check or context IP depending on the day. Meanwhile, the query chemtrail evidence invoked a fact-check on 6/18/23, but invoked a context IP on every other day of data collection. The query what are chem trails? invoked a fact-check IP on 9 out of 16 dates and a context IP on 7 out of 16 dates, not clustered together.

Thus, date is more influential on IP occurrence than location, though there was still very slight variation between locations for some queries. About 1/5 of all queries with an IP varied by date and less than 1/10 varied by location. This suggests that in future audits it might not be necessary to use a complex setup to account for different locations as that is less likely to influence the results. Meanwhile, periodically collecting data over time will be very important to capture changes in coverage.

Heatmap of IP distribution for subset of queries that show up inconsistently across date.

Figure 5: Information panel distribution across date for queries that invoke information panels for only some dates. Colorbar values represent the number of locations with information panels out of 14 total locations. X-axis values are the dates of data collection in 2023 from March to July.

Three types of IPs appeared in the dataset: context, fact-check, and news. Each type has a different purpose, as stated by YouTube7. We broke down the data by IP type and conducted a qualitative analysis of the queries that invoke IPs for the purpose of investigating what about a query triggers an IP.

Table 5: All queries that invoke news information panels.

Stacked bar graph of proportion of queries in handpicked topics that invoke an IP at least once.

Figure 6: Proportion of queries in handpicked topics that invoke a context information panel at least once. Handpicked topics with no queries that invoke a context IP are not shown. Orange represents the number of context-IP-invoking queries. Blue represents the total number of queries in a topic. The mean number of queries that invoke context IPs is 10 with a standard deviation of 14.

4.3.1 Context IPs. Context IPs made up the majority of all IPs in the dataset, with 21,162 SERPs having context IPs or 15.24% of the dataset. We have a total of 28 unique context IPs triggered by 97 queries. IPs tend to cover more than one query (Table  3). On average, one context IP is invoked by about 4 queries with a standard deviation of 6. The number of queries that invoke one context IP ranges from 1 to 30.

30 of these queries invoked one IP to the Encyclopedia Britannica page "assassination of John F. Kennedy" at least once in the dataset (this is the same IP shown in Figure  1). However, there are 50 queries within the handpicked topic jfk assassination. When observing the 30 queries that invoked a context IP, all included John F. Kennedy's name or the acronym of his name, which is also present in the text of the IP. Two of the 20 queries within the handpicked topic jfk assassination that did not invoke a context IP do not include the name of John F. Kennedy: magic bullet theory and the magic bullet theory. However, 18 of the 20 were qualitatively very similar, with some reference to the event. For example, who shot john f. kennedy?, government killed jfk, and john f kennedy inside job.

A similar behavior is true for all queries that invoked a context IP, wherein keywords in the query match part of the IP's text, except for the query global warming, which invokes an IP that does not mention the phrase directly in the text or in the article but includes the synonym phrase climate change.

To determine the relationship between the popularity of handpicked topics and how often the queries within those topics invoke IPs at least once, we compared the total number of queries in a topic to the number of queries that invoked IPs (Figure  6). We expected to see a functional relationship between the number of IP-invoking queries and the total number of queries in the handpicked topics, but the proportion varied.

4.3.2 Fact-Check IPs. Fact-check IPs made up the next largest share of IPs in the dataset, with 1,892 SERPs having a fact-check IP or 1.36% of the dataset. Compared to context IPs, the number of fact-check IPs roughly matched the number of invoking queries. 16 queries invoked a total of 17 fact-check IP URLs. Examples of these queries are shown in Table  4.

Similarly to the context IP queries, the queries that invoked fact-checks had at least one keyword that matched the IP text. However, we found that queries invoking fact-check IPs tend to have more than one keyword in the query phrase matching the text of the IP.

Only three queries invoke a fact-check IP on every day of data collection in every location: covid vaccine nanobot, is epstein alive?, melania trump double, indicating that fact-check IPs are less reliable than context IPs.

Two queries invoked more than one fact-check IP URL. The first query michelle obama male invoked three different IP URLs that change according to the date. At the start of the data collection period in March of 2023, the query resulted in a SERP with an IP that linked to a Politifact.com article published in May 2021. On 4/19/23, the IP linked to a different Politifact.com article published February 2023. The IP changed again 5/16/23 to a Politifact.com article published just a few days before data collection, 5/8/23. Each of the three articles debunks a different piece of evidence for the conspiracy theory.

The second query jfk alive links to two different fact-check articles, both from Politifact.com with similar content, published on the same day. One article shows up for 3 days of data collection in the month of June, while the other shows up for 6 scattered days across the data collection period.

4.3.3 News IPs. News IPs were the most infrequent type of IP in the dataset, with only 46 SERPs having a news IP or 0.03% of the dataset. Similar to fact-check IPs, the number of news IPs roughly matched the number of invoking queries. 3 queries invoked a total of 4 news IPs. All three queries and the IPs they invoke are shown in Table  5.

The news IPs were highly dependent on time, with IPs showing up for one day for two queries (trumps and school shootings) and two days for the third query (“deep state”8).

The first query school shootings invoked an IP on 5/16/23 linked to a CNN.com article reporting on a school shooting that occurred on 5/15/23.

The second query “deep state” invoked three different news IPs. On one day, one location showed an IP that linked to a WashingtonPost.com article and 15 locations showed an IP that linked to a TheDailyBeast.com article. On the second day the IP was present, 6/11/23, the same query showed an IP linked to a Ledger-Enquirer.com article. On the same day, the third query trumps invoked the same Ledger-Enquirer.com IP. Each of these three articles reported on completely different events relating to Donald Trump. None of the articles included the phrase “deep state”.

Interestingly, a separate query deep state was present in the list of queries (without quotation marks) and did not invoke an IP across the data collection dates and locations.

These qualitative case studies reveal that related keywords are seemingly required for the presence of context and fact-check IPs, but this relationship is less obvious for news IPs. While keywords might be required, they are not necessarily sufficient. Some queries with high word overlap did not show IPs. Additionally, some queries both had word overlap and showed IPs on some days, but not on other days. More research is needed to understand the variables that connect queries and IPs.

A clear result from this work is the variation between IP type and date. When a query is connected to a context IP, it will show in almost all conditions. However, fact-check IPs are much more dependent on date, though the pattern of this relationship is not obvious. News IPs are highly dependent on date. This relationship is somewhat clear and intuitive based on the purpose of the news IPs: the news story within the IP is published and shown to users during the time frame of the breaking news event. In one example, the query school shooting matched an IP giving information on a school shooting that happened the day prior to data collection. However, this is just one example out of only 3 queries that invoked IPs. More news IPs need to be analyzed to understand their function and reach. This is why in the following section, we discuss an extension of the audit methodology intended to inform future audits that will try to replicate and expand upon these findings.

One of our goals for this study was to investigate how to best conduct meaningful audits of YouTube, that allow us to capture its changing behavior toward problematic content, in a quasi-realistic way. We experimented with three variables: crowd-sourced queries, time, and location (the geographical location of machines on which the audit was carried out). We find that IP presence varies more over time than location. Some of this variation is due to links updating, particularly for news and fact-check IPs. Other times, an IP disappears completely with no indication as to why.

We further tested this time-dependency hypothesis by collecting data for an additional two dates (10/30/23 and 11/07/23) across the same 14 locations using the same 620 input queries. This round of data collection resulted in 17,360 SERPs and will be referred to as the “2-week follow-up dataset”.

Comparing the 2-week follow-up dataset to the original 16-week dataset reveals a jump between 7/23/23 and 10/30/23 in the highest number of IP-invoking queries in one day from 108 to 130. In total, 32 queries invoked IPs in the 2-week follow-up dataset that did not invoke any IPs in the 16-week dataset. Most of these added queries (26) were related to the 9/11 topic, while the other 6 were unrelated to each other. One new context IP was added (a Britannica article on 9/11 attacks) along with 4 fact-check IPs (on various topics). No news IPs were encountered. At the same time, 17 total IPs had “disappeared”, including all 4 news IPs and 13 fact-check IPs from the 16-week dataset.

Thinking about these results in the context of global news events, we wonder if the addition of the IP showing the Encyclopedia Britannica article on the 9/11 attacks was related to the attack on Israel by Hamas on 10/7/23, just a few weeks before we collected our follow-up data. We think this is plausible considering the comparisons made between the 9/11 attacks and the events in Israel [19].

Our findings indicate that news IPs are highly dependent on time. This matches YouTube's description of news IPs that they show up for “developing news stories”9. Fact-check IPs display a similar dependence on time, with articles updating or disappearing altogether for some dates.

Given the dependence on time for IPs overall and our hypothesis that YouTube may be updating IPs in response to real-world events, based on our follow-up data, we believe that it is important for researchers to develop methods that enable auditing that is acute and responsive to real-world events. We refer to this as “news-cycle-aware auditing”. This type of auditing would allow researchers to capture changes in content labeling that occur in response to problematic content arising on platforms as a result of real-world events. A challenge for such audits is the choice of keywords. Typically, one cannot predict ahead of time what queries will be used by YouTube users to search for a developing story, but also, what old conspiracy theories might resurface due to perceived similarities to an unfolding event. This is why more research is needed to design methods for news-cycle-aware audits.

Each type of IP (context, fact-check, news) seems to rely on different occurrence criteria, with one common thread: keywords.

We observed that queries invoking fact-check and news IPs typically matched multiple keywords in the article being shown, while context IPs had at least one keyword in the query phrase. This is also indicated by the fact that one context IP typically covers multiple query phrases while the fact-check and news IPs are targeted to specific query formulations. This matches YouTube's explanation of showing fact-check IPs in response to “specific claim[s]”10.

While keywords seem to be a requirement for YouTube's algorithm to recognize the need for an IP, it does not always guarantee the presence of an IP. We found that some queries that invoke IPs are qualitatively similar to queries that do not invoke any IP. This inconsistent triggering of IPs is puzzling and is one of the areas that needs to be researched further.

Once a query has triggered an IP, a process of source selection, article selection, and panel text selection needs to be completed before an IP is displayed on the SERP. The methods YouTube uses to complete these processes are also unclear based on the results of this audit.

For example, YouTube states that the sources for news IPs are chosen using a range of criteria, including local relevance, relevance to the topic, date posted, reporting intent, association with websites whose content surfaces on Google News, and more. In our data, we encountered four sources for news IPs: WashingtonPost.com & CNN.com (widely recognized news sources worldwide) and TheDailyBeast.com & Ledger-Enquirer.com (low levels of recognition and disputed factual accuracy). Given the importance of accurate information during developing news events, the choice of the news source to show in the IP is very important.

Further, many queries in the dataset seem to seek information about a specific popular claim, like earth is flat, but did not invoke a fact-check IP. YouTube gives this explanation11:

If a fact-check doesn't appear, it may be because an eligible publisher hasn't published a fact-check article relevant to your search.

However, a quick search on Politifact.com for the phrase “earth is flat” reveals many articles on the topic.12

We collected the crowd-sourced queries by explicitly asking crowdworkers to provide a query they would use to search for a conspiracy theory. In some cases, the query phrase does not indicate that intent explicitely in its formulation (for example, jfk or moon landing). As researchers, we know the intent of the query, but the algorithm does not recognize the connection to problematic content and therefore does not show an IP. In these cases, we would say that the algorithm is not sensitive enough to queries with problematic intent, either by design or not.

We speculate that the YouTube algorithm may have some threshold for recognizing queries with conspiratorial intent that is higher than what we would expect. We hypothesize that this is the case because YouTube might not want to expose the YouTube audience to conspiracy theories. That is, YouTube might be using an editorial policy known as “strategic silence” [4]. Testing such a hypothesis might require a different audit setup, especially one that would take into account a user's prior search behavior.

In addition to calling for the YouTube algorithm to add content labels more consistently and with better coverage, we believe the content of the IPs should be meaningful. Context IPs themselves often don't warn the user that the topic may be connected to problematic content. Of the 31 context IPs, only 7 of them explicitly mention the topic's connection to conspiracy theories. And only a single context IP uses the word “disproven”. Thus, it's unclear how effective context IPs are at deterring users from seeking and believing problematic content. Future work should evaluate the efficacy of different IP types in deterring YouTube users from uncritically consuming or believing problematic content.

Given the importance of YouTube on the information environment and the potential for problematic content to have real-world harm (i.e., the Marjory Stoneman Douglas shooting “crisis actors” narrative), informing consumers about the truth consensus with respect to certain topics is incredibly important. YouTube seemingly recognizes the need for a system that supports users in understanding the content on the platform, thus implementing and updating IPs over the years. In our view, if YouTube is serious about the value of the IP strategy in discouraging the consumption of problematic content within the platform, the implementation of the policy should be clearly stated or, at the very least, easily discerned from interacting with the platform. We provide evidence in this paper that the criteria YouTube uses to decide if and when an IP is shown to users are obscure.

While our goal with crowd-sourcing queries was to capture the variation in query formulation by real people searching YouTube for conspiracy theories, the queries were ultimately prompted by our survey, as opposed to having been extracted directly from a user's interaction with YouTube, for example, through a browser extension. Furthermore, this process also does not capture a user's “on-the-fly information need” which is usually prompted by external factors like coming across an unknown (conspiracy) theory in everyday life. Despite these shortcomings related to the ecological validity of the query selection, our queries cover a wide range of topics and formulations not addressed in other studies. Further, we believe that this strategy has some unique benefits, including avoiding collection of user data and thereby protecting participant privacy. In this study, we were interested in focusing on a few key variables (time, space, and query) in order to understand the general implementation of IPs. IPs as a content labelling strategy are intended to warn users coming across potentially problematic content, so they should be available regardless of individual user demographics or behavior, an additional reason to disregard personalization as a variable in this audit.

Another limitation is related to the lack of news-relevant queries, which led to a very small sample size for news IPs in our dataset. This is due to the time gap between query collection and SERP collection. For one to form a comprehensive study of news IPs on YouTube, a news-cycle-aware audit methodology must be adopted.

Finally, our audit is limited to the United States and the English language. While YouTube has declared that the IP feature will be available in other countries and languages other than English, our audit did not consider other localities. However, our audit methodology (the use of virtual machines in Google Data centers located across the world, can be used to perform audits of YouTube in other countries.

Though this study examines only variations in the occurrence of YouTube's IPs, there are many possible avenues for continuing to understand how this intervention is shaping the information landscape on the YouTube platform and broader web.

One important consideration is how the presence of IPs effects a users watch behavior on YouTube. A user-centered study may help to understand real-time video-watching behavior when IPs are present on YouTube's search results page or on video watch pages. Some research has already been done to characterize which features of YouTube videos watchers utilize to assess the credibility, including some interpretation of the IP design [11]. An additional consideration could be how demographic information and account personalization influence conspiracy-related query searches; how often people are exposed to IPs when browsing YouTube; and the perceived value of IPs as credibility signals.

We decided to focus on YouTube due to the immense popularity of the platform and its previous history of hosting conspiratorial content. However, understanding content labeling on YouTube has implications for the broader information landscape. Researchers are working to centralize information about the types of interventions employed online to decrease the spread of misinformation. One example is an online toolbox of interventions, including data about each conceptual intervention like scalability [20]. This study can be used to inform such efforts, which will in turn provide policymakers with knowledge of effective and practical forms of content moderation.