Think-Aloud protocol's theoretical grounding is rooted in the concept of inner speech [52, p.237-239] and its interaction with abstract thought [52, p.263-264], as an inner movement through different planes (abstract and verbal). “Inner speech is an autonomous speech function” that can be regarded as a “distinct plane of verbal thought.” It comes to fruition as “a complex, dynamic process involving the transformation of the predicative, idiomatic structure of inner speech into syntactically articulated speech intelligible to others” [52, p.263]. To Vygotsky, “thought is not merely expressed in words; it comes into existence through them” (p.231). Charters [10] noted that the application of inner speech to Think-Aloud protocol makes its predicate dominance clearer because the subject of the Think-Aloud is usually visible and evident to the participant.

Even though Think-Aloud protocols are widely used to gather data in studies involving cognitive psychology, there is still some controversy regarding insufficient knowledge on two issues: its completeness and accuracy and its reactivity [27]. Completeness and accuracy raised questions regarding whether a Think-Aloud report is consistent with one's thinking process and product [27]. About the matter, we argue that Vygotsky's concept of inner speech is a robust indication against this particular controversy.

On the other hand, reactivity raised questions about whether one's task performance under the condition of thinking aloud differs from one's performance in the same task under the condition of not thinking aloud [53]. Regardless of the thought-word connection, the requirement to Think-Aloud while working may affect how participants handle tasks, the time it takes them to carry out tasks, and their eventual success in task completion [48]. Indeed, reactivity can change the sequence of thought processes and interfere with cognitive processing [11], which motivated us to verify the impact of verbalization/non-verbalization on the searching process.

Hypertext theory spans works from computer science to literary studies and became closely identified with post-structuralist theory [6], especially the concept of deconstruction [24, p.2-3]. Deconstruction considers that a text has not just one but many different meanings and should be seen as an endless stream of signifiers, with words only pointing to other words without any final meaning [13, p.6-26]. In this sense, Landow argues that conceptual systems founded upon ideas of center, margin, hierarchy, and linearity must be abandoned and replaced by ideas of multilinearity, nodes, links, and networks [24, p.2]. Barthes proposes a division to the concept of text, claiming that texts are either readerly or writerly [7, p.4]. Readerly means there is a unilateral relationship between a text and reader (traditional text), and writerly requires the reader to perform a more active role by browsing from a departure node to a destination node by links (hypertext). Hypertext would mark a new paradigm, introducing a mutual process of communication1, establishing a bilateral (or multilateral) relationship between text and reader and requiring an active and attentive reader who has to find out the text's meaning [7, p.142-148].

Hypertext theory adheres to our study through the user's browsing behavior. The notion of author is reconfigured to be conceived not as the author of the text but as the text itself [25, p.126] as well as the reader. They became a centerless network of codes (the author, reader, and text), which serve as nodes within another centerless network (the internet) [25, p.127]. Hence, identifying and relating users’ searching behaviors to the knowledge gained during the Web search session is related to hypertext and information exploration and visualization. It is a step toward improving Web search engines to aid users in satisfying their information retrieval-related objectives.

26 Master and Ph.D. students were selected from five different Brazilian universities2 to be part of this qualitative study. They were chosen due to their information searching experience, which can be determined based on the students’ daily information searching time and their knowledge of information searching techniques [5] [30]. The profile was determined following Aula and Nordhausen's procedure [5] to ascertain the participants’ background by tracking their years of computer and Web experience; frequency of using search engines, the Web and computers; and their own evaluation of search skills. Their familiarity with the tasks was also evaluated. The data saturation process is described in Appendix 1.

The participant's sessions took place through online chat (i.e., Google Meet), and the chat screens were recorded with the participants’ consent. The role of the researchers in the online sessions consisted mostly in reminding the participants to verbalize their thoughts, as required by the chosen Think-Aloud protocol. The participants were asked to perform three different search tasks (Table 1) based on instructional activities developed by Kelley et al. [21]. The difficulty of the tasks was planned with different levels, with the first two tasks meant to encourage the participant to perform a search with learning intent and the last task proposing a challenging activity using the information retrieved from the previous two searches.

Table 1: Overview of the proposed search tasks.

We asked the participants to open a Web search engine of their choice to initiate the tasks, and we captured the query states, starting with the terms they used to formulate the initial query. The other query states derived from the ensuing reformulation performed by the participants either on the search engine or a website or platform embedded searching system. In addition to the terms, we registered the whole sequence of actions and spoken descriptions based on the Tsai and Tsai's framework [47] as described by Reisoğlu et al. [42] to identify the online information searching strategy (OISS). Reisoğlu et al. [42] thoroughly described codes and indicators for each OISS, which we transcribed in Table 2 to support the reader's understanding. The framework is composed of three main domains3: Behavioral, Procedural and Metacognitive. The Behavioral domain has two sets of strategies, Control (with ten indicators) and Disorientation (with four indicators). The Procedural domain has two sets of strategies, Trial and Error (with three indicators) and Problem-Solving (with two indicators). Finally, the Metacognitive domain has three sets of strategies, Purposeful Thinking (with four indicators), Select Main Ideas (with eight indicators), and Evaluation (with four indicators).

Table 2: Online Information Searching Strategies’ indicators [42].

The framework presented in Table 2 aided us in establishing a set of codes that could be applied to categorize our data. The coding process was executed by the researchers themselves. We registered the terms used in each search task by each user, the mapped query state4, and their online information searching strategies.

We transcribed the participants’ search sessions to show how keywords and terms evolved throughout the search and identified the searching strategies and domains based on the indicators from the participants’ behaviors (see Table 2 for the codes’ description). The online information searching strategy was determined by the user's attitudes and explanations during the Think-Aloud procedures.

Controlled cognitive processing is an information-processing approach defined by cognitive psychology as the required conscious effort from individuals to comprehend, elaborate, and restore into memory the information presented by an information source [2, p.9-10]. Its effects can be measured by different outcome variables such as knowledge, affect, attitude, intention, or actual behavior [53]. Controlled cognitive processing is applied to examine the range of cognitive responses related to knowledge gain. We analyzed all the identified indicators considering their intrinsic concept to determine knowledge gain strategies. For that matter, it is essential to observe the appearance of four specific elements during a cognitive activity, e.g., a Web search, namely the concepts of cognitive absorption, comprehension, elaboration, and memory.

Cognitive absorption is a key prerequisite for controlled cognitive processing related to attention by selectively concentrating on certain objects while ignoring perceivable others [53]. The background of cognitive absorption derives from three closely inter-related domains: the trait of absorption (the propensity of deep awareness and engagement); the state of flow (characterized by a total concentration in an activity); and the notion of cognitive engagement (defined as a significant learning outcome derived from technology-mediated learning) [1]. Cognitive absorption is influenced by the perceived usefulness of an information source [34].

Comprehension determines the extent to which one can understand a piece of information and comprises the content and the structure of the message being communicated [53]. It involves the perception of the encoding used (e.g., the language), the process by which the content is transformed into a mental representation, and the understanding of how to use the mental representation (e.g., if the mental representation is a question, one may answer it) [2, p.313].

Elaboration is built upon comprehension and is a key asset to substantive knowledge gain [53]. It is defined as the degree to which individuals cognitively process the information to scrutinize its argument, weigh the evidence, analyze the logic, and assess the soundness of the claims presented [45]. Elaboration manifests itself as the ability to generate relevant thoughts and fill in additional information about the information retrieved, and, in Web searching, it has often been linked to thoughts users generated in response to website information [53].

Memory has been commonly operationalized as recall and recognition, and it refers to the storage and retrieval of information presented [53]. It confers the ability to judge whether one has encountered an object, idea, or situation in the past [35]. It is another essential cognitive component of knowledge gain in Web searching because users tend to jump back and forth among pages and to skip and attend only to certain parts of a message [53].

To observe the appearance of the four specific elements during the search sessions, we borrowed constructs from the mathematical logic to use in the concepts of cognitive absorption, comprehension, elaboration, and memory. The term “construct” is used in this work as understood by Set Theory's logic. Quine defines it as sets of ordered pairs matching each member of the population to a specific value for the variable [41, p.291]. As constructs, the concepts’ attributes (components or characteristics) and the indicators can be represented as a two-place logical relation5 [33]. It means that the variable (i.e., the concept's attribute) can be treated as a predicate that either applies to a particular subject (i.e., the indicator) or not [33]. For example, if H represents “honesty” as a simple binary, non-vague predicate, then for any person, x, either H(x) or ¬H(x). That is, each individual has the property of honesty or not [33].

Then, the concepts’ attributes were chosen based on the following definitions: cognitive absorption as the construct that better describes the state of full engagement and immersion [31]; comprehension as the construct that better describes the abilities to interpret word meaning in context, to recognize relevant information, and to synthesize scattered information [46]; elaboration as the construct that better describes the abilities to organize knowledge into a coherent structure and integrate new information with existing knowledge structures [20]; memory as the construct that better describes the abilities to encode, store, and retrieve data or information when needed [38].

The concept attributes were used to evaluate the 31 indicators based on their description shown in Table 2. As the attribute is binary, a score of 1 indicates presence and a tagged 0 otherwise. Table 7, in the Appendix 2, presents the knowledge gain concepts’ presence/absence evaluation. We were able to determine sixteen indicators with at least one of the attributes related to knowledge gain: PS2, PT1, PT3, PT4, SMI1, SMI2, SMI3, SMI4, SMI5, SMI6, SMI7, SMI8, E1, E2, E3, and E4. We called them knowledge gain indicators due to their relation to the knowledge gain attributes.

On reactivity (see Subsection 2.1), some factors may modulate the potential difference of performance. We argue that articulating thoughts in words during a search task may not account for the normal condition of one's search performance. During our analysis of the recorded searches, we realized that the participant reflected on the content seen in a non-verbal way in many cases. Also, we realized that some of the 31 identified indicators (16 of them) could be related to knowledge gain concepts [53]. The cited knowledge gain require higher-order thinking skills and beliefs about the usefulness of the information or knowledge in question [43, p.317]. Then, we decided to verify if verbalization periods6 discourages knowledge gain during Web searches by analyzing the participants’ behavior regarding the searches’ periods of verbalization or non-verbalization compared to the remainder of the observed indicators.

In this sense, to answer the question “does verbalization discourages knowledge gain during Web searches?”, we sought to verify the ensuing null and alternative hypotheses:

Null-hypothesis (H0): knowledge gain indicators are not related to search activities during periods of verbalization;

Alternative-hypothesis (H1): knowledge gain indicators are related to search activities during periods of verbalization, but not related to search activities during periods of non-verbalization;

Alternative-hypothesis (H2): knowledge gain indicators are related to search activities during periods of both verbalization and non-verbalization.

If the null-hypothesis (H0) is confirmed, it means that there are statistically sufficient grounds to believe there is a negative relationship between verbalization and knowledge gain periods because thinking aloud would affect the participants’ online information searching strategies during the search sessions (i.e., preventing them or interfering with their ability to gain knowledge). If the null-hypothesis (H0) is rejected, it means that verbalization would not influence their online information searching strategies. Nevertheless, their ability to gain knowledge might be positively related to their choice of whether think aloud or not (H1), or yet not be affected at all by it (H2).

The traditional approach to evaluating the association between two categorical variables uses the Pearson chi-square test of independence [8] [3]. As the chi-square test of independence is designed under the assumption that all observed frequencies are in the determined form (i.e., are not an approximate estimation) [3], it is a good fit to test whether the two criteria of classification presented in either hypothesis (null and alternative) are independent or not. For completeness, the chi-square test equation is shown in Appendix 3. The calculated chi-square is then compared to a critical value, which is determined by the level of significance (typically, p-value = 0.05) and the degrees of freedom (df), which equation is also shown in the Appendix 3.

We also analyze the nature of the dependence between the row and the column variables to determine the most contributing cells to the total chi-square. We achieve it by calculating the Pearson residuals (PeaRes) for each cell. Once again, its equation is shown in Appendix 3. The Pearson residuals are used to verify the type of association between a particular row and column. If the value is positive, it is said that they have a positive association (also known as an attraction). On the other hand, if the value is negative, it is said that they have a negative association (a.k.a. repulsion).

We examined the data for patterns based on the constant comparative method (CCM) [9]. Each participant's performance was analyzed to label the searching strategies with the most appropriate codes. Then, we compared the participants’ searches. The purpose of this type of comparison is twofold: firstly, it intends to identify similar strategies which could be grouped together, forming a pattern; secondly, to observe the impact of adding a new participant and determine the overall number of participants to the study. In this case, when new participants did not bring any new information to our analysis and the formula we used to calculate the discovered sample (see Appendix 1) indicated the standard percentage of 80-85% of discovered information, we considered the study saturated.

An amount of 26 participants was needed to reach the discovered sample equivalent to 0.86 (above the 80-85% threshold). The average time of the participants’ search session was one hour and nine minutes (average time=1:09:19) and the standard deviation was twenty-nine minutes and fifty-three seconds (σ=0:29:53), considering the three search activities (see Table 1). We identified 31 out of 33 possible indicators7 from a total of 78 search activities.

We counted the verbalization and non-verbalization periods from the search session videos. The analysis was conducted independently, with each video annotator reviewing separately all the videos. By comparing different parts of the search with the answers provided by the participant, the interview consistency was examined during the coding process and the annotators agreed on the results by comparing their notes from their coded tables. When the participants verbalized what they were doing or their thoughts regarding their searching activities and results checking, we considered it a period of verbalization. When the participants remained silent while performing a searching activity and results checking or some task-relating activities (e.g., reading or writing) or yet used a verbal filler, which is an apparently meaningless word, phrase, or sound that marks a pause or hesitation in verbal speech (e.g., terms like “uh-huh” or “mmhm”) [12], we considered it a period of non-verbalization. The observed frequencies for each indicator identified are shown in Table 3.

Table 3: The observed frequencies for each indicator.

The calculations were done using R. The data were entered as shown in Table 3, with rows representing each indicator, columns accounting for the periods of verbalization or non-verbalization, and values representing the frequencies scored by the indicators for each period type. The expected values (Table 4 - its formula is shown in Appendix 3) and the degrees of freedom (df=30) were calculated.

Table 4: The expected frequencies under the null hypothesis of no association.

The expected values are the values that would be expected under the null hypothesis if there is no association between the variables. The chi-square (χ2) is based on a test that measures the observed data's divergence from the expected values. The calculated chi-square is the following: (χ2=536.24), which indicates a substantial divergence between the observed data and the null hypothesis8. However, the number by itself is not that useful because we cannot certify if the χ2=536.24 is unusually large since there is no previous calculation regarding the instance. Hence, we need to place it into a broader context to determine whether it is an extreme value or not. The observed p-value=4.326807e-94 can provide a broader context. Note that the value is indeed less than the assigned significance level (≤ 0.05). In fact, it is very close to zero, which would indicate a significant result.

To cross-validate the results, we calculated the Pearson residuals. The statistical model underlying the Pearson chi-square test of independence is that the cell counts are independent Poisson random variables [4]. It means that the number of events (periods of verbalization or non-verbalization associated with the indicators) that happen during a fixed time interval (during the search tasks) is a random number with a Poisson distribution (no particular indicator changes the probability of when the next indicator will occur). Since a total row or column count is imposed, the tally's resulting conditional distributions become multinomial [44], allowing calculating the residuals. A residual is a measure of the distance a data point is from a particular line; in other words, it shows how well this particular line fits an individual data point. The calculated Pearson residuals are shown in Table 5.

Table 5: The Pearson residuals.

Note that the positive and negative residuals do bear a relation to the frequencies shown in Table 3. Figure 1 provides a better look at the nature of the dependence between the variables. We slightly changed the indicators’ position (8 out of 16 knowledge gain indicators in the first positions) to enhance the effect. Positive residuals are in blue and negative residuals are in red. The knowledge gain indicators positively associate with the non-verbalization column. Besides the eight indicators (PT3, SMI6, SMI7, SMI8, E1, E2, E3, and E4), the C6 and C9 indicators also bear an attraction to the non-verbalization while the remaining indicators show a repulsion. In this sense, we can reject the null hypothesis (H0). Concerning the alternative hypotheses, we argue for the acceptance of the H2 since half of the identified knowledge gain indicators bear an attraction to non-verbalization periods, and half of them bear an attraction to verbalization periods.

Figure 1: Residuals visualization - positive residuals are in blue and negative residuals are in red.

Concerning the impact of Think-Aloud on knowledge gain processes, verbalization does affect how participants handle their search tasks, as the Pearson residuals show, 21 out of 31 indicators positively associate with verbalization. However, a better part of the identified Metacognitive strategies is performed in a non-verbal way (8 out of 15). Eight knowledge gain indicators with a positive association with non-verbalization also have at least two concept attributes associated with knowledge gain (see Table 7 in Appendix 2). These knowledge gain-related activities tend to be performed in quietness despite Think-Aloud's verbalization influence. Their appearing indicates a controlled cognitive processing applied by the participants when performing strategies related to metacognition (i.e., indicators related to Purposeful Thinking, Selection of Main Ideas, and Evaluation). As the result shows a predominance of non-verbal periods during metacognitive-based searching activities, it may indicate that Think-Aloud protocols should not only rely on verbalization periods for indication of knowledge gain, non-verbalization periods should be considered as well. We argue that non-verbalization periods is still relevant when Concurrent Think-Aloud protocol is not used, as there is the case with other Think-Aloud methods in which participants are not required to verbalize their thoughts when performing tasks, such as the Retrospective and the Partial Concurrent Think-Aloud. Although verbalization may not disrupt the thought process, it might cut in on the cognitive process as the participant tries to explain her action while performing it. Also, search engines could stimulate more interactions for exploration and information-adding focused on knowledge gain interactions, regardless they are from verbal or non-verbal behavior. As Marchionini suggests, it is “increasingly apparent that information seeking augments our intellects at various levels of life” [32]. A computationally enhanced search process focused on knowledge gain interactions (e.g., grounding on the concepts of cognitive absorption, comprehension, elaboration, and memory) could facilitate the knowledge gain during online search sessions with impacts on identifying user information needs and adding personalized information throughout the search.