Abstract

Although the internet is a means for disseminating information and facilitating social interactions, these benefits are limited due to individuals’ propensity for engaging within a narrow range of communities that share similar beliefs. A portion of these online communities facilitate radicalist viewpoints, including toward marginalized populations, contributing to misbehavior and exacerbating social inequalities. Although a variety of theories propose to explain the processes of online radicalization, less work has empirically examined how users’ communication patterns change over time, especially in terms of novelty versus regularity of user comment features. The present research demonstrates a new modeling approach for examining the extent to which low-level, multimodal comment patterns evolve as users communicate within a Reddit forum well-known for its extreme misogynism. Our results confirm that low-level comment patterns predict high-level features of radicalization, aligning with theory on attitude polarization and contributing to literature on detection and interventions to mitigate extremism.

CCS Concepts

CCS CONCEPTS • Social and professional topics~User characteristics~Gender • Human-centered computing~Collaborative and social computing~Empirical studies in collaborative and social computing • Security and privacy~Software and application security~Social network security and privacy

Keywords

KEYWORDS Emergent patterns, Multimodal, Nonlinear dynamical systems theory, Multidimensional Recurrence Quantification Analysis, Radicalization, Sexism, Social Media

ACM reference format: ACM Reference format: Aaron Necaise, Aneka Williams, Hana Vrzakova, and Mary Jean Amon. 2021. Regularity versus novelty of users’ multimodal comment patterns and dynamics as markers of social media radicalization. In Proceedings of the 32nd ACM Conference on Hypertext and Social Media (HT ’21), August 30-September 2, 2021, Virtual Event, Ireland. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3465336.3475095

1. Introduction

Online extremism occurs when users zealously adhere to a belief or value system with advocacy beyond the norm [9, 29], often engaging in hate speech or spreading misinformation. Despite the benefits of social media, these platforms support extremism due to their low publication threshold, world-wide audience, and increased level of anonymity [3, 5, 19, 22, 23, 27, 33]. The increase in online extremism has real-world consequences. For example, writing posts and interacting with others online when engaging in sexist dialogue increases self-reported sexism [12], and misogynistic language on Twitter has been linked to real-world sexual violence [13]. In this way, patterns of behavior are shaped by experience and personalized to new interactions such that users become “socialized” to extremism [30].

While some people enter social media already having

extremist tendencies, other users become radicalized by virtue of their interactions, adopting more extreme views over time [35]. In this way, what may appear to be spikes in extremism arise from dynamic group processes that result in an escalation of social conflict [18]. Radicalization unfolds over time with theories commonly highlighting progressive ‘stages,’ ‘pathways,’ or ‘staircases’ that lead to extremism [11, 15, 21]. Despite the fact that radicalization is inherently a process, little research has empirically examined dynamics of online radicalization. Moreover, research is needed to examine how low-level multimodal comment patterns, as opposed to word and topic usage, are associated with radicalization [22].

We examine how the dynamics of low-level comment

features change as a function of engagement with an extremist Reddit forum (subreddit) that has been described by mainstream media as perpetuating misogynistic material and ‘quarantined’ for offensive content against women. 1 Multidimensional recurrence quantification analysis, MdRQA [34], with a pattern analysis extension developed by the authors was used to understand how multimodal communication patterns (across multiple streams of comment features) are linked to frequency of commenting on the extremist subreddit. We also examine how multimodal patterns evolve as a function of subreddit engagement and how specific comment patterns are associated with user dynamics. Consistent with research that highlights indicators of radicalization, such as rigid attitudes [18], we predicted that high-frequency users of the extremist subreddit would exhibit increased regularity in their multimodal comment patterns over time compared to other users.

2. Related Work

According to Sunstein [32], for a well-functioning social system to thrive, people must encounter information that they would not have chosen themselves and people from diverse backgrounds should have shared experiences to support mutual feelings of understanding. However, the potential benefits of social forums are limited by echo chambers, where extremism and radicalization thrive, as people seek information, encounters, and communities that reinforce a set of attitudes and behaviors [7]. These dynamics are especially insidious when members become less diverse over time and more coherent, such that discussions are one-sided [32]. The appearance of solidarity within an online community builds confidence in a particular set of beliefs and emboldens group members, creating a “breeding ground” for extremism [32] with potential for increased bigotry, distress, civil unrest, violence, and decreased capacity for decision-making due to misinformation. The scope of the issue is illustrated by the recent move by Reddit to ban approximately 2,000 of their subreddits over chronically abusive language, some of which included as many as 800,000 users [4].

Identifying indicators of emerging extremism is a priority for

counter-extremist agencies [11], but methods for detecting radicalization material remain imprecise. Attempts to understand patterns and emergent processes underlying extremism and radicalization in social media have been limited [11]. In particular, there is a need for computationally sound and interpretable models to facilitate understanding of intensive longitudinal data associated with radicalization on social media

platforms. Put simply, understanding the mechanisms of radicalization must go beyond studying content to examine process [10].

Research on online radicalization has expanded in recent

years, however, several challenges remain. First, although theories have sought to explain how radicalization occurs, only a small number of articles focused on real-world dynamics of radicalization [11]. Given that radicalization is a “process,” understanding user dynamics or change over time is key to understanding radicalization and supporting early detection and intervention. Our work complements prior research by focusing on emergent patterns and dynamics of online radicalization.

Second, despite the wealth of data available from extremist

online communities, research on radicalization has not focused much on discourse [8, 17]. Moreover, research on radicalization discourse that centers on word and topic detection [25] is undoubtedly important to the field but has its limitations. As outlined by Mondal and colleagues [22], natural language processing (NLP) in hate speech detection is challenged by the amount of well-labeled data needed for machine and deep learning. This is especially true given that this approach typically relies on manual labeling, which is not scalable to large-scale cohorts [1] and human coders oftentimes have a difficult time identifying hate speech, such as sexism against women [31]. Moreover, NLP and classification of extremist content is hindered by short and noisy content [2]. For these reasons, Mondal and colleagues [22] proposed using a sentence-based approach that does not rely on the keywords. Models that relate low-level comment patterns to high-level user outcomes may be more generalizable compared to models that center on word and topic detection alone, as low-level patterns may be similar across topics, forums, and languages. In turn, we examine low-level multimodal patterns of social media posts (e.g., word count, downvotes and upvotes, subreddit where comment was posted) and how they evolve over time as users become more-or-less engaged with an extremist online forum.

Lastly, previous approaches for studying online

communication dynamics are typically unimodal and examine only the word content of the message (i.e., one data stream). In contrast, we use a multimodal approach to investigate how the combination of comment features (i.e., multiple data streams) or “patterns” predict engagement with an extremist subreddit over-and-above individual comment features. This is achieved using a novel modeling technique drawn from nonlinear dynamical systems theory (NDST) that quantifies the degree of regularity versus novelty in users’ multimodal comment features, identifying distinct “patterns” that distinguish between users who are more-or-less involved with an extremist subreddit. This approach can be generalized to accommodate categorical or continuous data, as well as a variable number of information streams. Next, we introduce our study method and modeling technique.

3. Modeling Multimodal Dynamics of Online Radicalization

3.1 Data Sampling Radicalization on Reddit

We identified users for inclusion in the sample based on their comment frequency in the extremist subreddit over the last two-years (low, median, and high). Those in the high-frequency group were the 100 most active users, while those in the low-frequency group were the 100 least active users. To identify median-frequency users, the median comment count was calculated and a random sample of 100 users within ten percentile points of that value was taken. The total number of comments made by each user was skewed, with low-frequency users making only a single comment on the subreddit, median-frequency users making between two and five comments, and high-frequency users representing those with hundreds of comments on the extremist subreddit.

Next, we retrieved the two-year comment history of each

user via Reddit’s public API. Each comment included the following metadata: 1) score (total of upvotes and downvotes from other users), 2) verbosity (number of words), and 3) subreddit where it was posted. These features were selected because they can be utilized in any language, are found on most social media platforms, and describe distinct qualities of a user’s online interactions such as the amount of elaboration in the comment or the type of feedback received. Although comment score is provided by other users (not the person commenting), comment score can be considered a proxy measure for efforts of the person commenting to appeal—or even conform—to the preferences of the broader online community. Further, we included the subreddit where each comment was posted to track whether the user was interacting with the extreme community or a more general Reddit audience. Considering theories of radicalization that emphasize social mechanisms like group polarization and isolation [15], this combination of features provides insight into how a user’s behaviors, and the social feedback they receive, adapt alongside daily engagement with Reddit communities of varying ideological backgrounds.

Comment features were aggregated per day (i.e., daily

average score and verbosity) to obtain a unified sampling rate, excluding days during which a user did not comment. To account for individual differences, time series were z-scored for each user. Lastly, these scores were binarized based on their medians to identify low (0) vs. high (1) score and verbosity. In addition, the parent subreddit of each comment was included and binarized depending on whether the user had participated with the extreme subreddit that day, Comments outside extremist subreddit (0) vs. Comments inside extremist subreddit (1). Days where users had no comments on Reddit were excluded, comments deleted by the user or Reddit were removed, and users with less than an average of one comment per week for the two-year period were excluded from the analysis. Finally, outliers in the remaining data were identified using a common method based on the interquartile range. Nine users were removed due to outliers in the number of comments made, six due to outliers in recurrence, and one due to average comment score, resulting in a final sample of 144 participants.

3.2 Multidimensional Recurrence Quantification Analysis

Nonlinear dynamical systems theory (NDST) describes the temporal evolution and emergent patterns of behavior commonly observed within a system (e.g., social network) and can be used to make predictions about system behaviors and outcomes [26]. Recurrence quantification analysis (RQA) is one NDST tool that quantifies the temporal organization of information streams by identifying repeat values over time. In doing so, RQA captures dynamic shifts between regular and novel patterns of behavior and is commonly used to identify repetitive behaviors during social interactions [14]. MdRQA [34] is a modification of RQA that quantifies the collective organization of multiple signals, including time series that represent different people or modalities. “Recurrence” in the context of MdRQA refers to periods during which the multidimensional system revisits states, though the individual information streams might not align. That is, a recurrent (regular) pattern might involve high extremist subreddit usage, low score, and high verbosity at time points 1 and 2.

MdRQA transforms multiple time series into a distance

matrix representing the Euclidean distances between data points at all possible time lags in the multidimensional time series. Next, a radius parameter is applied to change the distance matrix into a recurrence matrix, where distances smaller than the selected radius are considered to represent sufficiently similar values and recoded to a value of 1 (recurrent point) and distances larger than the radius are considered dissimilar and are recoded to 0 (non-recurrent point). For example, a pattern that occurs at timepoint 1 might repeat at timepoint 2 (recurrent) but not timepoint 3 (non-recurrent).

A recurrent point is one at which the signals collectively

return to the same state as they were in previously. The radius is held constant across all participants to allow for comparison across participants, and the procedure for selecting the radius in human-subjects research is widely accepted [6]. Currently, we binarized time series to be categorical (i.e., high versus low for each signal) to highlight distinct multimodal patterns and, therefore, set the radius very low (r = .0001) such that only exact matches were identified as recurrent.

Figure 1: Illustration of MdRQA procedure. The left panel depicts the time series that were binarized and transformed into a distance matrix (middle panel), where darker shades refer to multimodal points in time that are similar to one another at different time lags. The main diagonal is solid because all time points are recurrent with themselves at lag 0. Diagonals parallel to the main diagonal represent distances between time series elements at a progressively greater time lags. Via pattern analysis, recurrent points (right panel) can then be color-coded to depict specific patterns that repeat, in this case, combinations of low (L) and high (H) levels of main subreddit activity, score, and verbosity. Here, we see a user frequently engaged with the extremist subreddit (red hues) with a temporary shift toward low engagement with the extremist subreddit (blue hues).

Among the more straightforward measures derived from

RQA is recurrence rate—which we refer to as “comment regularity”—or the percentage of recurrent points in the matrix. In contrast, “comment novelty” refers to the degree of non-recurrent points identified via MdRQA. We also present an extension of RQA in the form of pattern analysis, which differentiates between individual patterns of recurrence. In this case, there are eight possible patterns based on the binarization of the three channels and their combination (e.g., high or low score, high or low comment word count, and primary or non-primary subreddit post). Figure 1 (right) illustrates color-coded recurrent points with respect to the combination of three binarized channels (whether a comment was made to the extremist subreddit, score, and verbosity). Readers can refer to our OSF page for an elaborated introduction to MdRQA.2

4. Results

We investigated patterns of online radicalization using a multimodal approach to characterize dynamics of low-level comment features. Collective daily patterns of behavior consisting of comment score (M = 5.93, SD = 3.75), verbosity (M = 40.26, SD = 30.24), and the subreddit where comments were posted (inside or outside extremist subreddit) were calculated using MdRQA. Users’ resulting recurrence rates ranged from 12.89% to 30.15% (M = 21.37, SD = 4.63), with higher recurrence indicating greater comment regularity in collective commenting patterns. Next, we 1) compare comment regularity of users based on their frequency of activity in the extremist subreddit, 2) demonstrate a novel method for identifying distinct patterns of user behaviors, and 3) examine how comment regularity changes over time. Refer to our OSF page for additional details of the results.2

4.1 High-Frequency Extremist Subreddit Users Had Less Comment Regularity

First, we used an ANCOVA model to compare high-, median-, and low-frequency users’ comment regularity. We entered users’ average comment score, average word count, and total comment count as covariates. This allowed us to examine the relationship between multimodal comment regularity and frequency of using the extremist subreddit, controlling for more common unimodal measures. We found that a users’ comment regularity was significantly associated with their level of activity on the extremist subreddit F(2,138) = 137.65, p <.001. Tukey post-hoc tests revealed that average comment regularity for high-frequency users (M = 17.35, SD = 3.37, n = 70) was significantly lower than median- (M = 25.11, SD = 1.14, n = 42, t = -14.77, p <.001) and low-frequency users (M = 25.23, SD = .97, n = 32, t = - 13.77, p <.001). Users who more frequently posted in the extremist subreddit were more novel (or less regular) in their comment patterns.

4.2 High-Frequency Extremist Subreddit Users Had Distinct Comment Patterns

The percentage of user comments was calculated for each of the eight possible multimodal patterns, representing the overall frequency that each individual pattern occurred per participant. Per Figure 1, an “HHH” pattern would refer high subreddit activity, high score, and high verbosity. Pearson correlations were used to examine the degree users’ percentage of specific comment patterns corresponded to overall comment regularity. We found that HHH, HHL, and HLH patterns were negatively correlated with comment regularity (rs = -.53 - -.30), whereas LHH, LHL, and LLH patterns were positively associated with regularity (rs = .21 - .64). In other words, H1-patterns (patterns involving posts within the extremist subreddit) were negatively associated with comment regularity. These patterns indicate that users who shift their activity more exclusively to the extremist subreddit are relatively non-repetitive in their score and verbosity. Patterns with low score and word count (HLL) were not related to regularity, p > .05. The correlation matrix for these data can be found on OSF.2

Figure 2: Example recurrence matrices for low-, median-, and high-frequency participants. Here we highlight two of the eight multimodal comment patterns: Blue points correspond to high score and verbosity outside of the extremist subreddit. Red points correspond to high score and verbosity within the extremist subreddit. The high-frequency user (bottom) demonstrates a qualitative shift to increased use of the extremist subreddit.

4.3 Comment Regularity Decreased for High-Frequency Extremist Subreddit Users

To examine how user comment patterns changed over time, we divided each user’s daily data into equally sized windows. Comment regularity was re-calculated within each window so that users had five data points representing their commenting regularity within each time window. Next, we fit mixed-effects models for each of the three user groups with regularity as the outcome, linear and quadratic time as fixed effects, and user ID as a random intercept. P-values were adjusted for multiple comparisons [16].

Comment regularity significantly decreased over time for

high-frequency users (β = -2.38, p <.001; Figure 3) but not low (β = .71, p = .80) or medium-frequency users (β = .10, p = 1.00). Model intercepts demonstrate that high-frequency users also had lower baseline regularity (β = 21.23, p <.001) than low (β = 27.57, p < .001) or medium-frequency users (β = 27.24, p <.001). The findings indicate that users who most frequently engage with the extremist subreddit not only begin with less comment regularity but also become less regular (and more novel) over time.

Figure 3: Panel A depicts recurrence rate grouped by the frequency of comments made to the extremist subreddit (low, median, high). Panel B shows the trajectory of recurrence for high frequency users of the extremist subreddit.

5. Discussion

We found that the most frequent users of the extremist subreddit exhibited less regularity in multimodal comment patterns. High-frequency users also became more novel in their comment features over time, whereas low- and median-frequency users exhibited relatively stable comment patterns. Further analysis of the comment patterns indicates that high-frequency users of the extremist subreddit were repetitive in being devoted commenters of the extremist subreddit, versus being generally devoted Reddit users and posting on other subreddits frequently. However, high-frequency users were less repetitive in their other low-level comment features of score and verbosity compared to other users.

Although contrary to our hypothesis that high-frequency

users would exhibit increasingly rigid commenting patterns, the findings are consistent with Sunstein’s [32] conceptualization of the societal effects of echo chambers, where he notes that frequent one-sided discussions in extremist forums tend to build confidence and embolden group members. In this way, high-frequency users of the extremist subreddit may have less of a filter when deciding when and how to respond, lending to more variable comment features and scores. High-frequency users may also be more likely to engage in inflammatory discussions with other users, which is suggested by the greater range in scores and length of comments, as well as longer average comment length.

Our findings are especially consistent with Myers and

Lamm’s [24] classic theory of opinion polarization, where belief systems are solidified through three stages: social motivation, action commitment, and cognitive foundation. First, all sampled users demonstrated some degree of social motivation by virtue of their engagement with social media, including the extremist subreddit. Second, in the action commitment phase, increased information processing and verbalization of a particular opinion solidifies one’s belief system and makes it less vulnerable to change. High-frequency users of the extremist subreddit demonstrated action commitment through favoring the

extremist subreddit over alternative subreddits. In other words, rather than being generally active Reddit users who sometimes engaged with the extremist subreddit, the high-frequency users shifted their attention to the extremist subreddit. Third, the cognitive foundation phase relates to in-depth cognitive processing and rehearsal of relevant arguments [28]. This phase relates to the less repetitive patterns in high-frequency users’ verbosity and scores from other users, where these users are increasingly ‘rehearsing’ information (i.e., producing more novel comment structures) and justifying their beliefs (e.g., through increased discussion). Notably, only the high-frequency users of the extremist subreddit exhibited significant changes in comment patterns over time, consistent with the idea that they became more “radicalized.” In summary, low-level comment features and their dynamics are meaningfully associated with markers of radicalization.

The present study has several limitations and future

directions. First, considering the abundancy of data on Reddit, our final sample was relatively small. A large number of users were excluded from the analysis because their activity on Reddit was limited and they did not consistently make comments (see § 3.1). In several cases, users were also excluded because a substantial proportion of their comment history was inaccessible after moderation by Reddit. Follow-up studies would benefit from a significantly larger sample size. Additionally, our initial approach focuses on the dynamics of low-level comment features and user interactions with an extreme community, but we do not consider how properties of the social media website may have contributed to the radicalization process. Further research is needed to determine the extent to which our findings are upheld across various online forums and communities. Finally, future research should examine the predictive validity of different data streams individually and collectively in their ability to identify users engaged in extremist and radicalized belief systems.

[^1]: To enhance the anonymity of users we withhold the name of the specific subreddit sampled. This was necessary for the current project, which samples based on frequency of subreddit engagement during a specific time period.

[^2]: Link to OSF: https://osf.io/agj2b/?viewonly=a79c04750bf94a628e8f1ebd4b65f80a

References

[1] Swati Agarwal and Ashish Sureka. 2014. A focused crawler for mining hate and extremism promoting videos on YouTube. In Proceedings of the 25th ACM conference on Hypertext and social media (HT ’14), Association for Computing Machinery, New York, NY, USA, 294–296. DOI:https://doi.org/10.1145/2631775.2631776

[2] Swati Agarwal and Ashish Sureka. 2015. Using KNN and SVM Based One- Class Classifier for Detecting Online Radicalization on Twitter. In Distributed Computing and Internet Technology (Lecture Notes in Computer Science), Springer International Publishing, Cham, 431–442. DOI:https://doi.org/10.1007/978-3-319-14977-647

[3] Swati Agarwal and Ashish Sureka. 2015. Applying Social Media Intelligence for Predicting and Identifying On-line Radicalization and Civil Unrest Oriented Threats. ArXiv151106858 Cs, November 2015. Retrieved from http://arxiv.org/abs/1511.06858

[4] Bobby Allyn. Reddit Bans TheDonald, Forum Of Nearly 800,000 Trump Fans, Over Abusive Posts. 2020. NPR.org. Retrieved from https://www.npr.org/2020/06/29/884819923/reddit-bans-thedonald-forum-of-nearly-800-000-trump-fans-over-abusive-posts

[5] Luke Bertram. 2016. Terrorism, the Internet and the Social Media Advantage: Exploring how terrorist organizations exploit aspects of the internet, social media and how these same platforms could be used to counter-violent extremism. J. Deradicalization 7, June 2016, 225–252.

[6] Moreno Coco and Rick Dale. 2014. Cross-recurrence quantification analysis of categorical and continuous time series: an R package. Front. Psychol. 5, (2014), 510. DOI:https://doi.org/10.3389/fpsyg.2014.00510

[7] Susan Collins and Joseph I. Lieberman. 2008. Violent Islamist Extremism, the Internet, and the Homegrown Terrorist Threat. United States. Congress. Senate. Committee on Homeland Security and Governmental Affairs. Retrieved from https://www.hsdl.org/?abstract&did=

[8] Maral Dadvar, Dolf Trieschnigg, Roeland Ordelman, and Franciska de Jong. 2013. Improving Cyberbullying Detection with User Context. In Advances in Information Retrieval (Lecture Notes in Computer Science), Springer, Berlin, Heidelberg, 693–696. DOI:https://doi.org/10.1007/978-3-642-36973-562

[9] Geoff Dean. 2019. Framing the Challenges of Online Violent Extremism: “Policing-Public-Policies-Politics” Framework. In Violent Extremism: Breakthroughs in Research and Practice. IGI Global, 226–259. DOI:https://doi.org/10.4018/978-1-5225-7119-3.ch017

[10] Charlie Edwards and Luke Gribbon. 2013. Pathways to Violent Extremism in the Digital Era. RUSI J. 158, 5, October 2013, 40–47. DOI:https://doi.org/10.1080/03071847.2013.847714

[11] Miriam Fernandez, Antonio Gonzalez-Pardo, and Harith Alani. 2019. Radicalisation Influence in Social Media. J. Web Sci. 6, 2019, 1–15.

[12] Jesse Fox, Carlos Cruz, and Ji Young Lee. 2015. Perpetuating online sexism offline: Anonymity, interactivity, and the effects of sexist hashtags on social media. Comput. Hum. Behav. 52, November 2015, 436–442. DOI:https://doi.org/10.1016/j.chb.2015.06.024

[13] Rachael Fulper, Giovanni Luca Ciampaglia, Emilio Ferrara, Yeol Ahn, Alessandro Flammini, Filippo Menczer, Bryce Lewis, and Kehontas Rowe. 2014. Misogynistic Language on Twitter and Sexual Violence. 1–4.

[14] Riccardo Fusaroli and Kristian Tylén. 2016. Investigating Conversational Dynamics: Interactive Alignment, Interpersonal Synergy, and Collective Task Performance. Cogn. Sci. 40, 1, January 2016, 145–171. DOI:https://doi.org/10.1111/cogs.12251

[15] Keiran Hardy. 2018. Comparing Theories of Radicalisation with Countering Violent Extremism Policy. J. Deradicalization 15, June 2018, 76–110.

[16] Sture Holm. 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scand. J. Stat. 6, 2 ,1979, 65–70.

[17] Daniel Karell and Michael Freedman. 2019. Rhetorics of Radicalism. Am. Sociol. Rev. 84, 4, August 2019, 726–753. DOI:https://doi.org/10.1177/0003122419859519

[18] Kim Knott, Ben Lee, and Simon Copeland. 2018. Reciprocal Radicalisation. Center for Research and Evidence on Security Threats. Retrieved July 18, 2021 from https://crestresearch.ac.uk/resources/reciprocal-radicalisation/

[19] Daniel Koehler. 2014. The Radical Online: Individual Radicalization Processes and the Role of the Internet. J. Deradicalization, 1, (December 2014), 116–134.

[20] Annette Markham and Elizabeth Buchanan. 2012. Recommendations from the AoIR Ethics Working Committee. Retrieved from https://aoir.org/reports/ethics2.pdf

[21] Fathali Moghaddam. 2005. The Staircase to Terrorism: A Psychological Exploration. Am. Psychol. 60, 2 (2005), 161–169. DOI:https://doi.org/10.1037/0003-066X.60.2.161

[22] Mainack Mondal, Leandro Araújo Silva, and Fabrício Benevenuto. 2017. A Measurement Study of Hate Speech in Social Media. In Proceedings of the 28th ACM Conference on Hypertext and Social Media (HT ’17), Association for Computing Machinery, New York, NY, USA, 85–94. DOI:https://doi.org/10.1145/3078714.3078723

[23] Pardis Moslemzadeh Tehrani, Nazura Abdul Manap, and Hossein Taji. 2013. Cyber terrorism challenges: The need for a global response to a multi-jurisdictional crime. Comput. Law Secur. Rev. 29, 3 (June 2013), 207–215. DOI:https://doi.org/10.1016/j.clsr.2013.03.011

[24] David Myers and Helmut Lamm. 1976. The Group Polarization Phenomenon. Psychol. Bull. 83, 4 (1976), 602–627.

[25] Amal Rekik, Salma Jamoussi, and Abdelmajid Ben Hamadou. 2019. Violent Vocabulary Extraction Methodology: Application to the Radicalism Detection on Social Media. In Computational Collective Intelligence (Lecture Notes in Computer Science), Springer International Publishing, Cham, 97–109. DOI:https://doi.org/10.1007/978-3-030-28374-29

[26] Michael Riley and John Holden. 2012. Dynamics of cognition. WIREs Cogn. Sci. 3, 6 (2012), 593–606. DOI:https://doi.org/10.1002/wcs.1200

[27] Hannah Rogan. 2006. Jihadism online - A study of how al-Qaida and radical Islamist groups use the Internet for terrorist purposes. Norwegian Defence Research Establishment. Retrieved from https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.107.9729&rep=rep1 &type=pdf

[28] Agnieszka Rychwalska and Magdalena Roszczyńska-Kurasińska. 2018. Polarization on Social Media: When Group Dynamics Leads to Societal Divides. In Proceedings of the 51st Hawaii International Conference on System Sciences, 2088–2097. DOI:https://doi.org/10.24251/HICSS.2018.263

[29] Gerard Saucier, Laura Geuy Akers, Seraphine Shen-Miller, Goran Kneževié, and Lazar Stankov. 2009. Patterns of Thinking in Militant Extremism. Perspect. Psychol. Sci. J. Assoc. Psychol. Sci. 4, 3 (2009), 256–271. DOI:https://doi.org/10.1111/j.1745-6924.2009.01123.x

[30] Alex P Schmid. 2016. Research on Radicalisation: Topics and Themes. Perspect. Terror. 10, 3 (2016), 26–32.

[31] Sima Sharifirad and Stan Matwin. 2019. When a Tweet is Actually Sexist. A more Comprehensive Classification of Different Online Harassment Categories and The Challenges in NLP. arXiv:1902.10584 (February 2019). Retrieved July 18, 2021 from http://arxiv.org/abs/1902.10584

[32] Cass Sunstein. 2007. Republic.com 2.0. Princeton: Princeton University Press. Retrieved from http://www.jstor.org/stable/10.2307/j.ctt7tbsw

[33] Ines Von Behr, Anais Reding, Charlie Edwards, and Luke Gribbon. 2013. Radicalisation in the digital era: The use of the internet in 15 cases of terrorism and extremism. RAND Europe, Brussels, Belgium.

[34] Sebastian Wallot, Andreas Roepstorff, and Dan Mønster. 2016. Multidimensional Recurrence Quantification Analysis (MdRQA) for the Analysis of Multidimensional Time-Series: A Software Implementation in MATLAB and Its Application to Group-Level Data in Joint Action. Front. Psychol. 7, (2016). DOI:https://doi.org/10.3389/fpsyg.2016.01835

[35] Online radicalisation to violent extremism: Awareness brief. Community Oriented Policing (COPS), Washington, DC. Retrieved from https://www.theiacp.org/sites/default/files/2018-07/RadicalizationtoViolentExtremismAwarenessBrief.pdf