This paper is in the following e-collection/theme issue:

Infodemic Management (44) Vaccination and Immunization in the Digital Age (347) Natural Language Processing (650) Infoveillance, Infodemiology, Digital Disease Surveillance, Infodemic Management (1015) Infodemiology and Infoveillance (1278) Discourse (38)

Published on in Vol 2, No 2 (2022): Jul-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/37635, first published .
Emotions and Incivility in Vaccine Mandate Discourse: Natural Language Processing Insights

Emotions and Incivility in Vaccine Mandate Discourse: Natural Language Processing Insights

Emotions and Incivility in Vaccine Mandate Discourse: Natural Language Processing Insights

Authors of this article:

Hannah Stevens1 Author Orcid Image ;   Muhammad Ehab Rasul1 Author Orcid Image ;   Yoo Jung Oh1 Author Orcid Image
Article Authors Cited by (7) Tweetations (14) Metrics

    Original Paper

    University of California, Davis, Davis, CA, United States

    Corresponding Author:

    Hannah Stevens, BA

    University of California, Davis

    1 Shields Ave

    Davis, CA, 95616

    United States

    Phone: 1 530 752 0966

    Email: hrstevens@ucdavis.edu


    Background: Despite vaccine availability, vaccine hesitancy has inhibited public health officials’ efforts to mitigate the COVID-19 pandemic in the United States. Although some US elected officials have responded by issuing vaccine mandates, others have amplified vaccine hesitancy by broadcasting messages that minimize vaccine efficacy. The politically polarized nature of COVID-19 information on social media has given rise to incivility, wherein health attitudes often hinge more on political ideology than science.

    Objective: To the best of our knowledge, incivility has not been studied in the context of discourse regarding COVID-19 vaccines and mandates. Specifically, there is little focus on the psychological processes that elicit uncivil vaccine discourse and behaviors. Thus, we investigated 3 psychological processes theorized to predict discourse incivility—namely, anxiety, anger, and sadness.

    Methods: We used 2 different natural language processing approaches: (1) the Linguistic Inquiry and Word Count computational tool and (2) the Google Perspective application programming interface (API) to analyze a data set of 8014 tweets containing terms related to COVID-19 vaccine mandates from September 14, 2021, to October 1, 2021. To collect the tweets, we used the Twitter API Tweet Downloader Tool (version 2). Subsequently, we filtered through a data set of 375,000 vaccine-related tweets using keywords to extract tweets explicitly focused on vaccine mandates. We relied on the Linguistic Inquiry and Word Count computational tool to measure the valence of linguistic anger, sadness, and anxiety in the tweets. To measure dimensions of post incivility, we used the Google Perspective API.

    Results: This study resolved discrepant operationalizations of incivility by introducing incivility as a multifaceted construct and explored the distinct emotional processes underlying 5 dimensions of discourse incivility. The findings revealed that 3 types of emotions—anxiety, anger, and sadness—were uniquely associated with dimensions of incivility (eg, toxicity, severe toxicity, insult, profanity, threat, and identity attacks). Specifically, the results showed that anger was significantly positively associated with all dimensions of incivility (all P<.001), whereas sadness was significantly positively related to threat (P=.04). Conversely, anxiety was significantly negatively associated with identity attack (P=.03) and profanity (P=.02).

    Conclusions: The results suggest that our multidimensional approach to incivility is a promising alternative to understanding and intervening in the psychological processes underlying uncivil vaccine discourse. Understanding specific emotions that can increase or decrease incivility such as anxiety, anger, and sadness can enable researchers and public health professionals to develop effective interventions against uncivil vaccine discourse. Given the need for real-time monitoring and automated responses to the spread of health information and misinformation on the web, social media platforms can harness the Google Perspective API to offer users immediate, automated feedback when it detects that a comment is uncivil.

    JMIR Infodemiology 2022;2(2):e37635

    doi:10.2196/37635

    Keywords

    vaccine hesitancy; COVID-19; vaccine mandates; natural language processing; incivility; LIWC; Linguistic Inquiry and Word Count; Twitter 


    Background

    The emergence of the novel coronavirus (COVID-19) has caused 5,878,328 confirmed deaths worldwide as of February 2022, along with 423,437,674 confirmed infections [1]. Despite vaccine availability, vaccine hesitancy has inhibited public health officials’ efforts to mitigate the COVID-19 pandemic, especially in the United States [2]. Although some US elected officials have responded by issuing vaccine mandates, others have amplified vaccine hesitancy by broadcasting messages that minimize vaccine efficacy [3,4].

    With 68% of American adults reporting social media as a source of their news diet [5], social media platforms such as Twitter have become important communication channels for US politicians to share their agendas [6]. As a result, social media have become a prominent source of political information and misinformation, including information surrounding COVID-19 vaccines [7-11]. The politically polarized nature of COVID-19 information on social media has given rise to an infodemic, wherein health attitudes often hinge more on political ideology than science [12-15]. As a result, political affiliation influences negative sentiment toward the vaccine [16]. Such negative sentiment may foster uncivil discourse toward the vaccines and mandates [17,18].

    Incivility on social media platforms has been widely studied and discussed in both political and health contexts, among others [19-25]. However, to the best of our knowledge, incivility has not been studied in the context of discourse regarding COVID-19 vaccines and mandates. Specifically, there is little focus on the psychological processes that elicit uncivil vaccine discourse. We aimed to bridge this gap by conducting a computational analysis of tweets. In this study, we investigated the role of negative emotion in predicting uncivil posts about COVID-19 vaccine mandates on Twitter. Ultimately, we argue that a more nuanced understanding of the psychological processes underlying uncivil vaccine discourse has practical implications for public health interventions.

    The Role of Negative Emotion in Vaccine Mandate Incivility

    Incivility has become a salient point of discussion in social media research. However, scholars across fields have found it difficult to conceptualize incivility. Incivility has been defined in a variety of ways, including impoliteness, profanity, and specific discriminatory acts (eg, former US president Trump caught on a hot mic in 2016 praising nonconsensual sexual encounters with women) [26-29]. Papacharissi [29] supplements this definition by including threat—in this case to democracy—as uncivil. Other scholars have operationalized incivility as including the use of all capital letters, accusations of lying, pejorative speech, ideologically extreme language, exaggerated argument, and misinformation [26,30-33]. Despite these inconsistent operationalizations, incivility is a concept that is nuanced and varies across individuals, perhaps because it is bound by cultural perceptions and understandings of what uncivil discourse is [16,18]. Informed by the operational inconsistency of incivility outlined in the literature, we conceptualize incivility as a multifaceted construct encompassing a diversity of uncivil behaviors, including toxicity, severe toxicity, profanity, threats, insults, and identity attacks in discourse. Recent studies have argued that uncivil behaviors are related to toxicity on social media platforms [34]. Tromble [28] asserts that profanity and insulting language constitute key indicators of uncivil behaviors. Likewise, scholars have argued that identity attacks and threatening language that aims to morally attack individuals or groups are also aspects of incivility and uncivil discourse [35]. We now shift our attention to explaining what causes incivility.

    Incivility does not have a single cause; instead, varying forms of uncivil behaviors are a result of diverse psychological processes. For example, a user may post profane content because they are anxious, whereas a user might make an insulting comment because they are angry. However, scholars often obscure these distinct underlying psychological mechanisms by conceptualizing incivility as a one-dimensional process with a unitary explanation [19,21]. In the context of COVID-19 vaccines and mandates, emotional responses such as anger and anxiety among other negative emotions are salient in the discourse about the pandemic [36,37]. In fact, studies have found negative emotions such as anger and anxiety to play a role in driving vaccine hesitancy [38]. We investigated 3 psychological processes that are likely to predict discourse incivility—namely, anxiety, anger, and sadness.

    Anxiety and Incivility

    Anxiety about the safety of the COVID-19 vaccine, paired with dismissive attitudes toward COVID-19’s threat, has a sizable segment of the United States indicating their unwillingness to get vaccinated [38-40]. In line with extant theory asserting that fear-based aggression is the most prevalent when a perceived threat is inescapable [41-43], a fear of harm from the vaccine, as perpetuated by elected officials and media alike, is often followed by avoidance strategies (eg, refusing the vaccine) [9-11,44]. Accordingly, policies that mandate the hesitant to get vaccinated inhibit the ability to escape the threat, and as a result, individuals may react with incivility. Indeed, stress and anxiety have been demonstrated to predict a wealth of uncivil behaviors, including cyber aggression and bullying during COVID-19 [45-47]. Thus, we posit the following.

    Hypothesis (H) 1: Anxiety will positively predict post incivility.
    Anger and Incivility

    COVID-19 vaccine mandates have drawn the ire of segments of the United States, including political elites and media outlets who have fueled public outrage about the threat to personal freedoms that vaccine mandates impose [48,49]. Simultaneously, the lack of confidence in vaccine safety and efficacy has segments of the population feeling threatened by the health risks they perceive to be associated with the vaccine. Anger can be understood as an adaptive response to a threat [44]; indeed, a study by Featherstone and Zhang [44] found vaccine misinformation to negatively impact attitudes toward vaccines through anger. Although anger has the functional value of suppressing fear and potentiating a sense of personal control in the face of threat, it can also propel uncivil behavior, including acts of aggression and dismissiveness directed toward those with opposing views [50-52]. Thus, we can expect anger to foster incivility in COVID-19 vaccine mandate discourse.

    H2: Anger will positively predict incivility.
    Sadness and Incivility

    Feelings of sadness have been linked with uncivil behavior, including acts of cyber aggression [47,53]. The freedom to travel, remain employed, socialize in groups, eat in restaurants, go to the gym, and more is increasingly determined by one’s vaccination status [54,55]. Thus, mandates that prohibit the unvaccinated from participating in the relationships and activities available to those who are vaccinated may exacerbate existing sadness and depression induced by preexisting COVID-19 lifestyle disruptors [56,57]. Furthermore, social exclusion can elicit sadness and feelings that a group (ie, the unvaccinated) has experienced wrongs that must be righted—a mindset political scientists have coined “victimhood” [58]. Victimhood mentality may prompt individuals to retaliate against vaccine mandates and manifest as uncivil behaviors. Accordingly, we predict the following.

    H3: Sadness will positively predict incivility.

    Data Collection

    The sample comprised posts shared to Twitter, a popular platform for seeking and sharing health information on the web, including (mis)information about vaccination and vaccines [7-11]. We opted to curate a list of vaccine-related words and scraped tweets containing those words. We curated a list of words that we believed would collect tweets related to the vaccine, without introducing bias into the data set. For example, “shot” was not included, because we noticed that it scraped tweets about gunshots, which are unrelated to the COVID-19 vaccine. The text of the 8014 tweets contained terms related to COVID-19 vaccine mandates (eg, “Moderna,” “required,” and “mandating”) from September 14, 2020, to October 1, 2021. See Figure 1 for a flowchart of the data collection process.

    Twitter’s code-free application programming interface (API) Tweet Downloader Tool (version 2) was used to extract posts about COVID-19 vaccine mandates. We were interested in words that would identify tweets about COVID-19 vaccine mandates rather than the COVID-19 vaccine generally. Thus, we filtered through a data set of 375,000 vaccine-related tweets posted from September 14, 2020, to October 1, 2021, to extrapolate tweets specifically related to vaccine mandates (eg, “forcing,” “required,” and “mandating”) from September 14, 2020, to October 1, 2021; the final sample contained 8014 tweets.

    ‎
    Figure 1. Flowchart of the data collection process.
    View this figure

    Natural Language Processing Procedures

    The data were analyzed using 2 different natural language processing approaches: (1) the Linguistic Inquiry and Word Count (LIWC) computational tool [59] and (2) the Google Perspective API [60].

    LIWC Sentiment Analysis

    LIWC is a natural language processing tool that measures psychological processes in texts by counting the percentage of words in a given tweet that fall into prespecified categories. It has been validated and used in investigations of mental health during the COVID-19 pandemic (eg, LGBTQ+ youth mental health) [12,61]. In contrast to other sentiment analysis lexicons that generate the valence of emotion (eg, Afinn and Bing, which assign texts a score from negative to positive) without extrapolating discrete emotions and sentiment analysis lexicons that produce binary outcomes (eg, NRC), we wanted a continuous measure of the extent to which texts had a particular sentiment [62]. Although there are multiple tools that continuously capture sentiment and emotions using natural language processing methods (eg, IBM Watson) [63], we specifically used the LIWC dictionary for emotion classification, because compared to the aforementioned natural language processing tools, the LIWC dictionary has been validated in multiple studies, and thus, we considered that it would present a more accurate estimate of the level of emotions reflected in the textual data. We leveraged LIWC to measure the valence of linguistic anger (eg, “frustrated,” and “annoyed”), sadness (eg, “hopeless,” and “miserable”), and anxiety (eg, “afraid,” and “stressed”) in texts [59]. Tweets had an average anxiety score of 0.79 (SD 1.67), an average anger score of 0.11 (SD 0.75) and an average sadness score of 0.09 (SD 0.52).

    Google Perspective API Machine Learning Analysis

    To measure dimensions of post incivility, we used the Google Perspective API to measure levels of toxicity, severe toxicity, insult, profanity, threat, and identity attacks in tweets related to vaccine mandates (see Table 1) [60]. The Google Perspective API is a tool designed by Google’s Counter-Abuse Technology Team that measures incivility in web-based posts.

    The Google Perspective API model is trained by human coders on a data set of millions of comments from a variety of web-based sources, including forums (eg, Wikipedia). The model is robust and has been used in a variety of contexts, from political incivility to rape culture to COVID-19 vaccine information [21,64,65]. For example, Hopp et al [64] asked respondents to self-report the degree to which they engage in uncivil communication on the web and then correlated that with trace data of participants’ social media content. The results indicated that those who self-disclose engaging in uncivil social media behavior also tend to generate uncivil content on social media, measured via the Google Perspective API. These dimensions of incivility have been tested across multiple domains and trained on substantial amounts of human-annotated comments [60].

    Table 1. Incivility variable attributes.
    Attribute namePerspective APIa description [60]Example postb
    Severe toxicity“A very hateful, aggressive, disrespectful comment or otherwise very likely to make a user leave a discussion or give up on sharing their perspective.”“F*ck the vaccine and f*ck COVID, this should not be required period!!!”
    Identity attack“Negative or hateful comments targeting someone because of their identity.”“DO NOT COMPLY. Screw liberals and their idiotic vaccine mandate.”
    Insult“Insulting, inflammatory, or negative comment towards a person or a group of people.”“Bank accounts are frozen for protesting mandates. How many more vaccines will you take before you wisen up? Wake up you stupid little sheep.”
    Profanity“Swear words, curse words, or other obscene or profane language.”“It must be hard to be a victim of the vaccine mandate. A**holes on the internet FROTH at the F*CKING mouth to dismiss your experience.”
    Threat“Describes an intention to inflict pain, injury, or violence against an individual or group.”“I’ll put a bullet in someone who tries to force my kid to get the vaccine.”

    aAPI: application programming interface.

    bCurse words have been censored to make the table suitable for publication.

    Ethical Considerations

    No personally identifiable information was included in this study. The institutional review board recognizes that the analysis of publicly available data does not constitute human subjects research. This study only used information in the public domain; thus, ethical review and approval was not required.


    Factor Analysis of Dimensions of Uncivil Discourse

    Prior to hypothesis testing, we conducted a repeated measures ANOVA to assess whether to model dimensions of incivility together or separately. The main effect for the within-subjects factor was significant (F4,32052=930.44; P<.001), indicating significant differences among identity attack, insult, profanity, threat, and severe toxicity (see Table 2).

    Tukey comparisons were used to test marginal mean differences in each combination of incivility dimensions. There were significant differences between each combination, except identity attack and profanity (see Table 3). Thus, we concluded that the 5 dimensions of incivility should be assessed separately in the main analysis.

    Table 2. Means table for within-subject variables (N=8014).
    Incivility dimensionMean (SD)
    Severe toxicity0.10 (0.14)
    Identity attack0.12 (0.12)
    Insult0.18 (0.20)
    Profanity0.12 (0.18)
    Threat0.17 (0.15)
    Table 3. The marginal means contrasts for each combination of within-subject variables for the repeated measures ANOVA.
    ContrastDifferenceSEt test (df)P value
    Severe toxicity – identity attack–0.020.001–15.11 (8013)<.001
    Severe toxicity – insult–0.080.001–66.07 (8013)<.001
    Severe toxicity – profanity–0.020.0008–25.79 (8013)<.001
    Severe toxicity – threat–0.060.001–43.18 (8013)<.001
    Identity attack – insult–0.060.002–36.78 (8013)<.001
    Identity attack – profanity–0.0040.002–2.39 (8013).12
    Identity attack – threat–0.050.002–30.34 (8013)<.001
    Insult – profanity0.060.00143.06 (8013)<.001
    Insult – threat0.010.0026.30 (8013)<.001
    Profanity – threat–0.040.002–21.48 (8013)<.001

    Logistic Regression Analyses

    Dichotomizing the Data

    The skewed distribution of the data necessitated that we dichotomize the incivility dimensions for regression. The Google Perspective API recommends flagging a comment as having an attribute if it scores a 0.7 or higher—thus, this value was used to dichotomize the data for logistic regression [60]. Of the 8014 tweets, 53 (0.66%) contained identity attacks, 405 (5.05%) contained insults, 317 (3.96%) contained profanity, 137 (1.71%) contained threats, and 91 (1.14%) contained severe toxicity.

    For hypothesis testing, we conducted 5 logistic regression analyses to assess whether anger, anxiety, and sadness in posts predicted uncivil tweets (see Table 4 and Figure 2). Variance inflation factors for anxiety, sadness, and anger on all dimensions of incivility were less than 1.5, indicating there was not any multicollinearity between our independent variables.

    Table 4. Binary logistic regression results with anxiety, anger, and sadness predicting dimensions of incivility. McFadden R2 was used to calculate model fit.
    VariableOdds ratio (95% CI)BP valueR2χ23
    Threat.0118.78

    		(Intercept)
    		–4.04<.001


    		Anxiety0.88 (0.78-1.01)–.12.06


    		Sadness1.27 (1.02-1.58).24.04


    		Anger1.21 (1.10-1.33).19<.001

    Identity attack.0958.64

    		(Intercept)
    		–5.06<.001


    		Anxiety0.70 (0.50-0.96)–.36.03


    		Sadness1.15 (0.74-1.77).14.54


    		Anger1.59 (1.40-1.80).46<.001

    Profanity.22567.15

    		(Intercept)
    		–3.58<.001


    		Anxiety0.90 (0.81-0.98)–.11.02


    		Sadness1.04 (0.83-1.31).04.75


    		Anger3.27 (2.93-3.67)1.19<.001

    Insult.08258.25

    		(Intercept)
    		–3.13<.001


    		Anxiety1.01 (0.95-1.07).008.79


    		Sadness0.85 (0.67-1.10)–.16.22


    		Anger2.03 (1.85-2.23).71<.001

    Severe toxicity.24239.27

    		(Intercept)
    		–.45<.001


    		Anxiety0.89 (0.75-1.06)–.11.20


    		Sadness1.01 (0.65-1.57).01.96


    		Anger2.37 (2.12-2.66).86<.001

    ‎
    Figure 2. Negative emotion predicting the odds of severe toxicity, threat, profanity, insult, and identity attack. Scores for anger, anxiety, and sadness were computed using the Linguistic Inquiry and Word Count computerized coding tool that measures psychological processes in texts by counting the percentage of words in a given tweet that fall into prespecified categories.
    View this figure
    Anxiety

    We found that the effect of anxiety on identity attack (B=–.36; odds ratio [OR] 0.70; P=.03) and profanity (B=–.11; OR 0.90; P=.02) were significant. However, contrary to our prediction that linguistic anxiety would increase incivility (H1), the results indicated that anxiety decreased the odds of identity attacks and profanity by approximately 30.48% and 10.43%, respectively. The results also reflected a stronger relationship between anxiety and identity attack than profanity. No other significant differences were found.

    Anger

    Consistent with our hypothesis (H2), the effect of anger on all 5 dimensions of incivility was significant (all P<.001). The results revealed that anger predicted the odds of profanity, insult, and severe toxicity to a greater extent than identity attacks and threats. The effect of the anger on threat (B=.19; OR 1.21; P<.001) and identity attack (B=.46; OR 1.59; P<.001) indicated that a 1-unit increase in anger increased the odds of threats by approximately 20.67% and identity attacks by approximately 58.9%. The effect of anger on insult (B=.71; OR 2.03; P<.001) and severe toxicity (B=.86; OR 2.37; P<.001) indicated that an increase in anger increased the odds of insults by approximately 103.15% and severe toxicity by approximately 137.29%. The results indicated that anger increased the odds of profanity the most (approximately 227.49%; B=1.19; OR 3.27; P<.001) when compared to the other 4 dimensions.

    Sadness

    H3 predicted that sadness will be positively associated with the level of incivility expressed in tweets. Our results showed that the effect of sadness on threat was significant (B=.24; OR 1.27; P=.04), indicating that a 1-unit increase in sadness increased the odds of threats by approximately 26.86%. Sadness did not have a significant effect on any other dimension of incivility.


    Principal Findings

    Incivility has been understood as a multifaceted construct, encompassing the breadth of conceptual and operational definitions offered in the literature. This study resolved discrepant operationalizations of incivility by introducing incivility as a multifaceted construct and explored the distinct emotional processes underlying 5 dimensions of discourse incivility. The findings reveal that 3 types of emotions—anxiety, anger, and sadness—were significantly associated with dimensions of incivility. With regard to the relationship between anxiety and incivility, we found that the anxiety was negatively associated with identity attacks and profanity expressed in Twitter posts. Individuals who expressed higher levels of anger were more likely to engage in all 5 dimensions of incivility, including profanity, insults, severe toxicity, identity attacks, and threats. Lastly, our findings revealed that sadness was positively associated with uncivil behavior, especially threats.

    Comparison With Prior Work

    Individuals who expressed higher anxiety were less likely to engage in uncivil behaviors such as posting hateful comments targeting individuals with a specific identity or using profane language in their posts. We suspect that individuals’ anxiety may have decreased the level of uncivil expressions about vaccine mandate policy, because individuals who are anxious about COVID-19 and its health consequences are more likely to seek ways to contain the threat (ie, spread of COVID-19) and exhibit positive attitudes and behaviors toward policies related to restricting the spread of COVID-19. Namely, when novel threatening stimuli are encountered and feelings of anxiety are induced, people may be motivated to attend to the issue at hand [66]. In line with this idea, previous studies suggest that anxiety can be an indicator of a “functional fear” that predicts individuals’ positive attitudes and behaviors (eg, compliance) toward COVID-19–related measures and policies [67]. For instance, an extant work shows that COVID-19–related anxiety and health-related fears were associated with more protective health behaviors and higher vaccine acceptance [68,69].

    It is noteworthy that anger, unlike anxiety or sadness, predicted all dimensions of incivility, demonstrating that this emotion is the strongest predictor of incivility.

    Evidence from previous studies has shown that prolonged risk and uncertainty about the level of risk can elicit anger and conflict within the community [70]. People have experienced increased levels of anger during the pandemic [71], and those who express anger have also exhibited disbelief toward COVID-19 vaccines [72]. Moreover, it has been shown that political polarization regarding the issues of vaccination and vaccine mandates has further fueled public outrage among groups with conflicting political views [51,52]. Thus, the strong association between anger and uncivil behaviors can be due to both social disruptions caused by the wide spread of COVID-19 and political conflicts partly induced by media outlets.

    Lastly, as the level of sadness increased, individuals were more likely to exhibit verbal intentions to inflict pain and hurt other individuals or groups. Such aggression toward other people, especially exhibiting intentions to hurt others, may be explained by depression and victimhood. Approximately over 2 years of the COVID-19 pandemic, individuals worldwide have experienced prolonged social isolation and lifestyle disruptions, which have led them to be depressed [56,57]. Furthermore, the direct health impacts of the spread of COVID-19 have caused many individuals to become the victims of multiple losses such as a loss of financial security, loss of family members, and loss of physical/mental health and general safety [73,74]. However, sadness may have been strongly associated with viewing themselves as victims of COVID-19, which could have led them to issue threats to others who were favorable toward vaccine mandates. Additionally, this victimhood mentality [58] among the unvaccinated may have also been high because they are prohibited from participating in relationships and activities available to those who are vaccinated. This prohibition may have led them to feel socially excluded and in turn prompt threats toward the outgroup members—proponents of vaccine mandates.

    Limitations

    Although the findings shed light on the psychological processes underlying vaccine mandate incivility, this study is not without limitations. The LIWC computational tool does not measure the nuances afforded by human coders. Although we endeavored to minimize this limitation by using well-validated measures [59], future work might employ human coders to analyze the specific topics related to uncivil discourse. Additionally, we focused on posts shared to Twitter and therefore cannot generalize our findings about incivility to other social media platforms. Given the role of platform community norms in predicting incivility, future work should investigate how incivility manifests itself on different platforms. Likewise, Twitter users are wealthier, younger, and more liberal than the wider population of Americans [75], and the sample was limited to English-speaking Twitter users, which makes it difficult to generalize our findings to the entire US population. Additionally, we acknowledge that social media posting data could have been biased based on individuals’ geographical area (eg, city and state), whether they were local residents or visitors in the area at the time of the post, as well as the types of activities completed during the course of a day [76,77]. These factors may have contributed to our study findings. Lastly, we did not measure how many different users were included in each stage in the data collection process. Future work should elucidate the extent to which a small number of active users produce uncivil vaccine mandate content.

    Conclusions

    The results suggest that our multidimensional approach to incivility is a promising alternative to understanding and intervening in the psychological processes underlying uncivil vaccine discourse. Given the need for real-time monitoring and automated responses to the spread of health information and misinformation on the web, social media platforms can harness the Google Perspective API to offer users immediate, automated feedback when it detects that a comment is uncivil [78]. Furthermore, the Perspective API is available in 17 languages—from Arabic to Korean, enabling the study of uncivil health discourse in non-English posts. Future work should explore cross-cultural differences in uncivil health discourse.

    Vaccine hesitancy still remains a threat to global health, and this work demonstrates that distinct emotional processes underlie distinct attitudes toward vaccines and vaccine-related policies. It is important for health practitioners and policy makers to first acknowledge negative emotions associated with vaccines and vaccine mandates while emphasizing the safety of COVID-19 vaccines in health campaigns, which would provide aid in reducing vaccine hesitancy. One avenue public health officials can take to combat vaccine hesitancy while simultaneously affirming discrete negative emotions toward the vaccine is by holding COVID-19 community listening sessions, where officials can hear directly from communities about COVID-19 concerns, including vaccination (see Figure 3 for an overview) [79]. After officials have a better understanding of the specific emotional processes underlying a communities’ vaccine hesitancy, public health campaigns can tailor messages to address these concerns (see Figure 3) [80,81].

    ‎
    Figure 3. Concrete recommendations for promoting vaccine uptake based on underlying emotions.
    View this figure

    Conflicts of Interest

    None declared.

    1. WHO coronavirus (COVID-19) dashboard. World Health Organization.   URL: https://covid19.who.int/ [accessed 2022-08-21]
    2. Chang K. Fauci and other top U.S. health officials stress the urgency of vaccinations and their safety. The New York Times. 2021 Apr 25.   URL: https://www.nytimes.com/2021/04/25/health/fauci-vaccines-safety.html [accessed 2022-08-21]
    3. See where 12 million U.S. employees are affected by government vaccine mandates. The New York Times. 2021 Dec 18.   URL: https://www.nytimes.com/interactive/2021/12/18/us/vaccine-mandate-states.html [accessed 2022-08-21]
    4. Sabahelzain MM, Hartigan-Go K, Larson HJ. The politics of COVID-19 vaccine confidence. Curr Opin Immunol 2021 Aug;71:92-96 [FREE Full text] [CrossRef] [Medline]
    5. Hitlin P, Olmstead K. The science people see on social media. Pew Research Center. 2018 Mar 21.   URL: https://www.pewresearch.org/science/2018/03/21/the-science-people-see-on-social-media/ [accessed 2022-09-06]
    6. Buccoliero L, Bellio E, Crestini G, Arkoudas A. Twitter and politics: evidence from the US presidential elections 2016. Journal of Marketing Communications 2018 Aug 16;26(1):88-114 [FREE Full text] [CrossRef]
    7. Benis A, Chatsubi A, Levner E, Ashkenazi S. Change in threads on Twitter regarding influenza, vaccines, and vaccination during the COVID-19 pandemic: artificial intelligence-based infodemiology study. JMIR Infodemiology 2021 Oct 14;1(1):e31983 [FREE Full text] [CrossRef] [Medline]
    8. De Choudhury M, Morris MR, White RW. Seeking and sharing health information online: comparing search engines and social media. Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI'14 New York, New York, USA: ACM Press; 2014 Apr 26 Presented at: CHI'14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; April 24 to May 1, 2014; Toronto, ON p. 1365-1376. [CrossRef]
    9. Featherstone JD, Bell RA, Ruiz JB. Relationship of people's sources of health information and political ideology with acceptance of conspiratorial beliefs about vaccines. Vaccine 2019 May 21;37(23):2993-2997. [CrossRef] [Medline]
    10. Ruiz JB, Bell RA. Predictors of intention to vaccinate against COVID-19: results of a nationwide survey. Vaccine 2021 Feb 12;39(7):1080-1086 [FREE Full text] [CrossRef] [Medline]
    11. Ruiz J, Featherstone JD, Barnett GA. Identifying vaccine hesitant communities on twitter and their geolocations: a network approach. 2021 Jan 05 Presented at: Proceedings of the 54th Hawaii International Conference on System Sciences; January 5, 2021; Kauai, HI p. 3964-3970. [CrossRef]
    12. Stevens H, Palomares NA. Constituents’ inferences of local governments’ goals and the relationship between political party and belief in COVID-19 misinformation: cross-sectional survey of Twitter followers of state public health departments. JMIR Infodemiology 2022 Feb 10;2(1):e29246. [CrossRef]
    13. Tsao SF, Chen H, Tisseverasinghe T, Yang Y, Li L, Butt ZA. What social media told us in the time of COVID-19: a scoping review. Lancet Digit Health 2021 Mar;3(3):e175-e194 [FREE Full text] [CrossRef] [Medline]
    14. Muhamad JW, Merle P. Identity and information overload: examining the impact of health messaging in times of crisis. In: O'Hair HD, O'Hair MJ, editors. Communicating Science in Times of Crisis. Hoboken, NJ: Wiley; 2021:110-127.
    15. Scoville C, McCumber A, Amironesei R, Jeon J. Mask refusal backlash: the politicization of face masks in the American public sphere during the early stages of the COVID-19 pandemic. Socius 2022 Apr 25;8:237802312210931. [CrossRef]
    16. Kennedy J. Populist politics and vaccine hesitancy in Western Europe: an analysis of national-level data. Eur J Public Health 2019 Jun 01;29(3):512-516. [CrossRef] [Medline]
    17. Landler M. Vaccine mandates rekindle fierce debate over civil liberties. The New York Times. 2021 Dec 10.   URL: https://www.nytimes.com/2021/12/10/world/europe/vaccine-mandates-civil-liberties.html [accessed 2022-09-06]
    18. Largent EA, Persad G, Sangenito S, Glickman A, Boyle C, Emanuel EJ. US public attitudes toward COVID-19 vaccine mandates. JAMA Netw Open 2020 Dec 01;3(12):e2033324 [FREE Full text] [CrossRef] [Medline]
    19. Masullo Chen G, Muddiman A, Wilner T, Pariser E, Stroud NJ. We should not get rid of incivility online. Soc Media Soc 2019 Jul 16;5(3):205630511986264. [CrossRef]
    20. Muddiman A, Stroud NJ. News values, cognitive biases, and partisan incivility in comment sections: uncivil comments. J Commun 2017 Jul 18;67(4):586-609. [CrossRef]
    21. Stevens H, Acic I, Taylor LD. Uncivil reactions to sexual assault online: Linguistic features of news reports predict discourse incivility. Cyberpsychol Behav Soc Netw 2021 Dec;24(12):815-821. [CrossRef] [Medline]
    22. Sun Q, Wojcieszak M, Davidson S. Over-time trends in incivility on social media: evidence from political, non-political, and mixed sub-reddits over eleven years. Front Polit Sci 2021 Nov 2;3:741605. [CrossRef]
    23. Theocharis Y, Barberá P, Fazekas Z, Popa SA. The dynamics of political incivility on Twitter. SAGE Open 2020 May 13;10(2):215824402091944. [CrossRef]
    24. Wojcieszak M, Azrout R, Vreese D. Waving the red cloth: media coverage of a contentious issue triggers polarization. Public Opin Q 2018 Dec 09;82(1):87-109. [CrossRef]
    25. Dubey AD. The resurgence of cyber racism during the COVID-19 pandemic and its aftereffects: analysis of sentiments and emotions in tweets. JMIR Public Health Surveill 2020 Oct 15;6(4):e19833 [FREE Full text] [CrossRef] [Medline]
    26. Muddiman A. Personal and public levels of political incivility. Int J Commun 2017 Aug;11:3182-3202.
    27. Muddiman A, Kearney M. What counts as incivility in the Trump era? an inductive approach to political incivility. 2018 Presented at: 2018 Annual Meeting of the Political Communication Division of the American Society of Political Scientists; August 30 to September 2, 2018; Boston, MA.
    28. Tromble R. Twitter "Healthy Conversations". Rebekah Tromble.   URL: https://www.rebekahtromble.net/political-communication [accessed 2022-08-21]
    29. Papacharissi Z. Democracy online: civility, politeness, and the democratic potential of online political discussion groups. New Media & Society 2016 Jun 30;6(2):259-283. [CrossRef]
    30. Sobieraj S, Berry JM. From incivility to outrage: political discourse in blogs, talk radio, and cable news. Polit Commun 2011 Feb 09;28(1):19-41. [CrossRef]
    31. Coe K, Kenski K, Rains SA. Online and uncivil? patterns and determinants of incivility in newspaper website comments. J Commun 2014 Jun 16;64(4):658-679. [CrossRef]
    32. Gervais BT. Incivility online: affective and behavioral reactions to uncivil political posts in a web-based experiment. J Inf Technol Politics 2015 Jan 14;12(2):167-185. [CrossRef]
    33. Stroud NJ, Scacco JM, Muddiman A, Curry AL. Changing deliberative norms on news organizations' Facebook sites. J Comput-Mediat Comm 2014 Oct 16;20(2):188-203. [CrossRef]
    34. Kim JW, Guess A, Nyhan B, Reifler J. The distorting prism of social media: how self-selection and exposure to incivility fuel online comment toxicity. J Commun 2021 Sep 03;71(6):922-946. [CrossRef]
    35. Rossini P. Disentangling uncivil and intolerant discourse in online political talk. In: A Crisis of Civility. New York, NY: Routledge; 2019:142-157.
    36. Lwin MO, Sheldenkar A, Lu J, Schulz PJ, Shin W, Panchapakesan C, et al. The evolution of public sentiments during the COVID-19 pandemic: case comparisons of India, Singapore, South Korea, the United Kingdom, and the United States. JMIR Infodemiology 2022 Feb 10;2(1):e31473. [CrossRef]
    37. Lwin MO, Lu J, Sheldenkar A, Schulz PJ, Shin W, Gupta R, et al. Global sentiments surrounding the COVID-19 pandemic on Twitter: analysis of Twitter trends. JMIR Public Health Surveill 2020 May 22;6(2):e19447 [FREE Full text] [CrossRef] [Medline]
    38. Chou WS, Budenz A. Considering emotion in COVID-19 vaccine communication: addressing vaccine hesitancy and fostering vaccine confidence. Health Commun 2020 Oct 30;35(14):1718-1722. [CrossRef] [Medline]
    39. Calvillo DP, Ross BJ, Garcia RJB, Smelter TJ, Rutchick AM. Political ideology predicts perceptions of the threat of COVID-19 (and susceptibility to fake news about it). Soc Psychol Personal Sci 2020 Jul 22;11(8):1119-1128. [CrossRef]
    40. Wiysonge CS, Ndwandwe D, Ryan J, Jaca A, Batouré O, Anya BM, et al. Vaccine hesitancy in the era of COVID-19: could lessons from the past help in divining the future? Hum Vaccin Immunother 2022 Dec 31;18(1):1-3 [FREE Full text] [CrossRef] [Medline]
    41. Blanchard DC. Translating dynamic defense patterns from rodents to people. Neurosci Biobehav Rev 2017 May;76(Pt A):22-28. [CrossRef] [Medline]
    42. Carver CS, Harmon-Jones E. Anger is an approach-related affect: evidence and implications. Psychol Bull 2009 Mar;135(2):183-204. [CrossRef] [Medline]
    43. Kessels LTE, Ruiter RAC, Wouters L, Jansma BM. Neuroscientific evidence for defensive avoidance of fear appeals. Int J Psychol 2014 Apr;49(2):80-88 [FREE Full text] [CrossRef] [Medline]
    44. Featherstone JD, Zhang J. Feeling angry: the effects of vaccine misinformation and refutational messages on negative emotions and vaccination attitude. J Health Commun 2020 Sep 01;25(9):692-702. [CrossRef] [Medline]
    45. Rief W. Fear of adverse effects and COVID-19 vaccine hesitancy: recommendations of the treatment expectation expert group. JAMA Health Forum 2021 Apr 16;2(4):e210804. [CrossRef]
    46. Barlett CP, Rinker A, Roth B. Cyberbullying perpetration in the COVID-19 era: an application of general strain theory. J Soc Psychol 2021 Jul 04;161(4):466-476. [CrossRef] [Medline]
    47. Wang Q, Luo X, Tu R, Xiao T, Hu W. COVID-19 information overload and cyber aggression during the pandemic lockdown: the mediating role of depression/anxiety and the moderating role of Confucian responsibility thinking. Int J Environ Res Public Health 2022 Jan 29;19(3):1540 [FREE Full text] [CrossRef] [Medline]
    48. Brnovich M. Why states are fighting Biden's vaccine mandates Internet. Fox News. 2022 Jan 24.   URL: https://www.foxnews.com/opinion/why-states-fighting-bidens-vaccine-mandates-arizona-ag-mark-brnovich [accessed 2022-08-21]
    49. Krugman P. What to do with our pandemic anger. The New York Times. 2022 Feb 08.   URL: https://www.nytimes.com/2022/02/07/opinion/covid-unvaccinated-anger.html [accessed 2022-08-21]
    50. Novaco RW. Anger. In: Zeigler-Hill V, Shackelford TK, editors. Encyclopedia of Personality and Individual Differences. Cham, Switzerland: Springer; Apr 22, 2020:166-171.
    51. Kerr J, Panagopoulos C, van der Linden S. Political polarization on COVID-19 pandemic response in the United States. Pers Individ Dif 2021 Sep;179:110892 [FREE Full text] [CrossRef] [Medline]
    52. Pennycook G, Rand DG. The psychology of fake news. Trends Cogn Sci 2021 May;25(5):388-402 [FREE Full text] [CrossRef] [Medline]
    53. Savage J, Verhulst B, Copeland W, Althoff RR, Lichtenstein P, Roberson-Nay R. A genetically informed study of the longitudinal relation between irritability and anxious/depressed symptoms. J Am Acad Child Adolesc Psychiatry 2015 May;54(5):377-384 [FREE Full text] [CrossRef] [Medline]
    54. Balch B. The cost of being unvaccinated is rising — will people be willing to pay the price? Association of American Medical Colleges. 2021 Oct 26.   URL: https://www.aamc.org/news-insights/cost-being-unvaccinated-rising-will-people-be-willing-pay-price [accessed 2022-08-21]
    55. Money L, Lin RGII, Breijo S. L.A. County plans to require proof of vaccination at indoor bars, nightclubs, breweries, wineries. Los Angeles Times. 2021 Sep 15.   URL: https:/​/www.​latimes.com/​california/​story/​2021-09-15/​la-to-require-covid-19-vaccines-at-bars-nightclubs-breweries-wineries [accessed 2022-08-21]
    56. Bignardi G, Dalmaijer ES, Anwyl-Irvine AL, Smith TA, Siugzdaite R, Uh S, et al. Longitudinal increases in childhood depression symptoms during the COVID-19 lockdown. Arch Dis Child 2020 Dec 09:791-797 [FREE Full text] [CrossRef] [Medline]
    57. Palgi Y, Shrira A, Ring L, Bodner E, Avidor S, Bergman Y, et al. The loneliness pandemic: loneliness and other concomitants of depression, anxiety and their comorbidity during the COVID-19 outbreak. J Affect Disord 2020 Oct 01;275:109-111 [FREE Full text] [CrossRef] [Medline]
    58. Horwitz RB. Politics as victimhood, victimhood as politics. J Policy Hist 2018 Jun 21;30(3):552-574. [CrossRef]
    59. Pennebaker JW, Boyd RL, Jordan K, Blackburn K. The development and psychometric properties of LIWC2015. The Universityt of Texas at Austin. 2015.   URL: https://repositories.lib.utexas.edu/bitstream/handle/2152/31333/LIWC2015_LanguageManual.pdf [accessed 2022-08-21]
    60. Perspective API.   URL: https://developers.perspectiveapi.com/s/about-the-api-attributes-and-languages [accessed 2022-08-21]
    61. Stevens HR, Acic I, Rhea S. Natural language processing insight into LGBTQ+ youth mental health during the COVID-19 pandemic: longitudinal content analysis of anxiety-provoking topics and trends in emotion in LGBTeens microcommunity subreddit. JMIR Public Health Surveill 2021 Aug 17;7(8):e29029 [FREE Full text] [CrossRef] [Medline]
    62. Miazga J, Hachaj T. Evaluation of most popular sentiment lexicons coverage on various datasets. 2019 Oct 08 Presented at: SSIP 2019: Proceedings of the 2019 2nd International Conference on Sensors, Signal and Image Processing; October 8-10, 2019; Prague, Czech Republic p. 86-90. [CrossRef]
    63. High R. The era of cognitive systems: an inside look at IBM Watson and how it works. Redbooks. 1994.   URL: https://www.redbooks.ibm.com/redpapers/pdfs/redp4955.pdf [accessed 2022-09-06]
    64. Hopp T, Vargo CJ, Dixon L, Thain N. Correlating self-report and trace data measures of incivility: a proof of concept. Soc Sci Comput Rev 2018 Dec 17;38(5):584-599. [CrossRef]
    65. Pascual-Ferrá P, Alperstein N, Barnett DJ. A multi-platform approach to monitoring negative dominance for COVID-19 vaccine-related information online. Disaster Med Public Health Prep 2021 May 03:1-24. [CrossRef] [Medline]
    66. Marcus GE, Neuman WR, Mackuen M. Affective Intelligence and Political Judgment. Chicago, IL: The University of Chicago Press; 2000.
    67. Harper CA, Satchell LP, Fido D, Latzman RD. Functional fear predicts public health compliance in the COVID-19 pandemic. Int J Ment Health Addict 2021 Apr 27;19(5):1875-1888 [FREE Full text] [CrossRef] [Medline]
    68. Stevens HR, Oh YJ, Taylor LD. Desensitization to fear-inducing COVID-19 health news on Twitter: observational study. JMIR Infodemiology 2021 Jul 16;1(1):e26876 [FREE Full text] [CrossRef] [Medline]
    69. Bendau A, Plag J, Petzold MB, Ströhle A. COVID-19 vaccine hesitancy and related fears and anxiety. Int Immunopharmacol 2021 Aug;97:107724 [FREE Full text] [CrossRef] [Medline]
    70. Smith LE, Duffy B, Moxham-Hall V, Strang L, Wessely S, Rubin GJ. Anger and confrontation during the COVID-19 pandemic: a national cross-sectional survey in the UK. J R Soc Med 2021 Feb;114(2):77-90 [FREE Full text] [CrossRef] [Medline]
    71. Shanahan L, Steinhoff A, Bechtiger L, Murray AL, Nivette A, Hepp U, et al. Emotional distress in young adults during the COVID-19 pandemic: evidence of risk and resilience from a longitudinal cohort study. Psychol Med 2022 Apr;52(5):824-833 [FREE Full text] [CrossRef] [Medline]
    72. Hurley D. An upshot of COVID-19—neurologists face instances of increased hostility and strategies to deal with it. Neurology Today 2022 Feb 03;22(3):8-31. [CrossRef]
    73. Wallace CL, Wladkowski SP, Gibson A, White P. Grief during the COVID-19 pandemic: considerations for palliative care providers. J Pain Symptom Manage 2020 Jul;60(1):e70-e76 [FREE Full text] [CrossRef] [Medline]
    74. Jacobson NC, Lekkas D, Price G, Heinz MV, Song M, O'Malley AJ, et al. Flattening the mental health curve: COVID-19 stay-at-home orders are associated with alterations in mental health search behavior in the United States. JMIR Ment Health 2020 Jun 01;7(6):e19347 [FREE Full text] [CrossRef] [Medline]
    75. Hitlin P, Olmstead K. The science people see on social media. Pew Research Center. 2018 Mar 21.   URL: https://www.pewresearch.org/science/2018/03/21/the-science-people-see-on-social-media/ [accessed 2022-09-06]
    76. Padilla JJ, Kavak H, Lynch CJ, Gore RJ, Diallo SY. Temporal and spatiotemporal investigation of tourist attraction visit sentiment on Twitter. PLoS One 2018 Jun 14;13(6):e0198857 [FREE Full text] [CrossRef] [Medline]
    77. Frank MR, Mitchell L, Dodds PS, Danforth CM. Happiness and the patterns of life: a study of geolocated tweets. Sci Rep 2013 Sep 12;3:2625 [FREE Full text] [CrossRef] [Medline]
    78. Mackey T, Baur C, Eysenbach G. Advancing infodemiology in a digital intensive era. JMIR Infodemiology 2022 Feb 14;2(1):e37115. [CrossRef]
    79. Bhat A, Browning-McNee LA, Ghauri K, Winckler S. COVID-19 vaccine confidence project. J Am Pharm Assoc (2003) 2022 Jan;62(1):288-295.e2 [FREE Full text] [CrossRef] [Medline]
    80. Chou WY, Burgdorf CE, Gaysynsky A, Hunter CM. COVID-19 vaccination* communication: applying behavioral and social science to address vaccine hesitancy and foster vaccine confidence. National Institutes of Health. 2020 Dec.   URL: https://www.obssr.od.nih.gov/sites/obssr/files/inline-files/OBSSR_VaccineWhitePaper_FINAL_508.pdf [accessed 2021-08-21]
    81. How to tailor COVID-19 vaccine information to your specific audience. Centers for Disease Control and Prevention. 2022.   URL: https://www.cdc.gov/vaccines/covid-19/hcp/tailoring-information.html [accessed 2022-08-21]

    ‎
    API: application programming interface
    H: hypothesis
    LIWC: Linguistic Inquiry and Word Count
    OR: odds ratio

    Edited by A Mavragani; submitted 28.02.22; peer-reviewed by R Gore, Y Li; comments to author 21.07.22; revised version received 21.08.22; accepted 30.08.22; published 13.09.22

    Copyright

    ©Hannah Stevens, Muhammad Ehab Rasul, Yoo Jung Oh. Originally published in JMIR Infodemiology (https://infodemiology.jmir.org), 13.09.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Infodemiology, is properly cited. The complete bibliographic information, a link to the original publication on https://infodemiology.jmir.org/, as well as this copyright and license information must be included.