This paper is in the following e-collection/theme issue:

Infoveillance and Social Listening (178) Infodemiology and Infoveillance (1269) Quality/Credibility of eHealth Information and Trust Issues (485) Vaccination and Immunization in the Digital Age (343) HPV Vaccination (51) Infoveillance, Infodemiology, Digital Disease Surveillance, Infodemic Management (1010) Vaccination Sentiment and Anti-Vaccination Infodemiology (49)

Published on in Vol 4 (2024)

This is a member publication of University of Toronto

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/50551, first published .
The Role of Scientific Research in Human Papillomavirus Vaccine Discussions on Twitter: Social Network Analysis

The Role of Scientific Research in Human Papillomavirus Vaccine Discussions on Twitter: Social Network Analysis

The Role of Scientific Research in Human Papillomavirus Vaccine Discussions on Twitter: Social Network Analysis

Authors of this article:

Geneviève Jessiman-Perreault1 Author Orcid Image ;   Jean-Christophe Boucher2 Author Orcid Image ;   So Youn Kim2 Author Orcid Image ;   Nicole Frenette1 Author Orcid Image ;   Abbas Badami2 Author Orcid Image ;   Henry M Smith2 Author Orcid Image ;   Lisa K Allen Scott1, 3, 4 Author Orcid Image
Article Authors Cited by Tweetations (2) Metrics

    Original Paper

    1Alberta Health Services, Calgary, AB, Canada

    2School of Public Policy, University of Calgary, Calgary, AB, Canada

    3Department of Oncology, University of Calgary, Calgary, AB, Canada

    4Department of Community Health Sciences, University of Calgary, Calgary, AB, Canada

    Corresponding Author:

    Geneviève Jessiman-Perreault, BA, MSc

    Alberta Health Services

    2210 2 St SW

    Calgary, AB, T2S 3C3

    Canada

    Phone: 1 587 794 4232

    Email: genevieve.jessiman-perreault@albertahealthservices.ca


    Background: Attitudes toward the human papillomavirus (HPV) vaccine and accuracy of information shared about this topic in web-based settings vary widely. As real-time, global exposure to web-based discourse about HPV immunization shapes the attitudes of people toward vaccination, the spread of misinformation and misrepresentation of scientific knowledge contribute to vaccine hesitancy.

    Objective: In this study, we aimed to better understand the type and quality of scientific research shared on Twitter (recently rebranded as X) by vaccine-hesitant and vaccine-confident communities.

    Methods: To analyze the use of scientific research on social media, we collected tweets and retweets using a list of keywords associated with HPV and HPV vaccines using the Academic Research Product Track application programming interface from January 2019 to May 2021. From this data set, we identified tweets referring to or sharing scientific literature through a Boolean search for any tweets with embedded links, hashtags, or keywords associated with scientific papers. First, we used social network analysis to build a retweet or reply network to identify the clusters of users belonging to either the vaccine-confident or vaccine-hesitant communities. Second, we thematically assessed all shared papers based on typology of evidence. Finally, we compared the quality of research evidence and bibliometrics between the shared papers in the vaccine-confident and vaccine-hesitant communities.

    Results: We extracted 250 unique scientific papers (including peer-reviewed papers, preprints, and gray literature) from approximately 1 million English-language tweets. Social network maps were generated for the vaccine-confident and vaccine-hesitant communities sharing scientific research on Twitter. Vaccine-hesitant communities share fewer scientific papers; yet, these are more broadly disseminated despite being published in less prestigious journals compared to those shared by the vaccine-confident community.

    Conclusions: Vaccine-hesitant communities have adopted communication tools traditionally wielded by health promotion communities. Vaccine-confident communities would benefit from a more cohesive communication strategy to communicate their messages more widely and effectively.

    JMIR Infodemiology 2024;4:e50551

    doi:10.2196/50551

    Keywords

    human papillomavirus; HPV; vaccine; immunization; social media; misinformation; social network analysis 


    Background

    Cervical cancer is one of the most preventable types of cancer in the world. Almost all cases are attributable to human papillomavirus (HPV), for which an effective vaccine exists [1]. Part of the global strategy to eliminate cervical cancer includes fully vaccinating 90% of girls with the HPV vaccine by the age of 15 years [2]. However, the global HPV immunization coverage currently remains suboptimal [3]. While many countries are experiencing vaccine supply issues, even high-income countries with reliable vaccine supply and comprehensive school-based programs are still failing to meet vaccine targets, largely due to vaccine hesitancy [4].

    Studies show that people now search the web for health information more often than they talk to health professionals about these matters [5]. The popularity of social media platforms has also created a phenomenon wherein people not only use the web to access health information but also play an active role in cocreating the information and ideas (in the form of opinions, anecdotes, and links to other sources of information) that they encounter in these web-based spaces [6]. Social media spaces create an important setting for people to interact and for communities to emerge, as they are not geographically bound but rather reflect patterns of shared interests, purpose, or identities [7]. As such, vaccine-confident and vaccine-hesitant groups represent distinctive ideologies and create distinctive web-based communities. The distinction between these 2 groups lies in their attitudes, beliefs, and behaviors associated with vaccine decision-making, in that vaccine-confident groups reflect public trust in vaccines and the evidence supporting their efficacy, effectiveness, and safety, which leads to their uptake of recommended vaccines. Vaccine-hesitant groups, for their part, tend to doubt this information, demonstrated by their reluctance or refusal to receive recommended vaccines [8,9].

    Despite a large body of evidence demonstrating the safety and efficacy of the HPV vaccine [10,11], attitudes toward the vaccine and the accuracy of information shared about this topic in web-based settings vary markedly from extremely negative and erroneous to supportive and factually accurate [12]. In addition, in recent years, there has been a rapid increase in the accessibility of scientific journals and subsequent dissemination of scientific findings through social media [13]. Simultaneously, there has been a decline in the role of unbiased science journalists and other communication experts as mediators between scientists and the public [14]. While these changes have had a democratizing effect on scientific knowledge and allowed for better communication between scientific communities and the public, this unfiltered access to scientific research also creates an environment where individuals may have difficulty in differentiating valid and credible information from biased and unreliable information or may misinterpret legitimate findings [15]. In contrast, researchers have also noted that the growth of open science can create opportunities for people to discuss novel research across polarized boundaries [16], but the type and quality of scientific research about HPV vaccination that is being shared in web-based discussions is unknown. Finally, with a wealth of open-access scientific research available, there are concerns about how ideologically motivated communities, such as vaccine-hesitant groups, integrate scientific knowledge into their social media communication strategies to amplify uncertainty around vaccines [17]. It is prudent to investigate how scientific research is integrated into web-based HPV vaccine discussions, given that web-based information is typically considered to be more credible, reliable, and authoritative if supported by scientific citation, notwithstanding the source of journal, authorship, or other features [18].

    Twitter (recently rebranded as X; as data collection occurred before the rebrand, we will be using its former name throughout this paper) is one of the largest, most popular, and most influential social media platforms in the world. Twitter has also traditionally been a preferred source of public opinion data for applied public health research [19-22]. This is because social media feeds such as Twitter offer an avenue for continuous, near-real–time collection of unsolicited information generated by many individuals regarding a variety of topics of interest [23,24]. Several studies have recently demonstrated the benefits of leveraging social media over traditional methods such as surveys as a source of primary data for health promotion interventions, including those aimed at increased participation in HPV immunization programs [25].

    Objectives

    Exposure to web-based discussions about HPV immunization on Twitter, regardless of geographic location, may influence peoples’ attitudes toward the vaccine [22,26,27]. Thus, there is significant interest among public health professionals to better understand how scientific knowledge about HPV immunization is wielded on Twitter, both to understand the impact of scientific knowledge on vaccine hesitancy and to identify opportunities for novel interventions aimed at countering or debunking misinformation and supporting increased uptake of the HPV vaccine [6,28]. Therefore, in this study, we aimed to do the following:

    1. Describe and visualize the vaccine-hesitant and vaccine-confident communities’ patterns of sharing HPV vaccination–related scientific literature on Twitter
    2. Thematically analyze the scientific literature shared by both vaccine-hesitant and vaccine-confident communities using a typology of research evidence
    3. Determine whether there are differences in shares, quality of evidence, and other bibliometric indicators of the scientific literature shared by each community

    Overview

    Our methods followed a multistep process. First, we conducted a rapid review to inform HPV and HPV vaccine keywords. Second, we used these keywords to filter tweets and create a data set. Third, we detected vaccine-confident and vaccine-hesitant communities and generated social network maps of each community based on tweets and retweet. Fourth, we detected the mentions of scientific literature in each community and extracted those papers for future statistical and social network analysis. A summary of these methods is presented in Figure 1 (adapted from the paper by Elyashar et al [29]), and further details are presented in the following sections.

    ‎
    Figure 1. Summary of the study methods. API: application programming interface; HPV: human papillomavirus.

    Literature Review to Inform Data Collection

    To determine the most applicable keywords to guide this study, a rapid review was first conducted to determine the most frequently used keywords in literature focused on HPV and HPV immunization discourse on Twitter. The rapid review methodology was selected due to its efficiency in synthesizing a large volume of information in a timely yet systematic manner [30]. This review yielded 13 papers published between 2015 and 2020 about the topic of HPV immunization discussions on social media, with 11 (85%) focusing on HPV immunization discussions on Twitter specifically. We extracted the keywords used in each paper to filter content on social media (Textbox 1). Then, we synthesized these keywords to compile a list of the most used keywords to represent HPV and HPV vaccine discussions on social media, and the top 3 keywords were used to generate the data set.

    Textbox 1. Papers yielded from the rapid review and the associated human papillomavirus (HPV)–related and HPV vaccine–related keywords.

    Papers and keywords

    • Shapiro et al [31]
      • “Gardasil,” “Cervarix,” “HPV AND vaccin*,” and “cervical AND vaccin*”
    • Massey et al [32]
      • “HPV,” “HPV vaccine,” “HPV shot,” “Gardasil,” and “Cervarix” (and hashtag equivalents)
    • Keim-Malpass et al [33]
      • “#HPV” and “#Gardasil”
    • Du et al [21]
      • “HPV,” “human papillomavirus,” “Gardasil,” and “Cervarix”
    • Nelon et al [34]
      • “#vaccines,” “#vaccine,” “#vaccinations,” and “#vaccination”
    • Surian et al [35]
      • “HPV AND vaccine,” “HPV AND vaccination,” “Gardasil,” “cervical AND vaccination,” “cervical AND vaccine,” and “Cervarix”
    • Zhou et al [36]
      • “HPV,” “vaccine,” “Gardasil,” “Cervarix,” “vaccination,” “cervical,” and “cancer”
    • Becker et al [37]
      • “Pentavalent OR pentavac OR quinvaxem”
    • Dyda et al [38]
      • “Cervical,” “Cervarix,” “HPV,” “human papillomavirus,” “vaccine,” “vaccination,” and “Gardasil”
    • Chakraborty et al [20]
      • “HPV,” “papilloma,” “pappiloma,” “papiolma,” “papillomavirus,” “Gardasil,” “Gardisil,” “Guardisil,” “Guardasil,” “Cervarix,” “cervical shot,” “cervical shots,” “cervical vaccine,” “cervical vaccines,” “cervical vax,” “cervical vaxine,” “cervical vaxines,” “cervical vaxx,” “cervical vaxxine,” “cervical vaxxines,” “cervical vaccination,” and “cervical vaccinations”
    • Dunn et al [39]
      • “Gardasil,” “Cervarix,” “HPV AND vaccine,” and “cervical AND vaccin”
    • Budenz et al [40]
      • “HPV,” “HPV vaccine,” “HPV shot,” “Gardasi,” and “Cervarix” (and hashtag equivalents)
    • Zhang et al [41]
      • “Cervarix,” “Gardasil,” “HPV,” “human papillomavirus,” “Gardasil,” “HPV AND vaccin*,” and “cervical AND vaccin*”

    Data Collection

    Using 3 of the most common keywords that emerged from the initial rapid review (“HPV” OR “Gardasil” OR “Cervarix”), a data set of tweets and retweets was created (N=596,987). Then, tweets were collected using the Academic Research Product Track application programming interface (API) from January 2019 to May 2021 [42]. Data were collected using the Twitter API Python wrapper (Python Software Foundation, version 3.8.5) [43]. The construction of the API, data collection, and data processing (ie, importing, exporting, and filtering of data) were performed in Python [44].

    Ethical Considerations

    This study received an exemption from ethics approval as determined by The Conjoint Faculties Research Ethics Board at the University of Calgary. This was due to its use of only publicly available information from an existing data set. Furthermore, the published results have omitted all identifiable information and are only presented in aggregate form.

    Social Network Analysis

    First, we created a social network of accounts by creating an edge list using retweets. The retweet edge list consisted of nodes representing individual Twitter accounts and edges representing accounts that are being retweeted. The individual Twitter accounts were identified using the “username” information from the API, and the source of the retweet account information was extracted using the account mentions beside the “RT” in the tweets’ text in our data set. Our data set consisted of 57,109 retweets and 25,898 original or quoted tweets. Retweet networks were analyzed as they are found on aggregate to better reflect agreement among users and thus represent an ideological community on issues such as vaccination [45]. Second, we used a Louvain modularity method to classify subclusters of web-based communities in the resulting social network [46]. This method was chosen because the algorithm was designed to accurately detect subcommunities within large networks and operate fast computationally. Third, the social network analysis map also illustrated a strong polarization of the subclusters. Through this polarization and the identification of primary influencers within a subcommunity, the vaccine-confident (n=234,015) and vaccine-hesitant (n=95,908) web-based communities were identified. The primary influencers were detected by measuring the degree centrality, which is the measure of the number of connections each user has within the network. Thus, the accounts with the highest measure of degree centrality were categorized as primary influencers, as a high degree centrality demonstrates a high number of connections an account has within the network. These primary influencers, along with the content of the account’s bio descriptions and tweets, were qualitatively studied to examine their expressed positions regarding HPV vaccination. Edge list was constructed using Python, and the retweet social network analysis was conducted using Gephi- (Gelphi, version 0.9.2) [47].

    Scientific Literature Sharing Network Analysis

    From the vaccine-confident and vaccine-hesitant data sets, we identified tweets that either mentioned or shared scientific literature through a Boolean search for tweets with an embedded http secure link or any of the select list of words (“paper,” “article,” “research,” “scientific,” “peer review,” “literature,” “scientists,” “study,” and “report”) [48]. This filter identified 220 papers from the vaccine-confident community and 30 papers from the vaccine-hesitant community. The titles of or links to these papers were extracted from the data set along with associated metrics such as number of shares for further analysis (as described in the Data Analysis section). We identified the top 20 most shared scientific publications in these respective communities. We chose to identify the top 20 most shared scientific publications due to the proportion of shares that these papers had—accounting for >97% of shares in the vaccine-hesitant community and approximately 61% in the vaccine-confident community. Then, we repeated the social network analysis steps by creating a retweet network of accounts sharing the top 20 prominent scientific publications within the vaccine-confident and vaccine-hesitant communities. The edge list for the vaccine-confident community comprised 989 nodes and 1013 edges, whereas the vaccine-hesitant group had 355 nodes and 422 edges. The primary influencers in this network were again identified using degree centrality measures, and we qualitatively analyzed these accounts on Twitter through their Twitter bio descriptions. The social network analysis of the scientific papers was conducted using Gephi (version 0.9.2) [47].

    Typology of Evidence for Thematic and Critical Appraisal

    Overall, 2 members of the research team (GJP and NF) with subject area expertise in HPV immunization independently reviewed all scientific papers from each network using a typology of evidence, proposed by Gray [49], based on the suitability of the study design for the research question posed. This typology was determined to be the most appropriate and feasible approach to critically appraise the scientific papers because it allowed for the ability to schematically differentiate between diverse study designs (from in vivo to clinical trials and reviews). First, we classified the objective, research question, or aim of the study based on 9 categories that were used to classify research papers based on the typology by Gray [49] (presented in the first column of Table 1). Next, we classified each paper according to the study design. On the basis of these 2 metrics, a score ranging from 0 to 2 was assigned to each paper, where 0 indicates the least appropriate study design for the research question posed and 2 indicates the most appropriate design for the research question posed (refer to Table 1 for details about the scoring of the typology of evidence). The same 2 members of the research team compared their classifications and scoring, and if consensus could not be reached, a third member of the research team (LKAS) made the final decision. In addition, we extracted information about the characteristics of the paper (study design, research question, or objective), journal (journal name and year published), and author (names, affiliations, and conflicts of interest; refer to Multimedia Appendices 1 [50-70] and 2 [52,71-89] for results of the top 20 most shared papers obtained from the vaccine-confident and vaccine-hesitant communities). These data were used to conduct bibliometric analyses of the journal and descriptive analysis of the research content shared by each community, which are further described in the following sections.

    Table 1. A typology of evidence (example questions in columns refer to human papillomavirus [HPV] vaccination for the prevention of cancer) based on appropriateness of study design for the research question posed (adapted from the papers by Gray [49] and Petticrew and Roberts [90]).

    		In vivo and in vitro studiesQualitative researchCross-sectional surveyCase-control studiesCohort studiesRCTsaQuasi-experimental studiesNonexperimental evaluationsScoping reviews and narrative reviews
    Effectiveness (does this work? does doing this work better than doing that?)000012102
    Process of service delivery (how does it work?)021000012
    Salience (does it matter?)022000002
    Safety (will it do more harm than good?)010112112
    Acceptability (will the focus population be willing to or want to take up the HPV vaccine?)021001112
    Cost-effectiveness (is it worth delivering this service?)000002002
    Appropriateness (is this the right service for this population?)022000001
    Satisfaction with the service (is this population satisfied with the service?)022110000
    Basic science (what is the cellular mechanism of action?)100000000

    aRCT: randomized controlled trial.

    Bibliometric Indicators

    Traditionally, the prestige and quality of a journal was evaluated using citation metrics such as impact factor [91]. In the past few years, as assessment of scientific information has grown exponentially, new tools have been developed to capture the visibility and reach of web-based scientific information. Examples of these alternative metrics or altmetrics include likes, shared tweets, and retweets [92]. To compare traditional scholarly measures of quality to altmetrics, we collected data about the number of times the paper was shared by each vaccine community and the impact factor of the journal the paper was published in. We also collected data about the number of citations each shared paper had received through Google Scholar. Given that citations are impacted by the length of time since publication, we used the SCImago Journal Ranking (SJR) indicator, which provides a weighted average score that remains consistent each year and accounts for the prestige of the citing journal and the differences across subject fields, allowing for more equal comparisons across subject fields [93]. Each paper was assigned an SJR indicator, where a lower score indicates lower-ranking journals and higher scores indicate higher-ranking journals [94]. Journals that were not indexed in the Scopus database were not assigned an SJR score and were marked as missing in our database. These metrics were used to assess the influence of the shared papers in scientific research and the prestige of the journal the shared papers were published in.

    Data Analysis

    Once these bibliometrics and typology-of-evidence scores were collected in a data set, basic descriptive results of these 4 metrics (number of shares on Twitter, number of citations, impact factor, and typology of evidence score) were calculated using median and IQR, given their skewed distributions. We also performed the Mann-Whitney U test, given the nonnormal distribution of these data [95], to determine whether there were statistically significant differences in the 4 indicators between the papers shared in the vaccine-hesitant and vaccine-confident communities. The four indicators examined were (1) the number of shares that the original tweet sharing the publication on Twitter received, (2) the SJR score of the journal the paper was published in, (3) the number of citations the paper received, and (4) the typology of evidence score that the paper received. Statistical significance was determined using P value <.05. Effect size was calculated using Cohen d, where a standardized difference of 0.2 indicates a small difference, difference of 0.5 indicates a medium difference, and difference of 0.8 indicates a large difference [96]. All data analyses were conducted using SAS Studio (SAS Institute, version 3.6).


    Overview

    In total, 250 scientific papers (n=30, 12% in the vaccine-hesitant community and n=220, 88% in the vaccine-confident community) shared between January 2019 and May 2021 were identified. These papers received a combined total of 2247 shares on Twitter, with 562 (25.01%) shares for vaccine-hesitant papers and 1685 (74.99%) shares for vaccine-confident papers. On average, vaccine-hesitant papers received approximately 19.2 (SD 35.6) shares, whereas vaccine-confident papers received approximately 7.7 (SD 30.5) shares. Of these 250 scientific papers, the top 20 most shared papers from each vaccine community were used to produce a social network map of all tweets interacting with or sharing scientific papers about the HPV vaccine on Twitter (Multimedia Appendix 3).

    Vaccine-Hesitant Social Network

    Figure 2 presents the social network of all tweets sharing or interacting with tweets discussing scientific papers among the vaccine-hesitant community. As can be seen in Figure 2, the retweet network of scientific literature in the vaccine-hesitant community can be categorized into 5 distinct subclusters. Accounts associated with the red cluster shared papers focusing on the safety and ethical considerations around vaccination, with a journalist from a conservative news network emerging as the most influential account holder in this cluster. The most commonly shared paper in this cluster was a case study about the safety of the HPV vaccine in the context of alleged adverse reactions to the HPV vaccine in Japan [50]. In the light green cluster, 1 particular influencer, whose account was later suspended by Twitter, was similarly influential by sharing a paper focused on the effectiveness of HPV vaccination in the prevention of cervical cancer, namely, a widely circulated review paper about this topic [51]. Leading accounts linked to the orange cluster and the dark green cluster were personal user accounts, and both shared the same paper as the light green cluster, calling into question the efficacy of the HPV vaccine in the prevention of cervical cancer.

    ‎
    Figure 2. Network analysis of the vaccine-hesitant community sharing scientific research on Twitter. HPV: human papillomavirus.

    The orange cluster of the vaccine-hesitant community circulated a retracted paper, which alleged that HPV vaccines affected the vaccine recipients’ fertility and focused on safety [52]. Furthermore, the orange cluster’s location in the network (ie, adjacent to the light green cluster) suggests social influence and connection between the 2 clusters. In contrast, there was little interaction between the accounts in the light green cluster and the dark green cluster, suggesting that the influential accounts in these clusters independently found the same scientific literature and circulated it among a relatively isolated cohort of users. Finally, in the blue cluster, a European support group for those who had experienced vaccine injuries was the leading influential account, whereas a medical society’s account that published a widely shared paper in this cluster [51] was an account of secondary influence. Again, the influential accounts in this cluster shared scientific papers, which were retweeted by accounts that are more peripheral to the central clusters of influential accounts. The primary scientific paper circulated among users in this cluster focused on the theme of safety of the HPV vaccine by measuring the serum levels of autoantibodies in a cohort of girls who had possible adverse reactions following the receipt of the HPV vaccine [53].

    Vaccine-Confident Social Network

    The retweet network of scientific research shared among the web-based vaccine-confident community can similarly be divided into 5 distinct subclusters, as shown in Figure 3. The red cluster primarily included users retweeting literature from the British Medical Journal. There were 2 main papers circulated in this cluster, both of which focused on the effectiveness of the HPV vaccine. The first was a retrospective population study about the efficacy of the HPV vaccine in the prevention of cervical cancer in Scotland, focusing on the theme of satisfaction with service [71], whereas the second was an observational study about the outcomes of HPV screening in high-risk populations in England [72]. In the orange cluster, we observed a similar influence exerted by a government-funded public health agency, which shared a popular paper about effectiveness, focusing on the potential of the HPV vaccine to lower the risk of cervical cancer in a cohort population [73].

    ‎
    Figure 3. Network analysis of the vaccine-confident community sharing scientific research on Twitter. HPV: human papillomavirus.

    In the red, orange, and dark green clusters, there were physicians and health care workers among the users who retweeted influential tweets. For example, in the orange cluster, 1 particularly influential physician circulated an editorial paper about the effectiveness of the HPV vaccine, which indicated that high HPV vaccine coverage could eradicate cervical cancer within a few decades [74]. A science correspondent for a pre-eminent American newspaper was the leading influencer in the light green cluster wherein the primary paper circulated was an editorial, also focused on effectiveness, related to the positive impacts of HPV vaccination in Scotland [75]. Finally, in the blue cluster, a leading cancer prevention researcher from a British research institute was the leading influencer and author of the scientific papers circulated. In this cluster, papers about the psychological impacts of HPV screening [76] and the sociodemographic correlates of cervical cancer risk among those who did not participant in cervical screening programs in the United Kingdom [77] were recirculated by the accounts influenced by the leading researcher. Unlike the other clusters, health care workers were not overrepresented in the light green and blue clusters.

    Overall, results from the vaccine-confident community suggest that health care, scientific, and news media communities are operating in closed systems. As we can see in Figure 3, there are relatively few bridging connections among the different communities discussing influential HPV vaccination literature in the vaccine-confident space. In contrast, the vaccine-hesitant space (Figure 2) is a more cohesive and tightly connected community, suggesting that there are stronger knowledge flows between subclusters in this group. Twitter accounts in the vaccine-hesitant community appear to be more efficient in sharing information than the more fragmented vaccine-confident community (Multimedia Appendix 3). Furthermore, the vaccine-hesitant Twitter accounts are more effective in communicating the results and research of interest to one another, whereas those in the vaccine-confident space appear to struggle to disseminate the research of interest beyond their personal and professional communities. These findings are supported by the descriptive statistics presented later in the paper, which indicated that while the vaccine-confident community shares far more scientific papers than the vaccine-hesitant community, the scientific literature shared by the vaccine-confident community received far fewer shares per paper despite being published in higher-ranked journals.

    Typology of Evidence and Bibliometric Analysis

    Table 2 presents the distribution of typology of evidence categorized by vaccine community type. Most of the scientific papers shared by the vaccine-hesitant community focused on safety (16/30, 55%) or effectiveness (8/30, 28%), exemplifying the key concerns legitimizing vaccine hesitancy. The vaccine-confident community shared papers related to a wider range of research themes, the most common being papers that focused on basic science (56/220, 25.7%), effectiveness (55/220, 25.2%), acceptability (49/220, 22.5%), and salience (38/220, 17.4%). While the level of focus on effectiveness was similar between the 2 communities, there was very little overlap in the specific papers selected for sharing.

    Table 2. Description of the typology of evidence of all papers shared on Twitter categorized based on vaccine-confident and vaccine-hesitant communities.

    		Vaccine-confident community (N=220), n (%)Vaccine-hesitant community (N=30), n (%)
    Effectiveness55 (25)8 (26.7)
    Safety12 (5.5)16 (53.3)
    Process of service delivery4 (1.8)1 (3.3)
    Satisfaction with the service1 (0.5)0 (0)
    Salience38 (17.3)1 (3.3)
    Acceptability49 (22.3)1 (3.3)
    Cost-effectiveness5 (2.3)0 (0)
    Basic science56 (25.5)3 (10)

    Table 3 presents the descriptive statistics about the 4 metrics for all the scientific papers shared by vaccine-confident (220/250, 88% papers) and vaccine-hesitant (30/250, 12% papers) communities. The 4 metrics described in Table 3 are the median shares per paper, the median number of citations each shared paper received, the median SJR score of the journal that published each shared paper, and the median typology of evidence score. Table 3 also presents the results from the Mann-Whitney U test. Tweets containing scientific papers shared by the vaccine-confident community received a median of 3 shares, compared to a median of 4 shares by the vaccine-hesitant community. Results from the Mann-Whitney U test indicate that there are statistically significant differences (P=.01) in shares of tweets containing papers about HPV vaccination between the vaccine-hesitant and vaccine-confident communities and that this difference is small (Cohen d=0.37). Scientific papers shared by the vaccine-confident community received a median of 13 citations compared to a median of 17 citations for the scientific papers shared by the vaccine-hesitant community. We did not find evidence of statistically significant differences in the number of citations received by papers shared between the vaccine-confident and vaccine-hesitant communities. Scientific papers shared by the vaccine-confident community received a median SJR score of 1.83 compared to a median score of 0.84 for the papers shared by the vaccine-hesitant community. Results from the effect size calculation found this to be a medium standardized difference (Cohen d=61). The Mann-Whitney U test also found evidence of statistically significant (P<.001) differences in SJR scores of the HPV-related papers shared between the vaccine-confident and vaccine-hesitant communities. Finally, scientific papers shared by both the vaccine-confident and the vaccine-hesitant communities received a median typology of evidence score of 1, and results from the Mann-Whitney U test did not find evidence of a statistically significant difference.

    Table 3. Results from the Mann-Whitney U test for shares, number of citations, SCImago Journal Ranking (SJR), and typology of evidence score categorized based on human papillomavirus vaccine-confident and vaccine-hesitant communities.

    		Vaccine-confident community (n=220), median (IQR)Vaccine-hesitant community (n=30), median (median)P valueEffect size (Cohen d)
    Shares3 (1.0-6.5)4 (2.0-15.0).0070.37
    Number of citations13 (5.0-75.0)17 (9.0-44).280.19
    SJR1.83 (1.25-3.44)0.84 (0.68-1.30)<.0010.61
    Typology of evidence1 (0.0-10)1 (1.0-1.0).220.14

    Principal Findings

    The increase in the volume of scientific publications shared on the web [13] and the growth of open-access scientific publishing [16] have created an environment of greater access to scientific literature among lay audiences. However, little is known about how scientific literature is being incorporated into web-based communication strategies of vaccine-confident and vaccine-hesitant communities. Our study examined how scientific literature focusing on the HPV vaccine is being shared by vaccine-hesitant and vaccine-confident networks on Twitter. We found that despite the increased quantity of scientific literature being shared, such literature is often used by the vaccine-hesitant community to proliferate misinformation about vaccination, which is amplified in a web-based environment such as Twitter. Therefore, Kata [97] has described four key tactics that are used by the antivaccination movement to spread their messages on the web: (1) skewing the science, (2) shifting the hypotheses, (3) censorship, and (4) attacking the critics. A study conducted by van Schalkwyk et al [17] demonstrated that vaccine-hesitant groups are strategic in their use of scientific literature on social media to amplify uncertainty about vaccine safety and that vaccine-hesitant accounts who use large arsenals of scientific literature play important roles in dissemination of information across multiple communication networks. Findings from our thematic analysis of the papers shared by the vaccine-hesitant networks confirm this. Our study also found that the vaccine-hesitant community was much more likely to share scientific publications that questioned the safety and effectiveness of the HPV vaccine, whereas the vaccine-confident community shared scientific publications on a wider range of topics. This aligns with the tactic of skewing the science (identified by Kata [97]), which focuses on criticizing scientific studies while simultaneously calling for more studies, particularly focusing on the need for randomized controlled trials that compare vaccinated children and unvaccinated children. Moreover, most of the papers shared by the vaccine-confident community focused on basic science (ie, in vitro or in vivo studies), and this focus lowered the typology of evidence score of the vaccine-confident community, while failing to contribute to a unified message in the vaccine-confident community.

    Furthermore, the quality of journals that published the papers shared in these communities varied markedly. The scientific publications shared by the vaccine-confident community were significantly more likely to be published in higher-ranked journals and therefore obtained higher SJR scores, compared with those shared by the vaccine-hesitant community. Other researchers have found that critical appraisal is often absent when vaccine-hesitant individuals share “scientific evidence” on the web, which often includes citations that blur the line between legitimate scientific publications and fraudulent studies [98]. However, there is little evidence of communication across networks, despite repeated calls from public health communication experts to prebunk and debunk vaccine misinformation on the web [99,100]. Notably, both communities share a retracted paper, but their framing of the paper varies. The vaccine-confident community mocks the paper for its outlandish claims, whereas the vaccine-hesitant community highlights the findings as if they were accurate. This highlights 2 issues. First, despite not supporting the findings of the retracted paper, the vaccine-confident community still shared the paper, thus amplifying its reach. Second, the vaccine-hesitant communities’ definition of “scientific evidence” does not align with accepted norms, as retracted papers can no longer be considered part of the scientific evidence base.

    Vaccine-hesitant groups have been shown to co-opt the perceived authority of professional sources (eg, WebMD and the American Medical Association) to bolster their claims, even when the associated evidence does not support their arguments [101]. Interestingly, past studies have shown that while both groups point out knowledge deficits in their counterparts and attempt to correct misinformation by offering alternate sources of evidence, vaccine-confident groups have been shown to infrequently cite scientific evidence to correct misinformation or present counterarguments in web-based forums [102]. However, our analysis shows that the vaccine-confident community often shares scientific literature on the web as a form of self-promotion or knowledge translation, rather than as a tool to counter misinformation or correct misinterpretations.

    Consequently, consistent with others in this field, we suggest that vaccine researchers should take a more active role in the HPV-related conversations that are occurring on the web, beyond simply promoting their own studies and instead countering misinformation and disinformation on the web [103]. Researchers and practitioners hoping to meaningfully contribute to the conversation about HPV vaccination on the web should explore training in science communication and social media engagement strategies, including the monitoring and correcting of public misinterpretation of their studies on various social media platforms [103,104]. Studies show that the way in which health information is communicated affects recipients’ perception of it, with transparent communication fostering trust in health authorities and reducing the proliferation of conspiratorial beliefs [105].

    Limitations

    While Twitter provides us with a large body of unfiltered discussions to examine, the use of Twitter is not universal, and younger individuals (aged 18-29 years) and minority groups tend to be overrepresented on Twitter [20,24]. Therefore, while this analysis is not universal for all demographics, such as those who do not use Twitter as a social media platform, it provides opportunities to collect information about the health opinions held by members of several priority populations. While this study provides a way of studying web-based social interaction, further studies are needed to understand vaccine hesitancy among the general population who may not use Twitter.

    The creation of the data set of HPV-related and HPV vaccine–related tweets was based on 3 commonly used hashtags derived from a rapid review of published papers; therefore, there is the potential that we missed some tweets that also discussed HPV and HPV vaccine but were not captured by these hashtags. In addition, we extracted a variety of metrics about the papers and journals included in our data set, but given the wide variation in study design among the extracted papers, conducting a formal critical appraisal of quality was unfeasible for this project and is an area for future study. Furthermore, this study did not measure the engagement rate of tweets, which is a new analytic metric offered by Twitter and is calculated by dividing the number of engagements (ie, total number of times a user interacted with a post including retweets, replies, likes, and follows) by the number of impressions (ie, number of times a user is shown a particular post in their timeline or search results). It should be reinforced that the number of shares of a tweet is not equivalent to the impact of the content shared.

    Another limitation is that one of the metrics collected in our study was the number of citations each paper had received, for which we chose to use the “cited by” count provided by Google Scholar. While there has been criticism about the cited by metrics provided by Google Scholar due to double counting of citations from published journals and other sources [51], Google Scholar covers a larger breadth of sources (eg, conference papers and book chapters) than alternative platforms such as Web of Science [106]. Finally, the time frame we selected to collect tweets for this study, that is, January 2019 to May 2021, presents a limitation. We chose to expand our data collection to 2021 to allow us to acquire a sufficiently large data set, because the COVID-19 pandemic began shortly after the start of our data collection period. With the emergence of the COVID-19 pandemic, health discussions on Twitter became heavily focused on COVID-19 instead of other topics, including HPV vaccination. We ultimately extended our data collection time frame beyond our original timeline to provide us with a sufficiently large corpus of tweets to analyze. Given the unique period of data collection (ie, before and during the COVID-19 pandemic), which influenced the quantity of discussion about non–COVID-19 topics, the generalizability of these findings is reduced. Our experience in collecting these data over the course of the COVID-19 pandemic has been explored further in another publication, where we examined the attitudes and sentiment on Twitter toward HPV vaccination amidst the context of the pandemic [107].

    Strengths

    This study contributes to the growing body of knowledge about the discussions about HPV immunization in web-based settings by using novel mixed methods to identify what papers about HPV and HPV vaccine are being shared on the web and how vaccine-confident and vaccine-hesitant communities are using this knowledge in their web-based communication strategies. Our study demonstrates that vaccine-hesitant communities are using strategies of scientific authority by presenting them as “scientific evidence” on Twitter, regardless of the quality of the papers themselves. Vaccine-confident communities do not appear to be sharing papers to build consensus, rather they share their scientific studies. These findings are relevant to health communication experts who aim to combat vaccine misinformation and disinformation on the web by providing them with concrete examples of papers used to create distrust in HPV vaccines. Moreover, HPV researchers and health promotion organizations that use Twitter might find these results helpful in crafting a more deliberative knowledge translation strategy.

    Our study has several strengths. First, we used a large body of data from Twitter to track near–real-time conversations about HPV vaccination on the web. Twitter, in its previous iteration, was one of the largest and most popular social media platforms and was seen as a preferred source of public opinion data for applied public health research due to the following features: (1) quick processes for collecting data sets, (2) low costs for data collection, (3) ability to monitor trends over time, and (4) ability to avoid researcher biases that are inherent to the design and delivery of traditional research tools such as surveys [21,24]. Therefore, this data set provided us access to a large number of unfiltered discussions from populations that are traditionally difficult to access through conventional data collection methods.

    Next, our use of social network analysis allowed us to examine how scientific literature is shared and its connection within wider networks representing communities of interest. Thus, we were also able to identify key influencers within networks who potentially act as leverage points to amplify future health communication campaigns, while also shedding light on the density of vaccine-hesitant influencers compared to vaccine-confident influencers within the respective social networks. Finally, while the vaccine-hesitant community has attempted to use or distort scientific literature to support their viewpoints for a long time, to the best of our knowledge, this is the first study to examine how scientific evidence has been used and shared on the web by comparing both vaccine-hesitant and vaccine-confident web-based communities in discussions specifically related to the HPV vaccine.

    Conclusions

    Many of the communication strategies initially used by health promotion communities, including the use of the logical fallacy such as appealing to scientific authority and scientific knowledge, appear to have been co-opted by the vaccine-hesitant community and are being used to create controversy by focusing on questions about the effectiveness and safety of the HPV vaccine. While the scientific literature shared within these vaccine-hesitant communities is often published in lower-ranked journals, they deliver a substantially more successful, coordinated strategy when it comes to communicating about HPV vaccine on Twitter, compared to the vaccine-confident communities. By widely sharing a curated selection of scientific publications among like-minded individuals, the vaccine-hesitant community members’ communication around the HPV vaccine yields much more interaction (ie, shares and retweets) than is observed in the vaccine-confident community’s efforts to disseminate research findings. While the scientific literature shared by members of the vaccine-confident community is published in higher-ranked journals, these papers receive far fewer interactions and have lesser reach on Twitter.

    While the vaccine-hesitant community has successfully incorporated communication tools that were traditionally wielded by health promotion communities to advance their agenda, the web-based vaccine-confident community could benefit from paying attention to their dissemination techniques for using web-based platforms such as Twitter to amplify their messaging. However, it is crucial that the vaccine-confident community’s messages ultimately be transmitted in a manner that fosters long-term trust and credibility, which stems from accurate and transparent communication.

    Acknowledgments

    Funding was provided, in whole or in part, by Alberta Health. Strategic direction and applied research support were provided by the Alberta Health Services Cancer Prevention and Screening Innovation team. Provision of funding by Alberta Health does not indicate that this project represents the policies or views of Alberta Health.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Summary of the top 20 most shared scientific papers on Twitter by the vaccine-hesitant community.

    DOCX File , 21 KB
    Multimedia Appendix 2

    Summary of the top 20 most shared scientific papers on Twitter by the vaccine-confident community.

    DOCX File , 23 KB
    Multimedia Appendix 3

    Retweet network map of human papillomavirus immunization conversations (N=596,987).

    DOCX File , 457 KB
    1. Volesky KD, El-Zein M, Franco EL, Brenner DR, Friedenreich CM, Ruan Y. Corrigendum to "Cancers attributable to infections in Canada" [Prev. Med. 122 (2019) 109-117]. Prev Med. Dec 2019;129:105730. [CrossRef] [Medline]
    2. Global strategy to accelerate the elimination of cervical cancer as a public health problem. World Health Organization. Nov 17, 2020. URL: https://www.who.int/publications/i/item/9789240014107 [accessed 2024-02-05]
    3. Hopkins TG, Wood N. Female human papillomavirus (HPV) vaccination: global uptake and the impact of attitudes. Vaccine. Mar 25, 2013;31(13):1673-1679. [CrossRef] [Medline]
    4. Dubé E, Gagnon D, MacDonald N, Bocquier A, Peretti-Watel P, Verger P. Underlying factors impacting vaccine hesitancy in high income countries: a review of qualitative studies. Expert Rev Vaccines. Nov 2018;17(11):989-1004. [CrossRef] [Medline]
    5. Tonsaker T, Bartlett G, Trpkov C. Health information on the internet: gold mine or minefield? Can Fam Physician. May 2014;60(5):407-408. [FREE Full text] [Medline]
    6. Lutkenhaus RO, Jansz J, Bouman MP. Mapping the Dutch vaccination debate on Twitter: identifying communities, narratives, and interactions. Vaccine X. Mar 21, 2019;1:100019. [FREE Full text] [CrossRef] [Medline]
    7. Castells M. Networks of Outrage and Hope: Social Movements in the Internet Age, 2nd Edition. Hoboken, NJ. Wiley; Apr 2015.
    8. Baldwin AS, Tiro JA, Zimet GD. Broad perspectives in understanding vaccine hesitancy and vaccine confidence: an introduction to the special issue. J Behav Med. Apr 2023;46(1-2):1-8. [FREE Full text] [CrossRef] [Medline]
    9. Larson HJ, Gakidou E, Murray CJ. The vaccine-hesitant moment. N Engl J Med. Jul 07, 2022;387(1):58-65. [FREE Full text] [CrossRef] [Medline]
    10. Huh WK, Joura EA, Giuliano AR, Iversen OE, de Andrade RP, Ault KA, et al. Final efficacy, immunogenicity, and safety analyses of a nine-valent human papillomavirus vaccine in women aged 16-26 years: a randomised, double-blind trial. Lancet. Nov 11, 2017;390(10108):2143-2159. [CrossRef] [Medline]
    11. Arbyn M, Xu L. Efficacy and safety of prophylactic HPV vaccines. A Cochrane review of randomized trials. Expert Rev Vaccines. Dec 2018;17(12):1085-1091. [CrossRef] [Medline]
    12. Madden K, Nan X, Briones R, Waks L. Sorting through search results: a content analysis of HPV vaccine information online. Vaccine. May 28, 2012;30(25):3741-3746. [CrossRef] [Medline]
    13. Piwowar H, Priem J, Larivière V, Alperin JP, Matthias L, Norlander B, et al. The state of OA: a large-scale analysis of the prevalence and impact of open access articles. PeerJ. Feb 13, 2018;6:e4375. [FREE Full text] [CrossRef] [Medline]
    14. Bucchi M. Credibility, expertise and the challenges of science communication 2.0. Public Underst Sci. Nov 2017;26(8):890-893. [CrossRef] [Medline]
    15. Steffens MS, Dunn AG, Leask J. Meeting the challenges of reporting on public health in the new media landscape. Aust J Rev. Dec 2017;39(2):119-132.
    16. Barberá P. How social media reduces mass political polarization. Evidence from Germany, Spain, and the U.S. In: Proceedings of the American Political Science Association Annual Meeting 2015. 2015. Presented at: APSA 2015; September 3-6, 2015; San Francisco, CA. URL: http://pablobarbera.com/static/barbera_polarization_APSA.pdf
    17. van Schalkwyk F, Dudek J, Costas R. Communities of shared interests and cognitive bridges: the case of the anti-vaccination movement on Twitter. Scientometrics. Jun 13, 2020;125:1499-1416. [CrossRef]
    18. Boothby C, Murray D, Waggy AP, Tsou A, Sugimoto CR. Credibility of scientific information on social media: variation by platform, genre and presence of formal credibility cues. Quant Sci Stud. Nov 05, 2021;2(3):845-863. [CrossRef]
    19. Bello-Orgaz G, Hernandez-Castro J, Camacho D. Detecting discussion communities on vaccination in Twitter. Future Gener Comput Syst. Jan 2017;66:125-136. [CrossRef]
    20. Chakraborty P, Colditz JB, Silvestre AJ, Friedman MR, Bogen KW, Primack BA. Observation of public sentiment toward human papillomavirus vaccination on Twitter. Cogent Med. Dec 12, 2017;4(1). [CrossRef]
    21. Du J, Xu J, Song H, Liu X, Tao C. Optimization on machine learning based approaches for sentiment analysis on HPV vaccines related tweets. J Biomed Semantics. Mar 03, 2017;8(1):9. [FREE Full text] [CrossRef] [Medline]
    22. Du J, Xu J, Song HY, Tao C. Leveraging machine learning-based approaches to assess human papillomavirus vaccination sentiment trends with Twitter data. BMC Med Inform Decis Mak. Jul 05, 2017;17(Suppl 2):69. [FREE Full text] [CrossRef] [Medline]
    23. Kunneman F, Lambooij M, Wong A, van den Bosch A, Mollema L. Monitoring stance towards vaccination in Twitter messages. BMC Med Inform Decis Mak. Feb 18, 2020;20(1):33. [FREE Full text] [CrossRef] [Medline]
    24. Tavoschi L, Quattrone F, D'Andrea E, Ducange P, Vabanesi M, Marcelloni F, et al. Twitter as a sentinel tool to monitor public opinion on vaccination: an opinion mining analysis from September 2016 to August 2017 in Italy. Hum Vaccin Immunother. May 03, 2020;16(5):1062-1069. [FREE Full text] [CrossRef] [Medline]
    25. Du J, Luo C, Shegog R, Bian J, Chen Y, Tao C. Deep learning and behavioral theory: an improved analytic method to understand HPV vaccination intentions from Twitter discussion. SSRN J. Oct 03, 2019.:1-30. [FREE Full text] [CrossRef]
    26. Shah Z, Surian D, Dyda A, Coiera E, Mandl KD, Dunn AG. Automatically appraising the credibility of vaccine-related web pages shared on social media: a Twitter surveillance study. J Med Internet Res. Nov 04, 2019;21(11):e14007. [FREE Full text] [CrossRef] [Medline]
    27. Sanawi JB, Samani MC, Taibi M. #vaccination: identifying influencers in the vaccination discussion on Twitter through social network visualisation. Int J Bus Soc. Oct 2017;18(S4):718-726.
    28. Caulfield T, Marcon AR, Murdoch B, Brown JM, Perrault ST, Jarry J, et al. Health misinformation and the power of narrative messaging in the public sphere. Can J Bioeth. Mar 20, 2019;2(2):52-60. [CrossRef]
    29. Elyashar A, Plochotnikov I, Cohen IC, Puzis R, Cohen O. The state of mind of health care professionals in light of the COVID-19 pandemic: text analysis study of Twitter discourses. J Med Internet Res. Oct 22, 2021;23(10):e30217. [FREE Full text] [CrossRef] [Medline]
    30. Dobbins M. Rapid Review Guidebook. Hamilton, ON. National Collaborating Centre for Methods and Tools; 2017.
    31. Shapiro GK, Surian D, Dunn AG, Perry R, Kelaher M. Comparing human papillomavirus vaccine concerns on Twitter: a cross-sectional study of users in Australia, Canada and the UK. BMJ Open. Oct 05, 2017;7(10):e016869. [FREE Full text] [CrossRef] [Medline]
    32. Massey PM, Leader A, Yom-Tov E, Budenz A, Fisher K, Klassen AC. Applying multiple data collection tools to quantify human papillomavirus vaccine communication on Twitter. J Med Internet Res. Dec 05, 2016;18(12):e318. [FREE Full text] [CrossRef] [Medline]
    33. Keim-Malpass J, Mitchell EM, Sun E, Kennedy C. Using Twitter to understand public perceptions regarding the #HPV vaccine: opportunities for public health nurses to engage in social marketing. Public Health Nurs. Jul 2017;34(4):316-323. [CrossRef] [Medline]
    34. Nelon JL, Moscarelli M, Stupka P, Sumners C, Uselton T, Patterson MS. Does scientific publication inform public discourse? A case study observing social media engagement around vaccinations. Health Promot Pract. May 2021;22(3):377-384. [CrossRef] [Medline]
    35. Surian D, Nguyen DQ, Kennedy G, Johnson M, Coiera E, Dunn AG. Characterizing Twitter discussions about HPV vaccines using topic modeling and community detection. J Med Internet Res. Aug 29, 2016;18(8):e232. [FREE Full text] [CrossRef] [Medline]
    36. Zhou X, Coiera E, Tsafnat G, Arachi D, Ong MS, Dunn AG. Using social connection information to improve opinion mining: identifying negative sentiment about HPV vaccines on Twitter. Stud Health Technol Inform. 2015;216:761-765. [Medline]
    37. Becker BF, Larson HJ, Bonhoeffer J, van Mulligen EM, Kors JA, Sturkenboom MC. Evaluation of a multinational, multilingual vaccine debate on Twitter. Vaccine. Dec 07, 2016;34(50):6166-6171. [CrossRef] [Medline]
    38. Dyda A, Shah Z, Surian D, Martin P, Coiera E, Dey A, et al. HPV vaccine coverage in Australia and associations with HPV vaccine information exposure among Australian Twitter users. Hum Vaccin Immunother. 2019;15(7-8):1488-1495. [CrossRef] [Medline]
    39. Dunn AG, Surian D, Leask J, Dey A, Mandl KD, Coiera E. Mapping information exposure on social media to explain differences in HPV vaccine coverage in the United States. Vaccine. May 25, 2017;35(23):3033-3040. [FREE Full text] [CrossRef] [Medline]
    40. Budenz A, Massey P, Leader A, Klassen A, Fisher K, Yom-Tov E. Let's talk about HPV: a focus on gender- and MSM-specific Twitter communication. In: Proceedings of the American Public Health Association Annual Meeting 2017. 2017. Presented at: APHA 2017; November 4-8, 2017; Atlanta, GA.
    41. Zhang H, Wheldon C, Dunn AG, Tao C, Huo J, Zhang R, et al. Mining Twitter to assess the determinants of health behavior toward human papillomavirus vaccination in the United States. J Am Med Inform Assoc. Feb 01, 2020;27(2):225-235. [FREE Full text] [CrossRef] [Medline]
    42. X data for academic research. X Developer Platform. URL: https://developer.twitter.com/en/use-cases/do-research/academic-research [accessed 2024-02-05]
    43. X API documentation. X Developer Platform. URL: https://developer.twitter.com/en/docs/twitter-api [accessed 2024-02-05]
    44. Van Rossum G, Drake FL. Python 3 Reference Manual. North Charleston, SC. CreateSpace; Mar 2009.
    45. Yan Y, Toriumi F, Sugawara T. Understanding how retweets influence the behaviors of social networking service users via agent-based simulation. Comput Soc Netw. Sep 13, 2021;8:18. [CrossRef]
    46. Blondel VD, Guillaume JL, Lambiotte R, Lefebvre E. Fast unfolding of communities in large networks. J Stat Mech. Oct 09, 2008;2008:P10008. [CrossRef]
    47. Bastian M, Heymann S, Jacomy M. Gephi: an open source software for exploring and manipulating networks. Proc Int AAAI Conf Web Soc Media. Mar 19, 2009;3(1):361-362. [CrossRef]
    48. pandas.Series.str.contains. Pandas. URL: https://pandas.pydata.org/docs/reference/api/pandas.Series.str.contains.html [accessed 2024-02-05]
    49. Gray JA. Evidence-Based Healthcare. London, UK. Churchill Livingstone; 1997.
    50. Beppu H, Minaguchi M, Uchide K, Kumamoto K, Sekiguchi M, Yaju Y. Lessons learnt in Japan from adverse reactions to the HPV vaccine: a medical ethics perspective. Indian J Med Ethics. Apr 1, 2017.:82. [FREE Full text] [CrossRef]
    51. Rees CP, Brhlikova P, Pollock AM. Will HPV vaccination prevent cervical cancer? J R Soc Med. Feb 21, 2020;113(2):64-78. [FREE Full text] [CrossRef] [Medline]
    52. DeLong G. A lowered probability of pregnancy in females in the USA aged 25-29 who received a human papillomavirus vaccine injection. Retracted in: J Clin Hypertens (Greenwich). Jun 11, 2018;81(14):661-674. [CrossRef] [Medline]
    53. Hineno A, Ikeda S, Scheibenbogen C, Heidecke H, Forster K, Junker J, et al. Autoantibodies against autonomic nerve receptors in adolescent Japanese girls after immunization with human papillomavirus vaccine. Ann Arthritis Clin Rheumatol. 2019;2:1014. [FREE Full text]
    54. Atkinson AE, Mandujano CA, Bejarano S, Kennedy LS, Tsongalis GJ. Screening for human papillomavirus in a low- and middle-income country. J Glob Oncol. May 2019;5:JGO1800233. [FREE Full text] [CrossRef] [Medline]
    55. Bizjak M, Bruck O, Praprotnik S, Dahan S, Shoenfeld Y. Pancreatitis after human papillomavirus vaccination: a matter of molecular mimicry. Immunol Res. Mar 2017;65(1):164-167. [CrossRef] [Medline]
    56. Blitshteyn S, Brinth L, Hendrickson JE, Martinez-Lavin M. Autonomic dysfunction and HPV immunization: an overview. Immunol Res. Dec 2018;66(6):744-754. [CrossRef] [Medline]
    57. Chang J, Campagnolo D, Vollmer TL, Bomprezzi R. Demyelinating disease and polyvalent human papilloma virus vaccination. J Neurol Neurosurg Psychiatry. Nov 2011;82(11):1296-1298. [CrossRef] [Medline]
    58. Debeer P, De Munter P, Bruyninckx F, Devlieger R. Brachial plexus neuritis following HPV vaccination. Vaccine. Aug 18, 2008;26(35):4417-4419. [CrossRef] [Medline]
    59. Fischer S, Bettstetter M, Becher A, Lessel M, Bank C, Krams M, et al. Shift in prevalence of HPV types in cervical cytology specimens in the era of HPV vaccination. Oncol Lett. Jul 2016;12(1):601-610. [FREE Full text] [CrossRef] [Medline]
    60. Guo F, Hirth JM, Berenson AB. Comparison of HPV prevalence between HPV-vaccinated and non-vaccinated young adult women (20-26 years). Hum Vaccin Immunother. 2015;11(10):2337-2344. [FREE Full text] [CrossRef] [Medline]
    61. Inbar R, Weiss R, Tomljenovic L, Arango MT, Deri Y, Shaw CA, et al. Behavioral abnormalities in female mice following administration of aluminum adjuvants and the human papillomavirus (HPV) vaccine Gardasil. Immunol Res. Mar 2017;65(1):136-149. [CrossRef] [Medline]
    62. Jørgensen L, Gøtzsche PC, Jefferson T. Benefits and harms of the human papillomavirus (HPV) vaccines: systematic review with meta-analyses of trial data from clinical study reports. Syst Rev. Mar 28, 2020;9(1):43. [FREE Full text] [CrossRef] [Medline]
    63. Little DT, Ward HR. Adolescent premature ovarian insufficiency following human papillomavirus vaccination: a case series seen in general practice. J Investig Med High Impact Case Rep. 2014;2(4):2324709614556129. [FREE Full text] [CrossRef] [Medline]
    64. Masson JD, Crépeaux G, Authier FJ, Exley C, Gherardi RK. Critical analysis of reference studies on the toxicokinetics of aluminum-based adjuvants. J Inorg Biochem. Apr 2018;181:87-95. [CrossRef] [Medline]
    65. Castle PE, Xie X, Xue X, Poitras NE, Lorey TS, Kinney WK, et al. Impact of human papillomavirus vaccination on the clinical meaning of cervical screening results. Prev Med. Jan 2019;118:44-50. [CrossRef] [Medline]
    66. Ran J, Yang JY, Lee JH, Kim HJ, Choi JY, Shin JY. Signal detection of human papillomavirus vaccines using the Korea Adverse Events Reporting System database, between 2005 and 2016. Int J Clin Pharm. Oct 2019;41(5):1365-1372. [CrossRef] [Medline]
    67. Riva C, Spinosa JP. Has the HPV vaccine approval ushered in an era of over-prevention? J Sci Pract Integr. Dec 2020;2(1). [CrossRef]
    68. Ryan M, Marlow L, Waller J. Socio-demographic correlates of cervical cancer risk factor knowledge among screening non-participants in Great Britain. Prev Med. Aug 2019;125:1-4. [FREE Full text] [CrossRef] [Medline]
    69. Álvarez-Soria MJ, Hernández-González A, Carrasco-García de León S, del Real-Francia MÁ, Gallardo-Alcañiz MJ, López-Gómez JL. [Demyelinating disease and vaccination of the human papillomavirus]. Rev Neurol. Apr 16, 2011;52(8):472-476. [FREE Full text] [Medline]
    70. Tomljenovic L, Shaw CA. No autoimmune safety signal after vaccination with quadrivalent HPV vaccine Gardasil? J Intern Med. Nov 2012;272(5):514-5; author reply 516. [FREE Full text] [CrossRef] [Medline]
    71. Palmer T, Wallace L, Pollock KG, Cuschieri K, Robertson C, Kavanagh K, et al. Prevalence of cervical disease at age 20 after immunisation with bivalent HPV vaccine at age 12-13 in Scotland: retrospective population study. BMJ. Apr 03, 2019;365:l1161. [FREE Full text] [CrossRef] [Medline]
    72. Rebolj M, Rimmer J, Denton K, Tidy J, Mathews C, Ellis K, et al. Primary cervical screening with high risk human papillomavirus testing: observational study. BMJ. Feb 06, 2019;364:l240. [CrossRef]
    73. Lei J, Ploner A, Elfström KM, Wang J, Roth A, Fang F, et al. HPV vaccination and the risk of invasive cervical cancer. N Engl J Med. Oct 2020;383(14):1340-1348. [CrossRef]
    74. Torjesen I. HPV vaccine: high coverage could eradicate cervical cancer within decades, say researchers. BMJ. Jun 27, 2019;365:l4450. [CrossRef] [Medline]
    75. Brotherton JM. The remarkable impact of bivalent HPV vaccine in Scotland. BMJ. Apr 03, 2019;365:l1375. [CrossRef] [Medline]
    76. McBride E, Marlow LA, Forster AS, Ridout D, Kitchener H, Patnick J, et al. Anxiety and distress following receipt of results from routine HPV primary testing in cervical screening: the psychological impact of primary screening (PIPS) study. Int J Cancer. Apr 15, 2020;146(8):2113-2121. [FREE Full text] [CrossRef] [Medline]
    77. Ryan M, Marlow L, Waller J. Socio-demographic correlates of cervical cancer risk factor knowledge among screening non-participants in Great Britain. Prev Med. Aug 2019;125:1-4. [FREE Full text] [CrossRef] [Medline]
    78. Artemchuk H, Eriksson T, Poljak M, Surcel HM, Dillner J, Lehtinen M, et al. Long-term antibody response to human papillomavirus vaccines: up to 12 years of follow-up in the Finnish maternity cohort. J Infect Dis. Jan 29, 2019;219(4):582-589. [CrossRef] [Medline]
    79. Baker P, Kelly D, Medeiros R, Morrissey M, Price R. Eliminating HPV-caused cancers in Europe: achieving the possible. J Cancer Policy. Jun 2021;28:100280. [CrossRef] [Medline]
    80. Colafrancesco S, Perricone C, Tomljenovic L, Shoenfeld Y. Human papilloma virus vaccine and primary ovarian failure: another facet of the autoimmune/inflammatory syndrome induced by adjuvants. Am J Reprod Immunol. Oct 2013;70(4):309-316. [CrossRef] [Medline]
    81. Cromwell I, Smith LW, van der Hoek K, Hedden L, Coldman AJ, Cook D, et al. Cost-effectiveness analysis of primary human papillomavirus testing in cervical cancer screening: results from the HPV FOCAL Trial. Cancer Med. May 2021;10(9):2996-3003. [FREE Full text] [CrossRef] [Medline]
    82. Drolet M, Bénard É, Pérez N, Brisson M, HPV Vaccination Impact Study Group. Population-level impact and herd effects following the introduction of human papillomavirus vaccination programmes: updated systematic review and meta-analysis. Lancet. Aug 10, 2019;394(10197):497-509. [FREE Full text] [CrossRef] [Medline]
    83. Gee J, Naleway A, Shui I, Baggs J, Yin R, Li R, et al. Monitoring the safety of quadrivalent human papillomavirus vaccine: findings from the Vaccine Safety Datalink. Vaccine. Oct 26, 2011;29(46):8279-8284. [CrossRef] [Medline]
    84. Gleber-Netto FO, Rao X, Guo T, Xi Y, Gao M, Shen L, et al. Variations in HPV function are associated with survival in squamous cell carcinoma. JCI Insight. Jan 10, 2019;4(1):e124762. [FREE Full text] [CrossRef] [Medline]
    85. Kardas-Nelson M. Vaccine uptake and prevalence of HPV related cancers in US men. BMJ. Mar 18, 2019;364:l1210. [CrossRef] [Medline]
    86. Klein NP, Hansen J, Chao C, Velicer C, Emery M, Slezak J, et al. Safety of quadrivalent human papillomavirus vaccine administered routinely to females. Arch Pediatr Adolesc Med. Dec 2012;166(12):1140-1148. [CrossRef] [Medline]
    87. Nagarsheth NB, Norberg SM, Sinkoe AL, Adhikary S, Meyer TJ, Lack JB, et al. TCR-engineered T cells targeting E7 for patients with metastatic HPV-associated epithelial cancers. Nat Med. Mar 2021;27(3):419-425. [FREE Full text] [CrossRef] [Medline]
    88. Simms KT, Hanley SJ, Smith MA, Keane A, Canfell K. Impact of HPV vaccine hesitancy on cervical cancer in Japan: a modelling study. Lancet Public Health. Apr 2020;5(4):e223-e234. [FREE Full text] [CrossRef] [Medline]
    89. Hoss A, Meyerson BE, Zimet GD. State statutes and regulations related to human papillomavirus vaccination. Hum Vaccin Immunother. 2019;15(7-8):1519-1526. [FREE Full text] [CrossRef] [Medline]
    90. Petticrew M, Roberts H. Evidence, hierarchies, and typologies: horses for courses. J Epidemiol Community Health. Jul 2003;57(7):527-529. [FREE Full text] [CrossRef] [Medline]
    91. Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA. Jun 05, 2002;287(21):2847-2850. [CrossRef] [Medline]
    92. Scarlat MM, Mavrogenis AF, Pećina M, Niculescu M. Impact and alternative metrics for medical publishing: our experience with International Orthopaedics. Int Orthop. Aug 2015;39(8):1459-1464. [CrossRef] [Medline]
    93. Guerrero-Bote VP, Moya-Anegón F. A further step forward in measuring journals’ scientific prestige: the SJR2 indicator. J Informetr. Oct 2012;6(4):674-688. [CrossRef]
    94. Scimago journal and country rank homepage. Scimago Journal & Country Rank. URL: https://www.scimagojr.com/ [accessed 2023-07-04]
    95. McKnight PE, Najab J. Mann-Whitney U Test. In: The Corsini Encyclopedia of Psychology. Hoboken, NJ. John Wiley & Sons; 2010.
    96. Cohen J. Statistical Power Analysis for the Behavioral Sciences Second Edition. Hillsdale, NJ. Lawrence Erlbaum Associates; 1988.
    97. Kata A. Anti-vaccine activists, Web 2.0, and the postmodern paradigm--an overview of tactics and tropes used online by the anti-vaccination movement. Vaccine. May 28, 2012;30(25):3778-3789. [CrossRef] [Medline]
    98. Martini C. Ad hominem arguments, rhetoric, and science communication. Stud Log Gramm Rhetor. 2018;55(1):66. [CrossRef]
    99. Vivion M, Anassour Laouan Sidi E, Betsch C, Dionne M, Dubé E, Driedger SM, et al. Prebunking messaging to inoculate against COVID-19 vaccine misinformation: an effective strategy for public health. J Commun Healthc. Mar 04, 2022;15(3):232-242. [CrossRef]
    100. van der Linden S, Roozenbeek J, Maertens R, Basol M, Kácha O, Rathje S, et al. How can psychological science help counter the spread of fake news? Span J Psychol. Apr 12, 2021;24:e25. [CrossRef]
    101. Love B, Himelboim I, Holton A, Stewart K. Twitter as a source of vaccination information: content drivers and what they are saying. Am J Infect Control. Jun 2013;41(6):568-570. [CrossRef] [Medline]
    102. Meyer SB, Violette R, Aggarwal R, Simeoni M, MacDougall H, Waite N. Vaccine hesitancy and Web 2.0: exploring how attitudes and beliefs about influenza vaccination are exchanged in online threaded user comments. Vaccine. Mar 22, 2019;37(13):1769-1774. [CrossRef] [Medline]
    103. Caulfield T, Bubela T, Kimmelman J, Ravitsky V. Let’s do better: public representations of COVID-19 science. FACETS. Jan 01, 2021;6(1):403-423. [CrossRef]
    104. Wang JT, Power CJ, Kahler CM, Lyras D, Young PR, Iredell J, et al. Communication ambassadors-an Australian social media initiative to develop communication skills in early career scientists. J Microbiol Biol Educ. Mar 2018;19(1):19.1.25. [FREE Full text] [CrossRef] [Medline]
    105. Petersen MB, Bor A, Jørgensen F, Lindholt MF. Transparent communication about negative features of COVID-19 vaccines decreases acceptance but increases trust. Proc Natl Acad Sci U S A. Jul 20, 2021;118(29):e2024597118. [FREE Full text] [CrossRef] [Medline]
    106. Harzing AW, Alakangas S. Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison. Scientometrics. Nov 26, 2015;106(2):787-804. [CrossRef]
    107. Boucher JC, Kim SY, Jessiman-Perreault G, Edwards J, Smith H, Frenette N, et al. HPV vaccine narratives on Twitter during the COVID-19 pandemic: a social network, thematic, and sentiment analysis. BMC Public Health. Apr 14, 2023;23(1):694. [FREE Full text] [CrossRef] [Medline]

    ‎
    API: application programming interface
    HPV: human papillomavirus
    SJR: SCImago Journal Ranking

    Edited by T Mackey; submitted 05.07.23; peer-reviewed by A Elyashar, WG Woodall, I Ballalai; comments to author 28.11.23; revised version received 13.02.24; accepted 20.03.24; published 09.05.24.

    Copyright

    ©Geneviève Jessiman-Perreault, Jean-Christophe Boucher, So Youn Kim, Nicole Frenette, Abbas Badami, Henry M Smith, Lisa K Allen Scott. Originally published in JMIR Infodemiology (https://infodemiology.jmir.org), 09.05.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Infodemiology, is properly cited. The complete bibliographic information, a link to the original publication on https://infodemiology.jmir.org/, as well as this copyright and license information must be included.