This paper is in the following e-collection/theme issue:

Information Demand (32) Infodemiology and Infoveillance (1350) Information Seeking, Information Needs (467) Theme Issue: Novel Coronavirus (COVID-19) Outbreak Rapid Reports (1518) JMIR Theme Issue: COVID-19 Special Issue (2453) Infoveillance and Social Listening (209) COVID-19 Vaccination (147) Vaccination and Immunization in the Digital Age (374)

Published on in Vol 1, No 1 (2021): Jan-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/28740, first published .
Examining the Public’s Most Frequently Asked Questions Regarding COVID-19 Vaccines Using Search Engine Analytics in the United States: Observational Study

Examining the Public’s Most Frequently Asked Questions Regarding COVID-19 Vaccines Using Search Engine Analytics in the United States: Observational Study

Examining the Public’s Most Frequently Asked Questions Regarding COVID-19 Vaccines Using Search Engine Analytics in the United States: Observational Study

Authors of this article:

Nicholas B Sajjadi1 Author Orcid Image ;   Samuel Shepard1 Author Orcid Image ;   Ryan Ottwell2, 3 Author Orcid Image ;   Kelly Murray4 Author Orcid Image ;   Justin Chronister5 Author Orcid Image ;   Micah Hartwell1, 6 Author Orcid Image ;   Matt Vassar1, 6 Author Orcid Image
Article Authors Cited by (22) Tweetations (21) Metrics

    Original Paper

    1Office of Medical Student Research, College of Osteopathic Medicine, Oklahoma State University Center for Health Sciences, Tulsa, OK, United States

    2Department of Internal Medicine, University of Oklahoma School of Community Medicine, Tulsa, OK, United States

    3Department of Dermatology, St. Joseph Mercy Hospital, Ann Arbor, MI, United States

    4Department of Emergency Medicine, College of Osteopathic Medicine, Oklahoma State University Center for Health Sciences, Tulsa, OK, United States

    5Department of Internal Medicine, College of Osteopathic Medicine, Oklahoma State University Center for Health Sciences, Tulsa, OK, United States

    6Department of Psychiatry and Behavioral Sciences, College of Osteopathic Medicine, Oklahoma State University Center for Health Sciences, Tulsa, OK, United States

    *all authors contributed equally

    Corresponding Author:

    Nicholas B Sajjadi, BSc

    Office of Medical Student Research

    College of Osteopathic Medicine

    Oklahoma State University Center for Health Sciences

    1111 W 17th Street

    Tulsa, OK, 74104

    United States

    Phone: 1 9185821972

    Fax:1 9185821972

    Email: nicholas.sajjadi@okstate.edu


    Background: The emergency authorization of COVID-19 vaccines has offered the first means of long-term protection against COVID-19–related illness since the pandemic began. It is important for health care professionals to understand commonly held COVID-19 vaccine concerns and to be equipped with quality information that can be used to assist in medical decision-making.

    Objective: Using Google’s RankBrain machine learning algorithm, we sought to characterize the content of the most frequently asked questions (FAQs) about COVID-19 vaccines evidenced by internet searches. Secondarily, we sought to examine the information transparency and quality of sources used by Google to answer FAQs on COVID-19 vaccines.

    Methods: We searched COVID-19 vaccine terms on Google and used the “People also ask” box to obtain FAQs generated by Google’s machine learning algorithms. FAQs are assigned an “answer” source by Google. We extracted FAQs and answer sources related to COVID-19 vaccines. We used the Rothwell Classification of Questions to categorize questions on the basis of content. We classified answer sources as either academic, commercial, government, media outlet, or medical practice. We used the Journal of the American Medical Association’s (JAMA’s) benchmark criteria to assess information transparency and Brief DISCERN to assess information quality for answer sources. FAQ and answer source type frequencies were calculated. Chi-square tests were used to determine associations between information transparency by source type. One-way analysis of variance was used to assess differences in mean Brief DISCERN scores by source type.

    Results: Our search yielded 28 unique FAQs about COVID-19 vaccines. Most COVID-19 vaccine–related FAQs were seeking factual information (22/28, 78.6%), specifically about safety and efficacy (9/22, 40.9%). The most common source type was media outlets (12/28, 42.9%), followed by government sources (11/28, 39.3%). Nineteen sources met 3 or more JAMA benchmark criteria with government sources as the majority (10/19, 52.6%). JAMA benchmark criteria performance did not significantly differ among source types (χ24=7.40; P=.12). One-way analysis of variance revealed a significant difference in mean Brief DISCERN scores by source type (F4,23=10.27; P<.001).

    Conclusions: The most frequently asked COVID-19 vaccine–related questions pertained to vaccine safety and efficacy. We found that government sources provided the most transparent and highest-quality web-based COVID-19 vaccine–related information. Recognizing common questions and concerns about COVID-19 vaccines may assist in improving vaccination efforts.

    JMIR Infodemiology 2021;1(1):e28740

    doi:10.2196/28740

    Keywords

    content; COVID-19; frequently asked questions; internet; machine learning; natural language processing; quality; question; SARS-CoV-2; search analytics; search engine; transparency; vaccine hesitancy; vaccine; web-based health information 


    As of August 01, 2021, COVID-19 has affected over 198 million people and has been responsible for over 4.2 million deaths worldwide [1,2]. In response to the pandemic, the US Food and Drug Administration issued emergency use authorizations for 2 COVID-19 vaccines in late 2020, 1 manufactured by Pfizer-BioNTech and the second by Moderna [3,4]. Overcoming logistical barriers will be crucial for enabling successful vaccine campaigns. Additionally, addressing the public’s perception of COVID-19 vaccines and the quality of available information is vital for promoting positive public reception and reducing vaccine hesitancy. Vaccine hesitancy, which refers to reluctance or refusal to receive vaccines, is complex and is determined by numerous factors such as trust in vaccine safety and efficacy, perceived risk of receiving or refusing a vaccine, and accessibility to and affordability of vaccines [5]. Hesitancy toward COVID-19 vaccines may hinder successful vaccination efforts.

    The pace of vaccine development, misinformation, and overall growth in vaccine hesitancy are factors potentially contributing to COVID-19 vaccine refusal [5,6]>. Identifying factors associated with COVID-19 vaccine refusal may assist in developing strategies to reduce vaccine hesitancy. To identify demographic factors associated with COVID-19 vaccine acceptance, Lazarus et al [7] surveyed individuals in 19 countries and reported that individuals who reported a high degree of trust in the government were more likely to report vaccine acceptance than those with low trust. In the United States, a survey study by the US Census Bureau showed that 49% of respondents were reluctant to receive a COVID-19 vaccine. Of those reluctant to receive COVID-19 vaccines, the most common reason for reluctance was concern for side effects. The second most common reason was planning to wait and see if the vaccines were safe [8]. A US survey conducted early in the pandemic sought to predict COVID-19 vaccine acceptance in the United States and found that several vulnerable populations reported low willingness [9]. The growing prevalence of vaccine hesitancy highlights the importance of clinician preparedness to address patients’ concerns as access to COVID-19 vaccines grows. Health care professionals should serve as reliable sources of vaccine information, instilling confidence in patients and potentially enhancing vaccine acceptance [10], especially for COVID-19 vaccines [11].

    Apart from consulting health care professionals, individuals frequently use the internet when seeking health care information; some use the internet as their primary source for health information [12]. In the United States, 61% of adults have searched the internet for medical information [13]. Searching the internet for medical information simultaneously presents benefits and challenges regarding patient-provider interactions [14]. The increasingly common practice of using the internet to obtain health care information makes it possible to study commonly held medical concerns by examining searching patterns and behaviors. Previous studies have documented the prevalence of COVID-19 vaccine hesitancy in the United States [8,15] and globally [7], but none of these studies explored the content of COVID-19 vaccine concerns evidenced by internet searching. Moreover, the quality of COVID-19 vaccine information resulting from internet searching has yet to be investigated. Thus, the primary objective of this study was to use Google’s RankBrain machine learning algorithm to characterize the content of the most frequently asked questions (FAQs) about COVID-19 vaccines in the United States. Secondarily, we sought to grade the transparency and quality of suggested information regarding COVID-19 vaccines. We aim to equip health care professionals and researchers with information about the common concerns regarding COVID-19 vaccines, possibly supporting more successful vaccination efforts. We hypothesize that most COVID-19 vaccine–related FAQs in the United States will pertain to safety and efficacy, as survey studies have indicated these concerns as the most important driver of COVID-19 vaccine hesitancy in the United States.


    Background

    We used Google to perform our search as it is the most frequently used search engine globally as of 2015 [16]. Moreover, Google’s search engine uses a powerful machine learning system called RankBrain [17] alongside the natural language processing technology known as Bidirectional Encoder Representations from Transformers [18] to detect patterns from large volumes of search queries. Google assesses the intent of a search query using rigorous language processing algorithms to sort through billions of indexed webpages and to suggest the ones most relevant to the search [19]. The resulting patterns and data are used to formulate lists of FAQs related to the original search contents. FAQs are found in boxes labeled “People also ask” or “Common questions.” Google assigns each FAQ a link to information that “answers” the question [20]. Google uses its webmaster guidelines to remove low-quality spam websites from search results and prioritize high-quality sources using a system called PageRank [19]. Taken together, these FAQs represent millions of common inquiries regarding medical information. Linked answers to each FAQ reveal which information sources individuals are likely to encounter when searching Google for medical information. Our methodology was adapted from a study by Shen et al [21], who used Google FAQs to reliably reveal common concerns about orthopedic procedures and to assess the transparency of the suggested information.

    Systematic Search

    On January 23, 2021, using a newly installed web browser to minimize personalized advertisement algorithms, we separately searched Google [22] for the following three terms: “covid 19 vaccine,” “pfizer covid vaccine,” and “moderna covid vaccine.” We selected these terms to capture the most likely general inquiries concerning the only 2 COVID-19 vaccines available at the time of our search. For each inquiry, we refreshed the list of FAQs found in the “Common questions” or “People also ask” box generated by Google. By expanding the tab on a FAQ, additional FAQs appear. We repeated this process until reaching a minimum of 150 FAQs for each search, as studies using similar methodology have recommended using 50-150 sources [21]. We used the high end of the recommended number of sources (150) for two reasons: to increase the likelihood of encountering an FAQ that would be pertinent to the current study and to reflect the precedent set in the literature. Since query results are tailored to the user’s location, search history, and search settings, we used clean browsers to minimize any influence of history and settings while allowing results to reflect queries from the United States [19].

    Data Extraction

    Of the resultant FAQs, we extracted only those directly pertaining to or mentioning COVID-19 vaccines along with their answer links. In a masked duplicated fashion, investigators NS and SS extracted these data using a Google Form on January 23, 2021. FAQ data extraction was completed on January 23, 2021. After extraction, any duplicate FAQs from the individual searches were removed, followed by the removal of any duplicate FAQs among the 3 searches. After the screening and reduction process, our searches resulted in a compilation of unique FAQs regarding COVID-19 vaccines.

    Question Classification and Answer Source Type

    Applying methodology adapted from previous studies [16,21], we first used the Rothwell Classification of Questions [23] to categorize FAQs under three broad categories: fact, policy, and value. Fact questions were further subclassified into four groups: safety and efficacy, vaccine administration schedule, cost, and technical details. Policy questions were subclassified into two groups: indications and complications. Value questions were subclassified into two groups: evaluation of credibility and appraisal of risk or benefit. Next, we categorized answer sources as either commercial, academic, medical practice, government, or media outlet according to previously established classification schemes [21,24]. Table 1 shows the Question Classification and Answer Source Type definitions. For each answer source, we extracted the country of origin.

    Table 1. The Rothwell Classification of Questions, Question Classification by Topic, and Answer Source Type.
    Rothwell classificationDescription
    FactAsks objective, factual information regarding COVID-19 vaccines (ie, “How long does it take the vaccine to work?”)
    PolicyAsks information on a specific course of action under given circumstances related to COVID-19 (ie, should people on immunosuppressants get the vaccine?)
    ValueAsks to conceptually evaluate COVID-19 vaccines (ie, “Will the COVID-19 vaccine work better than masks?”)
    Question subclassification by topic

    		Fact


    		Safety and efficacyQuestions about vaccine safety including side effects and how well the vaccine works


    		Vaccine administration scheduleSpecific questions about the vaccine schedule, number of shots, and vaccine distribution


    		CostCost of the vaccine, whether it is free, or who is paying for it


    		Technical DetailsMechanism by which the vaccine works, including specific questions about immunologic responses

    		Policy


    		IndicationsWho should or should not receive a COVID-19 vaccine


    		ComplicationsQuestions about specific complications after being vaccinated

    		Value


    		Evaluation of CredibilitySeeking authoritative approval from a trustworthy source; seeking ethos


    		Appraisal of Risk or BenefitNecessity of preventive measures after vaccination (ie, “Is getting vaccinated worth it?”)
    Answer source type

    		CommercialOrganization that publishes medical information that is not otherwise associated with an academic institution, government agency, health care system, or nonmedical news outlet such as WebMD and Healthline

    		AcademicInstitution with clear academic affiliations, as evidenced by information on the website that did not better meet criteria for another classification or website ending in “.edu,” such as Mayo Clinic and Harvard University

    		Medical practiceAffiliation with a health care system or individual health care professional who did not explicitly state a commercial, academic, or government affiliation, such as private practice and a hospital system

    		GovernmentWebsites hosted by government organizations or sources from websites ending in “.gov,” such as the Centers for Disease Control and the US Food and Drug Administration

    		Media outletNonmedical organizations or social media pages claiming to publish news-related stories for the purpose of information-sharing in the form of interviews, blog posts, or articles, such as the National Public Radio, Wall Street Journal, and USA Today

    Information Transparency and Quality

    The Journal of the American Medical Association’s (JAMA’s) benchmark criteria [25] was then used to assess information transparency for each answer source. JAMA benchmark criteria have been used to effectively screen web-based information for fundamental aspects of information transparency [21,26-28]. JAMA benchmark criteria were also used to characterize web-based misinformation regarding COVID-19 in early 2020 [29]. Sources meeting 3 more criteria are considered to have high transparency, while sources meeting less than 3 criteria have poor transparency. Table 2 lists the JAMA benchmark criteria definitions.

    Table 2. Journal of the American Medical Association’s benchmark criteria.
    CriteriaDescription
    AuthorshipClearly identifiable author and contributors with affiliations and relevant credentials present.
    AttributionReferences and sources clearly listed with any copyright information disclosed.
    CurrencyClearly identifiable posting date of any content as well as the date of any revisions.
    DisclosureWebsite ownership clearly disclosed along with any sponsorship, advertising, underwriting, and financial support.

    The information quality was assessed using the Brief DISCERN information quality assessment tool. DISCERN is a series of questions originally developed by Charnock et al [30] as a means for patients and providers to quickly and reliably ascertain the quality of written health care information regarding medical treatments. The DISCERN quality assessment tool has been used to assess the quality of internet sources in a variety of medical fields [31-33]. Khazaal et al [34] developed an abbreviated 6-item version (Brief DISCERN) with comparable reliability and validity, which preserves the advantages of the original tool while affording a potentially more user-friendly format. Thus, we used the Brief DISCERN quality assessment tool, which has been previously used [35,36]. Sources are scored from 1 to 5 based on the criteria listed in Table 3.

    Authors NS and SS applied the JAMA benchmark criteria and the Brief DISCERN tool in a masked duplicate fashion, and author MH resolved any discrepancies. This protocol was submitted to the institutional review board of Oklahoma State University Center for Health Sciences and was determined to be non–Human Subjects Research.

    Table 3. Brief DISCERN questions and scoring.
    QuestionsScore

    		Low (1) “No”Moderate (2-4) “Partially”High (5) “Yes”
    Is it clear what sources of information were used to compile the publication (other than the author or producer)?No sources of evidence for the information are mentionedThe sources are clear to some extent and are referenced in the text or in a bibliographyThe sources are very clear and are referenced in text and in a bibliography
    Is it clear when the information used or reported in the publication was produced?No dates have been givenOnly the date of the publication itself is clear, or dates for some of but not all acknowledged sources are givenDates for all acknowledged sources are clear
    Does it describe how each treatment works?None of the descriptions about treatments include details of how it worksDescriptions of some but not all treatments are given or the details provided are unclear or incompleteThe description of treatment includes details of how it works
    Does it describe the benefits of each treatment?No benefits are describedA benefit is described for some but not all treatmentsA benefit is described for each treatment
    Does it describe the risk of each treatment?No risks are described for any of the treatments.A risk is described for some but not all treatments.A risk is described for each treatment.
    Does it describe how the treatment choices affect overall quality of life?There is no reference to overall quality of life in relation to treatment choices.The publication includes a reference to overall quality of life in relation to treatment choices, but the information is unclear or incomplete.The publication includes a clear reference to overall quality of life in relation to any of the treatment choices mentioned.

    Analyses

    Frequencies and percentages were reported for each FAQ’s classification. Chi-square tests were used to determine associations between JAMA benchmark criteria by source type. One-way analysis of variance was used to determine whether the mean Brief DISCERN score differed by source type. Post hoc comparisons, performed using t tests with Bonferroni correction, were used to identify mean differences between source type categories. Interrater agreement for each assessment was determined using intraclass correlation coefficients.


    A total of 467 FAQs were generated from all 3 searches: 161 from “covid 19 vaccine,” 155 from “moderna covid vaccine,” and 151 from “pfizer covid vaccine.” Of these, “covid 19 vaccine” yielded 5 vaccine-related FAQs, “moderna covid vaccine” yielded 22, and “pfizer covid vaccine” yielded 14. After removing duplicates, our searches yielded a total of 28 unique FAQs regarding COVID-19 vaccines (Table 4).

    Table 4. List of the 28 unique frequently asked questions regarding COVID-19 vaccines.
    Frequently asked questionsRothwell classificationSubclassificationAnswer sourceJAMA benchmark criteria (≥3)Brief DISCERN score
    Are both Covid vaccines 2 doses?FactVaccine administration scheduleCommercialNo15
    Are you immune to Covid after vaccine?FactSafety and efficacyMedia outletNo21
    Can I get COVID-19 right after being vaccinated?FactTechnical detailsGovernmentYes29
    Can the COVID-19 vaccine make you sick?FactSafety and efficacyGovernmentYes29
    Can you still get Covid after first vaccine?FactTechnical detailsMedia outletNo18
    Can you test positive for Covid after vaccine?FactTechnical detailsMedia outletNo9
    Do COVID-19 vaccines require more than one shot?FactVaccine administration scheduleGovernmentYes29
    Do you have to wait 90 days after Covid to get the vaccine?FactVaccine administration scheduleMedia outletYes15
    Do you have to wear mask after Covid vaccine?ValueRisk/benefit appraisalMedia outletYes28
    Does Covid vaccine Stop Spread?ValueRisk/benefit appraisalMedia outletYes22
    Has the Pfizer-BioNTech COVID-19 vaccine been authorized by the FDA?FactSafety and efficacyGovernmentYes30
    How does the COVID-19 mRNA vaccine work?FactTechnical detailsGovernmentNo25
    How effective is the Pfizer COVID-19 vaccine?FactSafety and efficacyMedia outletNo17
    How long do you have to wait between Covid vaccines?FactVaccine administration scheduleMedia outletNo16
    How many shots of Moderna COVID-19 vaccine should I get?FactVaccine administration scheduleGovernmentYes29
    Is it safe to take the COVID-19 vaccine?FactSafety and efficacyGovernmentYes29
    Is the Moderna vaccine for COVID-19 approved by the FDA?FactSafety and efficacyAcademicYes30
    Should you get the Covid vaccine if you were previously infected with Covid?PolicyIndicationsMedia outletYes15
    What are some common side effects of the COVID-19 vaccine?FactSafety and efficacyGovernmentYes29

    Question Classification

    Using the Rothwell classification system, the majority of FAQs were seeking factual information (22/28;78.6%). Among these factual questions, the most common topic was safety and efficacy (9/22, 40.9%) followed by technical details (6/22, 27.3%), vaccine administration schedule (6/22, 27.3%), and cost (1/22, 4.5%) (Table 4).

    Answer Sources

    The most common answer source type overall was media outlets (12/28, 42.9%), followed by government sources (11/28, 39.3%), commercial sources (3/28, 10.7%), academic sources (1/28, 3.55%), and medical practice (1/28, 3.55%). FAQs classified as technical details were most frequently answered by a media outlet (4/6, 66.7%). Of FAQs classified as fact, most were answered by government sources (11/22, 50%). Government sources also most commonly answered FAQs related to safety and efficacy (5/9, 55.6%), cost (1/1, 100%), and vaccine administration schedule (3/6, 50%) (Table 4). In total, 26 of 28 (92.8%) answer sources were from the United States, 1 was from the United Kingdom (3.6%), and 1 was from Australia (3.6%).

    Information Transparency

    In total, 19 sources met 3 or more JAMA benchmark criteria, of which government sources were the majority (10/19, 52.6%), followed by media outlets (7/19, 36.8%), commercial sources (1/19, 5.3%), and academic sources (1/19, 5.3%). Among sources meeting less than 3 criteria, media outlets were the most common (5/9, 55.6%), followed by commercial sources (2/9, 22.2%), medical practice (1/9, 11.1%), and government sources (1/9, 11.1%). Approximately 92.7% (11/12) of government sources met 3 or more JAMA benchmark criteria, whereas 58.3% (7/12) of media outlets met 3 or more criteria. The overall JAMA Benchmark Criteria performance did not significantly differ among source types (χ24=7.40; P=.12); however, we found significant associations between individual source’s performance on meeting JAMA benchmark criteria for authorship and the source type (χ24=18.03, P<.001), with 11/28 (39.3%) media outlet sources meeting authorship criteria compared to 10/28 (35.7%) government sources not meeting the authorship criteria. We also found a similar but negative relationship with JAMA benchmark criteria’s disclosure criteria and source type (χ24=15.36; P=.004) with 10/28 (35.7%) government sources meeting these criteria compared to 9/28 (32.1%) media outlets not meeting these criteria (Tables 5 and 6).

    Table 5. Journal of the American Medical Association’s benchmark criteria and by source type.
    Sources meeting 3 or more JAMA benchmark criteriaSource type, n (%)TotalChi-square (df)P value

    		AcademicCommercialGovernmentMedical practiceMedia outlet
    Journal of the American Medical Association’s benchmark criteria7.40 (4).12

    		3+1 (3.6)1 (3.6)10 (35.7)0 (0.0)7 (25.0)19 (67.9)

    		<30 (0.0)2 (7.1)1 (3.6)1 (3.6)5 (17.9)9 (32.1)
    Authorship18.03 (4).001

    		No1 (3.6)2 (7.1)10 (35.7)0 (0.0)1 (3.6)14 (50.0)

    		Yes0 (0.0)1 (3.6)1 (3.6)1 (3.6)11 (39.3)14 (50.0)
    Attribution7.21 (4).13

    		No0 (0.0)2 (7.1)1 (3.6)1 (3.6)4 (14.3)8 (28.9)

    		Yes1 (3.6)1 (3.6)10 (35.7)0 (0.0)8 (28.9)20 (71.4)
    Currency1.60 (4).81

    		No0 (0.0)0 (0.0)1 (3.6)0 (0.0)0 (0.0)1 (3.6)

    		Yes1 (3.6)3 (10.7)10 (35.7)1 (3.6)12 (42.9)27 (96.4)
    Disclosure15.36 (4).004

    		No0 (0.0)3 (10.7)1 (3.6)1 (3.6)9 (32.1)14 (50.0)

    		Yes1 (3.6)0 (0.0)10 (35.7)0 (0.0)3 (10.7)14 (50.0)
    Table 6. Brief DISCERN scores by source type.

    		Source typeAverage (SD)F value (df)P value

    		AcademicCommercialGovernmentMedical practiceMedia outlet
    Brief DISCERN score, mean (SD)30.0 (0.0)17 (2.6)28.6 (1.4)18.0 (0.0)19.6 (5.6)23.2 (6.2)10.27 (4, 23)<.001

    Information Quality

    ANOVA revealed significant differences in mean Brief DISCERN scores by source type (F4,23=10.27; P<.001), suggesting important differences in quality among the different source types. Post hoc analysis with Bonferroni correction revealed significant differences in Brief DISCERN scores between government and commercial sources (P=.002) and between government sources and media outlets (P<.001). Mean (SD) values of Brief DISCERN scores by source are provided in Table 6. Interrater agreement for our analyses was high (interclass correlation=0.96; 95% CI 0.95-0.97).


    Principal Findings

    Using Google and its search analytics, we were able to identify the most frequently asked questions regarding COVID-19 vaccines in the United States. Google generated these FAQs by using millions of search queries nationwide. Additionally, we evaluated the assigned “answer” source for each FAQ, assessing each source’s information transparency and quality. To our knowledge, this study is the first of its kind to evaluate the public’s most frequently asked questions concerning the COVID-19 vaccines in the United States using Google search analytics. Our study is also the first of its kind to identify common answer sources used to address COVID-19 vaccine–related concerns and to assess their transparency and quality. In the following discourse, we discuss the importance of knowing COVID-19 FAQs in the context of the current COVID-19 vaccination campaigns while also providing recommendations for improving the public’s confidence and willingness to be vaccinated.

    FAQs

    The most popular COVID-19 vaccine–related questions sought factual information regarding safety and efficacy, indicating greater public concern regarding these topics. Consistent with our findings, survey studies found that safety and efficacy were among the most common COVID-19 vaccine concerns reported by the public and health care workers [37-40]. Additionally, studies have identified safety concerns as being one of the most common reasons for COVID-19 vaccine hesitancy [8,38-42]. In the United States, surveys indicate that 10% to 20% of adults and an estimated 8% of health care workers will refuse COVID-19 vaccines [8,37,39,43]. While the willingness to receive the COVID-19 vaccines has increased, the alarmingly high percentage of adults refusing vaccination creates a significant barrier to protecting our most vulnerable populations [43-45]. The potential cost of vaccine hesitancy and refusal in the United States is not exclusive to the COVID-19 pandemic. For example, an outbreak of measles virus, a pathogen for which vaccines effectively control outbreaks, occurred in Clark County, Washington, in 2019 [46]. Of 71 individuals involved, 61 (86%) were unvaccinated and 52 (73%) were children [46,47]. Moreover, vaccination rates in Clark County have been 10%-14% below the national average (88%) since 2013. The measles outbreak in 2019 was estimated to cost US $3.3 million to $3.5 million in labor, direct medical costs, and productivity losses [48]. It is likely that the cost of the Clark County measles outbreak could have been mitigated or reduced with adequate vaccination [47]. Thus, to prevent similar, but likely far worse, outcomes with COVID-19, effectively educating the public on the safety of COVID-19 vaccines is paramount for enhancing COVID-19 vaccine acceptance [49].

    Answer Sources

    Overall, COVID-19 vaccine FAQs were most often answered by media outlets, followed by government sources. FAQs about safety and efficacy were answered more often by government sources, while media outlets frequently answered FAQs about technical details. The answer sources linked to each FAQ are found in “People also ask” or “Common concerns” boxes and are direct answers generated by Google [50]. These direct answers are supplied from Google’s “trusted entities” database and are based on relational topics and machine learning [50]. While “trusted entities” seems rather vague, it appears that Google considers direct answers to be “trusted” based on clarity, completeness, and the lack of excessive promotional jargon. With the public’s trust and willingness to accept the vaccine being a key element in a successful vaccination campaign [44,51-53], it may be more appropriate for direct answers addressing COVID-19 vaccine FAQs to be based on scientific integrity, objectivity, and transparency.

    Transparency and Quality of the Answer Source

    The FAQs with direct answers from government sources were more likely to meet 3 or more JAMA benchmark criteria, indicating that government answers were more transparent. Additionally, government and academic sources were found to be of significantly higher quality. While media outlets are unquestionably an important source of health information to the public, these findings suggest that government sources may be better for addressing the public’s COVID-19 vaccine concerns. Although media outlets had moderate transparency and quality, there are notable reasons to use more reliable and objective sources. Generally, COVID-19 misinformation is rampant and the public opinion can be easily manipulated [29,45]. Indeed, media outlets are a frequent source of COVID-19 misinformation, and false claims are amplified by widespread news coverage [29,54]. For example, news stories early in the pandemic touting hydroxychloroquine as a “cure” perpetuated this misinformation in the absence of evidence [55]. More recently and more specifically related to the COVID-19 vaccines, rumors that COVID-19 vaccines cause infertility in women have circulated on social media [56]. Lastly, the politicization and polarization of news coverage surrounding the COVID-19 pandemic heavily influenced the public’s attitude to COVID-19 response policies [55,57-60]. Taken together, trouble with media outlets as trustworthy sources further supports the use of unbiased answer sources such as government agencies.

    Recommendations

    Above all, we recommend that individuals consider health care professionals as the primary source of information regarding COVID-19 vaccines. However, in cases where access to a health care professional is limited, web-based sources unquestionably present opportunities to quickly provide high-quality and accurate information regarding COVID-19 vaccines. We agree with Mills and Sivelä [61] that a successful COVID-19 vaccination campaign depends on gaining the public’s trust in health care systems and government agencies, such as the Centers for Disease Control and Prevention and the World Health Organization, while also minimizing vaccine misinformation. Additionally, government sources must strive to translate scientifically dense literature into easily understandable information that answers widespread concerns. Therefore, the dissemination of this study’s findings may promote the public’s trust in these institutions as we have shown that government and academic sources provided the most transparent and highest-quality information addressing COVID-19 vaccine–related concerns.

    Google recently demonstrated their willingness to support these COVID-19 vaccination campaigns by collaborating with Ohio State University to combat COVID-19 misinformation [62]. This partnership aims to ensure that people receive accurate information about COVID-19 vaccines to increase the public’s confidence and willingness to be vaccinated. Thus, in alignment with Google’s current intentions, we recommend that all COVID-19 vaccine FAQs be linked to government and academic answer sources; this would provide people with transparent and quality vaccine information. At a minimum, FAQs on safety and efficacy should be answered by government sources, as safety and efficacy concerns are among the primary drivers of COVID-19 vaccine hesitancy [39-42].

    Strength and Limitations

    Our study’s primary strength is the incorporation of Google FAQs as a novel source of insight regarding millions of individual inquiries about COVID-19 vaccines, which is an application of methodology adapted from the published literature [21,26-28,34,35] and improved upon herein. Using FAQs generated by Google to explore the content of concerns regarding COVID-19 vaccine safety and efficacy may prevent common limitations of survey studies such as low response rates, reporting biases, and selection bias. Additionally, Google’s large data set is continuously analyzed in real time and may offer improved and more specific targets when approaching the public’s medical concerns. All classifications and assessments were performed in a masked duplicate fashion in accordance with standards set by the Cochrane Review and experts in the meta-research field [63,64] with high interrater reliability between investigators.

    Our study is not without limitations though, such as those due to the dynamic nature of Google’s search outputs. As searching for COVID-19 vaccine–related information continues, new and updated FAQs will be generated, limiting the generalizability of our study to the time when our search was performed. Additionally, the transparency and quality assessments we used do not check for information accuracy, as this would require source-by-source comparison to generally accepted truths regarding COVID-19 vaccines, rendering our assessments as gauges of information transparency and not of information accuracy. Lastly, the categorizing of FAQs and answer sources was limited owing to their subjectivity. Although the categories were developed in line with previous reports and had high interobserver reliability, there is still potential for overlap between categories.

    Conclusions

    The expedient development and approval of COVID-19 vaccines is the culmination of the world’s greatest scientific achievements; however, without positive public reception and adequate counseling and education, COVID-19 vaccination efforts may be hindered. Using Google allowed us to obtain a list of FAQs based on millions of searches for content related to COVID-19 vaccines, which reflected widespread and common concerns. We found that the most common COVID-19 vaccine–related questions pertained to vaccine safety and efficacy, which is supported by the findings of survey studies. We found that government and academic sources provided the most transparent and highest-quality web-based information for answering the public’s most frequently asked questions about COVID-19 vaccines. Recognizing common concerns about COVID-19 vaccines may better assist health care professionals, researchers, and government agencies in improving vaccination efforts. Ensuring a successful vaccination campaign requires the public’s trust, which may be enhanced through the availability of high-quality and transparent COVID-19 vaccine information, such as that provided by government sources.

    Conflicts of Interest

    Author KM declares that her college received research grant funding as a part of a study funded by Eli Lilly and the National Institute of Allergy and Infectious Diseases of which she was a sub-investigator. The other authors declared no conflicts of interest.

    1. Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis 2020 May;20(5):533-534 [FREE Full text] [CrossRef] [Medline]
    2. COVID-19 Dashboard. Johns Hopkins Coronavirus Resource Center.   URL: https://coronavirus.jhu.edu/map.html [accessed 2021-08-01]
    3. Pfizer-BioNTech COVID-19 Vaccine. US Food and Drug Aminidtration.   URL: https:/​/www.​fda.gov/​emergency-preparedness-and-response/​coronavirus-disease-2019-covid-19/​pfizer-biontech-covid-19-vaccine [accessed 2021-01-21]
    4. Moderna COVID-19 Vaccine. US Food and Drug Administration.   URL: https:/​/www.​fda.gov/​emergency-preparedness-and-response/​coronavirus-disease-2019-covid-19/​moderna-covid-19-vaccine [accessed 2021-01-21]
    5. MacDonald NE, SAGE Working Group on Vaccine Hesitancy. Vaccine hesitancy: Definition, scope and determinants. Vaccine 2015 Aug 14;33(34):4161-4164 [FREE Full text] [CrossRef] [Medline]
    6. Kennedy J. Vaccine Hesitancy: A Growing Concern. Paediatr Drugs 2020 Apr;22(2):105-111. [CrossRef] [Medline]
    7. Lazarus JV, Ratzan SC, Palayew A, Gostin LO, Larson HJ, Rabin K, et al. Author Correction: A global survey of potential acceptance of a COVID-19 vaccine. Nat Med 2021 Feb;27(2):354 [FREE Full text] [CrossRef] [Medline]
    8. File T, Mohanty A. Around Half of Unvaccinated Americans Indicate They Will “Definitely” Get COVID-19 Vaccine. United States Census Bureau. 2021 Jan 27.   URL: https:/​/www.​census.gov/​library/​stories/​2021/​01/​around-half-of-unvaccinated-americans-indicate-they-will-definitely-get-covid-19-vaccine.​html [accessed 2021-02-25]
    9. Kelly BJ, Southwell BG, McCormack LA, Bann CM, MacDonald PDM, Frasier AM, et al. Predictors of willingness to get a COVID-19 vaccine in the U.S. BMC Infect Dis 2021 Apr 12;21(1):338 [FREE Full text] [CrossRef] [Medline]
    10. Shen SC, Dubey V. Addressing vaccine hesitancy: Clinical guidance for primary care physicians working with parents. Can Fam Physician 2019 Mar;65(3):175-181 [FREE Full text] [Medline]
    11. Bailey SR. Physicians provide key voice in building vaccine confidence. American Medical Association. 2021 Feb 25.   URL: https://tinyurl.com/3jzer8s7 [accessed 2021-01-27]
    12. Gualtieri L. The doctor as the second opinion and the internet as the first. In: CHI '09 Extended Abstracts on Human Factors in Computing Systems. 2009 Presented at: CHI '09: CHI Conference on Human Factors in Computing Systems; April 4-9, 2009; Boston, MA p. 2489-2498   URL: https://dl.acm.org/doi/abs/10.1145/1520340.1520352 [CrossRef]
    13. Cohen RA, Adams PF. Use of the internet for health information: United States, 2009. NCHS Data Brief 2011 Jul(66):1-8 [FREE Full text] [Medline]
    14. Tan SS, Goonawardene N. Internet Health Information Seeking and the Patient-Physician Relationship: A Systematic Review. J Med Internet Res 2017 Jan 19;19(1):e9 [FREE Full text] [CrossRef] [Medline]
    15. Largent EA, Persad G, Sangenito S, Glickman A, Boyle C, Emanuel EJ. US Public Attitudes Toward COVID-19 Vaccine Mandates. JAMA Netw Open 2020 Dec 01;3(12):e2033324 [FREE Full text] [CrossRef] [Medline]
    16. Kanthawala S, Vermeesch A, Given B, Huh J. Answers to Health Questions: Internet Search Results Versus Online Health Community Responses. J Med Internet Res 2016 Apr 28;18(4):e95 [FREE Full text] [CrossRef] [Medline]
    17. Schachinger K. A Complete Guide to the Google RankBrain Algorithm. Search Engine J 2017 [FREE Full text]
    18. Devlin J, Chang M, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. Preprint posted online May 24, 2019 [FREE Full text]
    19. How Search algorithms work. Google Search.   URL: https://www.google.com/search/howsearchworks/algorithms/ [accessed 2021-04-21]
    20. Nayak P. Understanding searches better than ever before. The Keyword. 2019.   URL: https://blog.google/products/search/search-language-understanding-bert/ [accessed 2021-07-27]
    21. Shen TS, Driscoll DA, Islam W, Bovonratwet P, Haas SB, Su EP. Modern Internet Search Analytics and Total Joint Arthroplasty: What Are Patients Asking and Reading Online? J Arthroplasty 2021 Apr;36(4):1224-1231 [FREE Full text] [CrossRef] [Medline]
    22. Google.   URL: http://google.com [accessed 2021-03-12]
    23. Rothwell J. In Mixed Company: Communicating in Small Groups. Boston, MA: Wadsworth Publishing; 2012.
    24. Starman JS, Gettys FK, Capo JA, Fleischli JE, Norton HJ, Karunakar MA. Quality and content of Internet-based information for ten common orthopaedic sports medicine diagnoses. J Bone Joint Surg Am 2010 Jul 07;92(7):1612-1618. [CrossRef] [Medline]
    25. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. JAMA 1997 Apr 16;277(15):1244-1245. [Medline]
    26. Cassidy JT, Baker JF. Orthopaedic Patient Information on the World Wide Web: An Essential Review. J Bone Joint Surg Am 2016 Feb 17;98(4):325-338. [CrossRef] [Medline]
    27. Kartal A, Kebudi A. Evaluation of the Reliability, Utility, and Quality of Information Used in Total Extraperitoneal Procedure for Inguinal Hernia Repair Videos Shared on WebSurg. Cureus 2019 Sep 04;11(9):e5566 [FREE Full text] [CrossRef] [Medline]
    28. Corcelles R, Daigle CR, Talamas HR, Brethauer SA, Schauer PR. Assessment of the quality of Internet information on sleeve gastrectomy. Surg Obes Relat Dis 2015;11(3):539-544. [CrossRef] [Medline]
    29. Cuan-Baltazar JY, Muñoz-Perez MJ, Robledo-Vega C, Pérez-Zepeda MF, Soto-Vega E. Misinformation of COVID-19 on the Internet: Infodemiology Study. JMIR Public Health Surveill 2020 Apr 09;6(2):e18444 [FREE Full text] [CrossRef] [Medline]
    30. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999 Feb;53(2):105-111 [FREE Full text] [CrossRef] [Medline]
    31. Fan KS, Ghani SA, Machairas N, Lenti L, Fan KH, Richardson D, et al. COVID-19 prevention and treatment information on the internet: a systematic analysis and quality assessment. BMJ Open 2020 Sep 10;10(9):e040487 [FREE Full text] [CrossRef] [Medline]
    32. Haragan AF, Zuwiala CA, Himes KP. Online Information About Periviable Birth: Quality Assessment. JMIR Pediatr Parent 2019 Jun 07;2(1):e12524 [FREE Full text] [CrossRef] [Medline]
    33. Azer SA, AlOlayan TI, AlGhamdi MA, AlSanea MA. Inflammatory bowel disease: An evaluation of health information on the internet. World J Gastroenterol 2017 Mar 07;23(9):1676-1696 [FREE Full text] [CrossRef] [Medline]
    34. Khazaal Y, Chatton A, Cochand S, Coquard O, Fernandez S, Khan R, et al. Brief DISCERN, six questions for the evaluation of evidence-based content of health-related websites. Patient Educ Couns 2009 Oct;77(1):33-37. [CrossRef] [Medline]
    35. Banasiak NC, Meadows-Oliver M. Evaluating asthma websites using the Brief DISCERN instrument. J Asthma Allergy 2017;10:191-196 [FREE Full text] [CrossRef] [Medline]
    36. Zheluk A, Maddock J. Plausibility of Using a Checklist With YouTube to Facilitate the Discovery of Acute Low Back Pain Self-Management Content: Exploratory Study. JMIR Form Res 2020 Nov 20;4(11):e23366 [FREE Full text] [CrossRef] [Medline]
    37. Shekhar R, Sheikh AB, Upadhyay S, Singh M, Kottewar S, Mir H, et al. COVID-19 Vaccine Acceptance among Health Care Workers in the United States. Vaccines (Basel) 2021 Feb 03;9(2):119 [FREE Full text] [CrossRef] [Medline]
    38. Akarsu B, Canbay Özdemir D, Ayhan Baser D, Aksoy H, Fidancı İ, Cankurtaran M. While studies on COVID-19 vaccine is ongoing, the public's thoughts and attitudes to the future COVID-19 vaccine. Int J Clin Pract 2021 Apr;75(4):e13891 [FREE Full text] [CrossRef] [Medline]
    39. Fisher KA, Bloomstone SJ, Walder J, Crawford S, Fouayzi H, Mazor KM. Attitudes Toward a Potential SARS-CoV-2 Vaccine : A Survey of U.S. Adults. Ann Intern Med 2020 Dec 15;173(12):964-973 [FREE Full text] [CrossRef] [Medline]
    40. Verger P, Scronias D, Dauby N, Adedzi KA, Gobert C, Bergeat M, et al. Attitudes of healthcare workers towards COVID-19 vaccination: a survey in France and French-speaking parts of Belgium and Canada, 2020. Euro Surveill 2021 Jan;26(3):2002047 [FREE Full text] [CrossRef] [Medline]
    41. Wang K, Wong EL, Ho K, Cheung AW, Yau PS, Dong D, et al. Change of Willingness to Accept COVID-19 Vaccine and Reasons of Vaccine Hesitancy of Working People at Different Waves of Local Epidemic in Hong Kong, China: Repeated Cross-Sectional Surveys. Vaccines (Basel) 2021 Jan 18;9(1):62 [FREE Full text] [CrossRef] [Medline]
    42. Su Z, McDonnell D, Cheshmehzangi A, Li X, Maestro D, Šegalo S, et al. With Great Hopes Come Great Expectations: A Viewpoint on Access and Adoption Issues Associated with COVID-19 Vaccines. JMIR Public Health Surveill 2021 Feb 01 [FREE Full text] [CrossRef] [Medline]
    43. Funk C, Tyson A. Intent to Get a COVID-19 Vaccine Rises to 60% as Confidence in Research and Development Process Increases. Pew Research Center. 2020 Dec 03.   URL: https:/​/www.​pewresearch.org/​science/​2020/​12/​03/​intent-to-get-a-covid-19-vaccine-rises-to-60-as-confidence-in-research-and-development-process-increases/​ [accessed 2021-02-13]
    44. Schaffer DeRoo S, Pudalov NJ, Fu LY. Planning for a COVID-19 Vaccination Program. JAMA 2020 Jun 23;323(24):2458-2459. [CrossRef] [Medline]
    45. Nahum A, Drekonja D, Alpern J. The Erosion of Public Trust and SARS-CoV-2 Vaccines- More Action Is Needed. Open Forum Infect Dis 2021 Feb;8(2):ofaa657 [FREE Full text] [CrossRef] [Medline]
    46. Carlson A, Riethman M, Gastañaduy P, Lee A, Leung J, Holshue M, et al. Notes from the Field: Community Outbreak of Measles - Clark County, Washington, 2018-2019. MMWR Morb Mortal Wkly Rep 2019 May 17;68(19):446-447 [FREE Full text] [CrossRef] [Medline]
    47. Porter A, Goldfarb J. Measles: A dangerous vaccine-preventable disease returns. Cleve Clin J Med 2019 Jun;86(6):393-398 [FREE Full text] [CrossRef] [Medline]
    48. Pike J, Melnick A, Gastañaduy PA, Kay M, Harbison J, Leidner AJ, et al. Societal Costs of a Measles Outbreak. Pediatrics 2021 Apr;147(4):e2020027037. [CrossRef] [Medline]
    49. Answering Patients’ Questions About COVID-19 Vaccine and Vaccination. Centers for Disease Control and Prevention.   URL: https://www.cdc.gov/vaccines/covid-19/hcp/answering-questions.html [accessed 2021-03-09]
    50. Hill J. People Also Ask Boxes and Related Questions. Hill Web Creations. 2020.   URL: https://www.hillwebcreations.com/people-also-ask-related-questions/ [accessed 2021-02-14]
    51. COCONEL Group. A future vaccination campaign against COVID-19 at risk of vaccine hesitancy and politicisation. Lancet Infect Dis 2020 Jul;20(7):769-770 [FREE Full text] [CrossRef] [Medline]
    52. Schwartz JL. Evaluating and Deploying Covid-19 Vaccines - The Importance of Transparency, Scientific Integrity, and Public Trust. N Engl J Med 2020 Oct 29;383(18):1703-1705. [CrossRef] [Medline]
    53. Trogen B, Oshinsky D, Caplan A. Adverse Consequences of Rushing a SARS-CoV-2 Vaccine: Implications for Public Trust. JAMA 2020 Jun 23;323(24):2460-2461. [CrossRef] [Medline]
    54. Evanega S, Lynas M, Adams J, Smolenyak K. Coronavirus misinformation: quantifying sources and themes in the COVID-19 ‘infodemic’. JMIR Preprints 2020 [FREE Full text] [CrossRef]
    55. Motta M, Stecula D, Farhart C. How Right-Leaning Media Coverage of COVID-19 Facilitated the Spread of Misinformation in the Early Stages of the Pandemic in the U.S. Can J Pol Sci 2020 May 01;53(2):335-342 [FREE Full text] [CrossRef]
    56. Rodriguez A. No, the COVID-19 vaccine doesn’t cause infertility in women. USA Today. 2020.   URL: https:/​/www.​usatoday.com/​story/​news/​health/​2020/​12/​10/​covid-vaccine-debunking-claims-causes-infertility-sterilization/​6497018002/​ [accessed 2021-02-04]
    57. Political polarisation impedes the public policy response to COVID-19. VoxEU & CEPR. 2020.   URL: https://voxeu.org/article/political-polarisation-impedes-public-policy-response-covid-19 [accessed 2021-02-11]
    58. Hart PS, Chinn S, Soroka S. Politicization and Polarization in COVID-19 News Coverage. Science Communication 2020 Aug 25;42(5):679-697 [FREE Full text] [CrossRef]
    59. Greiner B, Ottwell R, Vassar M, Hartwell M. Public Interest in Preventive Measures of Coronavirus Disease 2019 Associated With Timely Issuance of Statewide Stay-at-Home Orders. Disaster Med Public Health Prep 2020 Dec;14(6):765-768 [FREE Full text] [CrossRef] [Medline]
    60. Hartwell M, Greiner B, Kilburn Z, Ottwell R. Association of Public Interest in Preventive Measures and Increased COVID-19 Cases After the Expiration of Stay-at-Home Orders: A Cross-Sectional Study. Disaster Med Public Health Prep 2020 Sep 10:1-5 [FREE Full text] [CrossRef] [Medline]
    61. Mills MC, Sivelä J. Should spreading anti-vaccine misinformation be criminalised? BMJ 2021 Feb 17;372:n272. [CrossRef] [Medline]
    62. Foresman B. Google, Ohio State join to fight COVID-19 vaccine misinformation. EDSCOOP. 2021.   URL: https://edscoop.com/google-ohio-state-join-to-fight-covid-19-vaccine-misinformation/ [accessed 2021-02-20]
    63. Higgins J, Thomas J, Chandler J. Cochrane Handbook for Systematic Reviews of Interventions. Hoboken, NJ: John Wiley & Sons; 2019.   URL: https://play.google.com/store/books/details?id=cTqyDwAAQBAJ [accessed 2020-05-03]
    64. Mbuagbaw L, Lawson DO, Puljak L, Allison DB, Thabane L. A tutorial on methodological studies: the what, when, how and why. BMC Med Res Methodol 2020 Sep 07;20(1):226 [FREE Full text] [CrossRef] [Medline]

    ‎
    FAQ: frequently asked question
    JAMA: Journal of the American Medical Association

    Edited by C Baur, T Mackey; submitted 12.03.21; peer-reviewed by W Zhang, M Das; comments to author 12.04.21; revised version received 15.04.21; accepted 20.06.21; published 04.08.21

    Copyright

    ©Nicholas B Sajjadi, Samuel Shepard, Ryan Ottwell, Kelly Murray, Justin Chronister, Micah Hartwell, Matt Vassar. Originally published in JMIR Infodemiology (https://infodemiology.jmir.org), 04.08.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Infodemiology, is properly cited. The complete bibliographic information, a link to the original publication on https://infodemiology.jmir.org/, as well as this copyright and license information must be included.