This paper is in the following e-collection/theme issue:

Misinformation and Disinformation Outbreaks and Information Prevalence Studies (44) Medicine 2.0: Social Media, Open, Participatory, Collaborative Medicine (1641) Social Media in Public Health informatics (32) Infectious Diseases (non-STD/STI) (1575) Respiratory Medicine (83) Drug Repurposing and Off-Label Use (25) Theme Issue: Social Media, Ethics, and COVID-19 Misinformation (36)

Published on in Vol 4 (2024)

This is a member publication of Michigan State University

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/56675, first published .
Association Between X/Twitter and Prescribing Behavior During the COVID-19 Pandemic: Retrospective Ecological Study

Association Between X/Twitter and Prescribing Behavior During the COVID-19 Pandemic: Retrospective Ecological Study

Association Between X/Twitter and Prescribing Behavior During the COVID-19 Pandemic: Retrospective Ecological Study

Authors of this article:

Scott A Helgeson1 Author Orcid Image ;   Rohan M Mudgalkar2 Author Orcid Image ;   Keith A Jacobs2 Author Orcid Image ;   Augustine S Lee1 Author Orcid Image ;   Devang Sanghavi3 Author Orcid Image ;   Pablo Moreno Franco4 Author Orcid Image ;   Ian S Brooks2 Author Orcid Image ;   National COVID Cohort Collaborative (N3C)5
Article Authors Cited by Tweetations Metrics

    Original Paper

    1Department of Pulmonary and Critical Care Medicine, Mayo Clinic, Jacksonville, FL, United States

    2School of Information Sciences, Center for Health Informatics, University of Illinois at Urbana-Champaign, Ubana-Champaign, IL, United States

    3Department of Critical Care Medicine, Mayo Clinic, Jacksonville, FL, United States

    4Department of Transplant Medicine, Mayo Clinic, Jacksonville, FL, United States

    5see Acknowledgments

    Corresponding Author:

    Scott A Helgeson, MS, MD

    Department of Pulmonary and Critical Care Medicine

    Mayo Clinic

    4500 San Pablo Road South

    Jacksonville, FL, 32224

    United States

    Phone: 1 9049532000

    Email: helgeson.scott@mayo.edu


    Background: Social media has become a vital tool for health care providers to quickly share information. However, its lack of content curation and expertise poses risks of misinformation and premature dissemination of unvalidated data, potentially leading to widespread harmful effects due to the rapid and large-scale spread of incorrect information.

    Objective: We aim to determine whether social media had an undue association with the prescribing behavior of hydroxychloroquine, using the COVID-19 pandemic as the setting.

    Methods: In this retrospective study, we gathered the use of hydroxychloroquine in 48 hospitals in the United States between January and December 2020. Social media data from X/Twitter was collected using Brandwatch, a commercial aggregator with access to X/Twitter’s data, and focused on mentions of “hydroxychloroquine” and “Plaquenil.” Tweets were categorized by sentiment (positive, negative, or neutral) using Brandwatch’s sentiment analysis tool, with results classified by date. Hydroxychloroquine prescription data from the National COVID Cohort Collaborative for 2020 was used. Granger causality and linear regression models were used to examine relationships between X/Twitter mentions and prescription trends, using optimum time lags determined via vector auto-regression.

    Results: A total of 581,748 patients with confirmed COVID-19 were identified. The median daily number of positive COVID-19 cases was 1318.5 (IQR 1005.75-1940.3). Before the first confirmed COVID-19 case, hydroxychloroquine was prescribed at a median rate of 559 (IQR 339.25-728.25) new prescriptions per day. A day-of-the-week effect was noted in both prescriptions and case counts. During the pandemic in 2020, hydroxychloroquine prescriptions increased significantly, with a median of 685.5 (IQR 459.75-897.25) per day, representing a 22.6% rise from baseline. The peak occurred on April 2, 2020, with 3411 prescriptions, a 397.6% increase. Hydroxychloroquine mentions on X/Twitter peaked at 254,770 per day on April 5, 2020, compared to a baseline of 9124 mentions per day before January 21, 2020. During this study’s period, 3,823,595 total tweets were recorded, with 10.09% (n=386,115) positive, 37.87% (n=1,448,030) negative, and 52.03% (n=1,989,450) neutral sentiments. A 1-day lag was identified as the optimal time for causal association between tweets and hydroxychloroquine prescriptions. Univariate analysis showed significant associations across all sentiment types, with the largest impact from positive tweets. Multivariate analysis revealed only neutral and negative tweets significantly affected next-day prescription rates.

    Conclusions: During the first year of the COVID-19 pandemic, there was a significant association between X/Twitter mentions and the number of prescriptions of hydroxychloroquine. This study showed that X/Twitter has an association with the prescribing behavior of hydroxychloroquine. Clinicians need to be vigilant about their potential unconscious exposure to social media as a source of medical knowledge, and health systems and organizations need to be more diligent in identifying expertise, source, and quality of evidence when shared on social media platforms.

    JMIR Infodemiology 2024;4:e56675

    doi:10.2196/56675

    Keywords

    social media; infodemic; COVID-19; healthcare utilization; misinformation; disinformation; Twitter; hydroxychloroquine; X; drugs; pharmacy; pharmacology; pharmacotherapy; pharmaceuticals; medication; prescription; sentiment; SARS-CoV-2; pandemic; respiratory; infectious 


    As of September 1, 2024, SARS-CoV-2 has led to 111,820,082 infections and 1,201,061 deaths reported in the United States alone [1]. When the first nontravel case of COVID-19 was reported in the United States on January 20, 2020, clinicians had no significant knowledge of the virus nor the management of the disease resulting from the virus and looked to colleagues from other countries for their experiences [2,3].

    Although peer-reviewed observational reports were being published rapidly in record amounts, and clinical trials were ongoing, clinicians also used social media to share and gain knowledge on the care of their patients in hopes of improving their patients’ outcomes [4-6]. Even though social media is increasingly used by health care providers and health care systems, its accuracy and role as a medium for sharing critical information has the potential to be corrupted, leading to the use of unproven, potentially harmful, and costly therapies [7].

    Multiple studies have shown misinformation as shown by false tweets was widely available and were even retweeted faster [8-11]. Not only were there concerns about true and false information spreading, but there was regional sentiment variation on X/Twitter throughout the United States [12]. This misinformation was not only spread by nonmedical people but also by physicians from a range of specialties [13]. To combat this infodemic, the World Health Organization established a repository of COVID-19 fact-checking groups that verify COVID-19 claims [14].

    We hypothesized that the circumstances of the COVID-19 pandemic (ie, lack of evidence-based medicine, overly stressed health care system, and health care providers) may have magnified the potential for social media to have influenced the care of patients affected by COVID-19. To explore this hypothesis, we aimed to see if we could identify a temporal association between the X/Twitter mentions of hydroxychloroquine and the “sentiment” of these X/Twitter mentions, against the daily use of the medication.


    Social Media Data

    All social media data was gathered before the rebranding of Twitter to X, so throughout this paper X will be referred to as X/Twitter and posts will be referred to as tweets. The commercial social media aggregator, Brandwatch was leveraged to gather X/Twitter data and perform sentiment analysis [15,16]. Brandwatch is an official partner of X/Twitter allowing better access to X/Twitter data [17]. The accuracy of Brandwatch’s sentiment analysis is around 75%, which is comparable to other sentiment analysis tools [18-20]. In the tool, a query for the drug “hydroxychloroquine” and “Plaquenil” was added, and the date range was set from November 30, 2019, to January 29, 2021. Unlike other tools that rely on a dictionary of “good” and “bad” words for performing sentiment analysis, Brandwatch’s sentiment analysis models are based on a library of hundreds of rules created based on natural language processing and are updated and audited regularly [16]. Brandwatch classifies tweets as positive or negative only if it is confident about the sentiment classification. If the sentiment cannot be accurately determined, such tweets are categorized as neutral. Tweets were reported as neutral, nonneutral, total, positive, and negative sentiment. Nonneutral tweets combined positive and negative tweets, while total tweets summed up neutral and nonneutral tweets. The tweets were grouped by date, and a CSV file was generated, which included a count of each category of tweets for each day.

    COVID-19 and Medication Data

    Medication data was obtained from the National COVID Cohort Collaborative (N3C), a national collection of 48 hospitals or data partners with 4.8 million patients [21]. The N3C cohort is comprised of patients diagnosed with COVID-19 by polymerase chain reaction (PCR) and a control group of patients without COVID-19 matched by age, sex, and race at a 2:1 ratio. For this study, the entire cohort from 2020 was used to capture new hydroxychloroquine prescriptions per day either in the inpatient or in the outpatient setting. Restriction to COVID-19 positivity or negativity was not carried out because the total usage of this medication was being assessed and not specifically the treatment of patients who were tested positive for COVID-19, as patients were treated with hydroxychloroquine before their COVID-19 status resulted, and some of these patients were likely to be negative. Data that could identify patients were not included in the data access, such as age, race, and gender, per N3C. New COVID-19 positivity numbers by PCR were collected with the start date being February 1, 2020, as this was the first date in the N3C database of positive tests. The “drug exposure” dataset of the N3C database was specifically queried for hydroxychloroquine, hydroxychloroquine sulfate, Quineprox, and Plaquenil. There were no exclusion criteria.

    Timeline Data

    Prepandemic basal rates of hydroxychloroquine prescriptions and tweets were defined as the period prior from January 1, 2020, to the first reported case of COVID-19 in the United States on January 20, 2020. Pandemic rates were determined using the timeframe between January 21, 2020, to December 31, 2020.

    The first peer-reviewed publications on hydroxychloroquine for COVID-19, and key sentinel media or public events or announcements were identified, including announcements from the FDA or other government and nongovernment authorities. A timeline illustrating the X/Twitter hits (including the sentiment) was overlayed to highlight impacts with key media events or announcements, and key peer-reviewed publications evaluating the efficacy of hydroxychloroquine on COVID-19.

    Statistical Analysis

    Median and IQRs were reported for descriptive statistics. COVID-19 PCR positivity rates were used to determine the relative use (percent) of hydroxychloroquine prescriptions per patient who tested positive for COVID-19. Estimation of the annual number of hydroxychloroquine prescriptions without the pandemic was carried out by using the median over the prepandemic period and multiplying by 365 (days) to come up with a yearly total. This was used to determine the difference in the number of prescriptions for unnecessary prescriptions. A 10-day prescription and the National Average Drug Acquisition Cost in 2021 were used to calculate the cost of a prescription [22,23].

    Statistical analysis was performed using a Granger causality test to determine whether a temporal association exists between X/Twitter mentions and hydroxychloroquine prescriptions [24]. For analysis, the tweets were classified into 3 sentiment categories: neutral, nonneutral, and total as Brandwatch sentiment analysis allows for the most accuracy with this approach.

    To obtain the optimum time lag length for Granger causality tests, a vector auto-regressive (VAR) model was fitted to the dataset. The VAR model was run for neutral, nonneutral, and total tweets, and the optimum time lag was noted for each test. Using the optimum time lags calculated from the VAR model, the Granger causality test was then used to determine if a causal relationship exists between X/Twitter mentions of hydroxychloroquine and its prescriptions. The chi-square metric was used to verify the significance of the time lag.

    As the Granger causality test does not specify the direction of the temporal association, the optimum lag day was used in a linear regression model to determine whether the number of prescriptions of hydroxychloroquine could be predicted by the number of tweets mentioning hydroxychloroquine in the preceding days (offset by the optimum lag day determined by the above VAR analysis). Similar analyses using a multivariate linear regression model, but using the tweet sentiments as covariates (positive, neutral, and negative sentiment tweets) were carried out to identify their independent impact on predicting the number of new hydroxychloroquine prescriptions. A P value of less than .05 was considered significant. Granger analysis was performed using Python (version 3.9.7; Python Software Foundation) using the “stats models” package and the linear regression models were performed in BlueSky (version 10.3; BlueSky Statistics).

    Ethical Considerations

    No identifiers were used in the collection or analysis of the data, and this study was considered exempt by the Mayo Clinic institutional review board (#21-002787). This study was approved to use a limited dataset (access level 3) by the N3C Data Use Request Committee. All data were deidentified and a waiver of informed consent was approved consistent with social media postings.


    COVID-19 and Hydroxychloroquine Prescriptions

    A total of 581,748 cases of confirmed patients with COVID-19 were identified. The median number of daily positive case counts of patients with COVID-19 was 1318.5 (IQR 1005.75-1940.3) as there was documented reinfection during this study’s period (Figure 1). At a baseline, the median daily use of hydroxychloroquine before the first confirmed case of COVID-19 on January 20, 2020, was 559 (IQR 339.25-728.25) new prescriptions per day (Figure 2). There appears to be a day-of-the-week effect resulting in oscillations of prescriptions and COVID-19 diagnosis. As the pandemic progressed, there was a significant increase in the use of hydroxychloroquine. During the pandemic in 2020, hydroxychloroquine was prescribed a median of 685.5 (IQR 459.75-897.25) times per day, which was a 22.6% increase above the baseline rate. The peak number of prescriptions during a single day was 3411, which occurred on April 2, 2020, which was a 397.6% increase in prescriptions during that day.

    ‎
    Figure 1. Using the N3C database, the daily census data of new patients with COVID-19 diagnosed through PCR testing was displayed from February 1, 2020, to December 31, 2020. N3C: National COVID Cohort Collaborative; PCR: polymerase chain reaction.
    ‎
    Figure 2. Using the N3C database, daily new hydroxychloroquine prescriptions for any reason were displayed from February 1, 2020, to December 31, 2020. N3C: National COVID Cohort Collaborative.

    X/Twitter Data

    Hydroxychloroquine mentions on X/Twitter did not increase significantly until March 2020. X/Twitter mentions and retweets on hydroxychloroquine at baseline were 9124 per day before January 21, 2020. It increased from the pre-COVID baseline and peaked at 254,770 tweets per day on April 5, 2020 (Figure 3 [3,25-31]). There were notable associations between several key publications, public events, or announcements, and a rise in X/Twitter mentions and retweets (Figure 3 [3,25-31]). X/Twitter data were separated into total, neutral, positive, and negative sentiments (Figure 4). For the investigational period, the total tweets mentioning hydroxychloroquine were 3,823,595, with 10.09% (n=386,115) positive, 37.87% (n=1,448,030) negative, and 52.03% (n=1,989,450) neutral sentiments.

    ‎
    Figure 3. The total X/Twitter mentions of hydroxychloroquine, obtained by Brandwatch, in a natural log scale was displayed from January 1, 2020, to December 31, 2020, and was marked with some key publications and media moments noted by numbers. (1) The first paper was published describing the use of chloroquine for COVID-19 from expert opinion on March 12, 2020 [25]. (2) On March 17, 2020, an open-label nonrandomized study was published showing success with the use of hydroxychloroquine in patients with COVID-19 [26]. (3) On April 5, 2020, the president of the United States promoted the use of hydroxychloroquine for COVID-19 [27]. (4) Then the Center for Disease Control deleted a report of physicians using hydroxychloroquine [3]. (5) The Food and Drug Administration on April 24, 2020, announced to use hydroxychloroquine with caution outside of the hospital or clinical trial secondary to arrhythmias [28]. (6) On May 18, 2020, the president of the United States reported tweets that he is taking hydroxychloroquine to protect himself from COVID-19 [29]. (7) The World Health Organization stopped a trial with hydroxychloroquine over safety concerns [30]. (8) On June 15, 2020, the Food and Drug Administration revoked the Emergency Use Authorization for chloroquine and hydroxychloroquine on COVID-19 [31].
    ‎
    Figure 4. Using Brandwatch, the volume of tweets by total amount and sentiment category (neutral and nonneutral divided out by positive and negative sentiment) was displayed from throughout the year 2020.

    Optimal Lag-Day and Temporal Causality Analysis

    The optimal lag-day of 1 day was identified using the VAR model, indicating that the tweets mentioning hydroxychloroquine were most strongly associated with hydroxychloroquine prescriptions within 1 day (Table 1). Although Granger analysis identified significant causal relationships between tweets and prescriptions of hydroxychloroquine across multiple lag days, it confirmed that the strongest statistical relationship occurred with a 1-day time lag identified (total tweets, P=.005; neutral tweets, P=.001; and nonneutral tweets, P=.02; Table S1 in Multimedia Appendix 1).

    Table 1. Univariate and multivariate analyses for the direction of relationships between tweets and the number of hydroxychloroquine prescriptions with a 1-day time lag through the 2020 year. Tweets were separated by sentiment and total number.
    Sentimentβ valueP value
    Univariate analysis

    		Positive.0641<.001

    		Negative.0155<.001

    		Neutral.0169<.001

    		Total.0078<.001
    Multivariate analysis

    		Positive.0029.92

    		Negative–.0149.04

    		Neutral.0271<.001

    Directional Analysis

    The optimal lag day of 1 day demonstrated that an increase in tweets increased hydroxychloroquine prescriptions the following day (Table 1). The biggest impact was with positive tweets, which showed that for every 100 positive sentiment tweets, there would be 6 new hydroxychloroquine prescriptions the next day. Negative tweets, as well as neutral tweets, appeared to be associated with an increase in hydroxychloroquine prescriptions the next day.

    However, in multivariate analysis with covariates of positive, negative, and neutral tweets, only neutral and negative tweets seemed to affect hydroxychloroquine prescriptions the next day. On average, the impact of 1000 positive tweets would result in 27.1 new prescriptions the following day, while 1000 negative sentiment tweets would result in 14.9 fewer hydroxychloroquine prescriptions the following day.

    Potential Harm and Costs

    In 1 meta-analysis of 9 phase 3 randomized controlled clinical trials, the rate of adverse events with hydroxychloroquine was 12%, with a computed number needed to harm of 9 [32]. The total number of hydroxychloroquine prescriptions over the basal prepandemic period was 68,893. This would translate to 7654.8 patients potentially risking harm if all the hydroxychloroquine prescriptions were for COVID-19. Additionally, hydroxychloroquine National Average Drug Acquisition Costs were estimated at US $3.43 for each new prescription, which may have contributed to an excess of US $236,473.85 in unnecessary costs during this study’s period.


    The main findings of our investigation provide temporal evidence that prescribing practices of health care providers were associated with X/Twitter tweets. Specifically, the number of hydroxychloroquine-related tweets was temporally associated with new hydroxychloroquine prescriptions the following day. Supporting the temporal association, multivariate analysis showed that negative sentiment tweets decreased the subsequent prescribing of hydroxychloroquine. Recognizing that hydroxychloroquine has been shown to be ineffective for COVID-19 infection, there may have been 68,893 unnecessary hydroxychloroquine prescriptions resulting in avoidable harm, and US $236,473.85 in excess costs from this study’s sample.

    Hydroxychloroquine is an antimalarial medication most often used in the United States to treat rheumatologic disorders. Early reports in the COVID-19 pandemic suggested a potential benefit of hydroxychloroquine in treating COVID-19 [26,32-34]. There was biological plausibility to these studies, with prior in vitro studies showing the activity of hydroxychloroquine against SARS-CoV-1 (severe acute respiratory syndrome coronavirus) and MERS-CoV (Middle East respiratory syndrome) [35,36]. None of the COVID-19 studies reported that considered hydroxychloroquine showed a strong enough level of evidence to meet the criteria for use in patients. The deviation from traditional drug approval pathways and its dissemination to the bedside, which normally takes years, because of the need for effective treatments accelerated this process and arguably led to shortcuts in the process [37]. This scenario led to the backdrop for our study.

    The internet era has undoubtedly contributed to improved access and sharing of knowledge and information with physicians and patients [38]. Social media platforms promised to facilitate and strengthen relationships between diverse people and opinions worldwide, by enhancing and supporting collaborations and improving knowledge sharing in the professional world [39,40]. The pace at which information can be shared and disseminated by social media is staggering, considering that previously a confirmed best practice may take more than a decade before its widespread implementation in clinical practice. False news spreads have been reported to spread about 6 times faster and to more than 10 times the people than accurate news when evaluating X/Twitter data [41]. However, much like modern-day centralized electronic health systems, a small error could be magnified and applied systematically across a population of patients unknowingly [42,43]. In the case of social media, there is misinformation (incorrect information) and disinformation (deliberately inaccurate information) on the care of patients that could be magnified, particularly under stressed circumstances, such as a devastating pandemic. X/Twitter responded to this concern by flagging potentially concerning information and removing 11,230 accounts because of misinformation [44]. In the specific case of clinical care, there is concern about circumventing the scientific peer-reviewed process to test the safety of interventions and their risks rigorously. Social media may prematurely propagate preliminary unvalidated therapies such as hydroxychloroquine for COVID-19. Examples of clinical findings conveyed to the media before peer review are suspect and should generally be avoided, with recent examples of disinformation through social media being high doses of vitamin C, thiamine, and hydrocortisone in patients who develop sepsis [45].

    COVID-19 caused an infodemic with huge releases of medical literature through peer-reviewed and non–peer-reviewed sources with more than 1000 publications monthly [4]. Along with these publications, technology and social media played a massive role in information dissemination, with millions of tweets weekly about COVID-19 [46]. This sheer volume of data could negatively impact patient care, given that a significant amount of all health- and COVID-19–related tweets contained false information [7]. The World Health Organization raised concerns that this infodemic adversely affected global health related to COVID-19 [47]. Poor-quality health care data have historically been the problem paving the way for a misinformation infodemic [48]. Other prior epidemics (Ebola and Zika) suggested the same infodemic concerns with misinformation spread by social media [49-56]. Outside of epidemics and pandemics, there has also been consistent misinformation about vaccinations portrayed on social media platforms [57-62]. With the COVID-19 pandemic, qualitative studies have shown significant misinformation (up to 70% of information was false or lacked evidence) on social media platforms [7,9,63-67]. One promising study on X/Twitter data showed that false information was tweeted more than science-based information, but science-based tweets were retweeted more [68]. Our study, regarding specifically hydroxychloroquine use, showed that X/Twitter was associated with the prescribing practice during the COVID-19 pandemic, which raises significant concern about a misinformation infodemic.

    Some potential fixes to this problem consist of education for physicians or clinicians and appropriate tagging of X/Twitter data. X/Twitter tried to combat misinformation with the implementation of “verified” status, which means that the account is authentic [69]. This verification does not mean that someone is an expert in the field or disease for which they are tweeting. On top of that, the verification system has stopped. Around 39,000 physicians were active on X/Twitter during the start of the pandemic, but it is not known how many provided accurate information. The appropriate use of X/Twitter for health care information or education has not been standardized. There are no educational programs about the appropriate use of X/Twitter and a physician’s responsibility for accurate and verified information. Many of the X/Twitter best practices are geared toward physicians on how to set up their accounts, become noticed, and create connections [70,71]. The development of physician best practices on X/Twitter needs to occur with a focus on disseminating information that is accurate and that nonmedical people will understand. This also creates a need for medical journals to enhance their presence on social media platforms so peer-reviewed studies take the forefront and people can gain access to these studies earlier than for printed copies.

    Our findings are compelling but have limitations that should be considered in their interpretation. First, this is an ecological study of multiple databases and their temporal trends, but they are not tied to individual prescribers or patients. For example, we do not know for sure that the use of hydroxychloroquine was prescribed specifically to patients who had COVID-19 and we cannot track whether these prescribing health care professionals used the social media platform. We also do not know if the patients who were prescribed hydroxychloroquine were diagnosed with COVID-19, only had symptoms, or were presumed positive. Additionally, although our findings support that social media did affect COVID-19 prescribing habits, this study did not look at physician versus nonphysician handles. The infodemic could have enhanced the public perception of possible therapies, leading to questioning clinicians taking care of their loved ones, as to why such treatments are being withheld, particularly under the pressure of a patient who is rapidly deteriorating or already on life support [72]. Using X/Twitter as the sample platform to gauge whether social media influenced clinician behavior is faulty, and we acknowledge that social media represents multiple ever-changing platforms and media. However, it is one of the most used social media platforms, with over 396 million users worldwide. The sheer number of hits supported that X/Twitter reasonably represents overall social media sentiments. Sentiment analysis can be difficult because distinguishing between someone being happy that hydroxychloroquine was being used and someone being happy that it was stopped cannot be performed. Brandwatch uses a set of proprietary rules to classify and score tweets into positive, negative, and neutral sentiments. As such, there is no disclosure as to how high these rules classify tweets and we were forced to use a “black box” algorithm, albeit a standard throughout this area of study. Lastly, X/Twitter is not the only source of information and is not the only means of creating undue influence on prescribing habits. There are likely additional factors that cannot be controlled for in this study that impacted prescribing patterns. Additionally, The World Health Organization, Center for Disease Control, hospitals, and medical societies, among others, could have actually mediated the association found in this study.

    Similarly, our sample of acute care hospitals in the United States may not precisely reflect the behavior of the rest of the country, as participation and providing data were voluntary, but this is the largest cohort of hospitals to evaluate for prescribing practices [21]. There may be regional differences between academic and nonacademic practices and between types of prescribers, which could not be assessed. We must follow up our findings with more systematic data as they become available (such as from multiple payers and the Center for Medicare Services) and numerous social media platforms. Despite these limitations, our results are compelling, which clinicians, payers, administrators, and policy makers must know as additional supportive data arise.

    In conclusion, X/Twitter information about hydroxychloroquine was significantly associated with the prescribing habits of clinicians within 1 day of the tweet. More granular data are necessary to evaluate how specific prescriber details and location affect prescribing practices. The impact of charging for verified X/Twitter accounts may affect future misinformation infodemics, but the primary responsibility for misinformation should be on the person spreading the information and not on the platform or method of dissemination. This study shows a strong analytical case for the dangers of social media and the inappropriate attention and influence on prescribers. Further study is necessary on how to prevent or reduce infodemics in the future as this will likely keep occurring if an intervention is not performed.

    Acknowledgments

    The authors would also like to thank Michael Wieck-Sosa, BS, and Yiren Liu, BS, for their help in with this project. The analyses described in this publication were conducted with data or tools accessed through the National Center for Advancing Translational Sciences (NCATS) N3C Data Enclave [21] and N3C Attribution & Publication Policy (version 1.2-2020-08-25b; grant NCATS U24 TR002306). This research was possible because of the patients whose information is included in the data and the organizations [73] and scientists who have contributed to the ongoing development of this community resource [74]. We gratefully acknowledge the following core contributors to N3C: Adam B Wilcox, Adam M Lee, Alexis Graves, Alfred (Jerrod) Anzalone, Amin Manna, Amit Saha, Amy Olex, Andrea Zhou, Andrew E Williams, Andrew Southerland, Andrew T Girvin, Anita Walden, Anjali A Sharathkumar, Benjamin Amor, Benjamin Bates, Brian Hendricks, Brijesh Patel, Caleb Alexander, Carolyn Bramante, Cavin Ward-Caviness, Charisse Madlock-Brown, Christine Suver, Christopher Chute, Christopher Dillon, Chunlei Wu, Clare Schmitt, Cliff Takemoto, Dan Housman, Davera Gabriel, David A Eichmann, Diego Mazzotti, Don Brown, Eilis Boudreau, Elaine Hill, Elizabeth Zampino, Emily Carlson Marti, Emily R Pfaff, Evan French, Farrukh M Koraishy, Federico Mariona, Fred Prior, George Sokos, Greg Martin, Harold Lehmann, Heidi Spratt, Hemalkumar Mehta, Hongfang Liu, Hythem Sidky, JW Awori Hayanga, Jami Pincavitch, Jaylyn Clark, Jeremy Richard Harper, Jessica Islam, Jin Ge, Joel Gagnier, Joel H Saltz, Joel Saltz, Johanna Loomba, John Buse, Jomol Mathew, Joni L Rutter, Julie A McMurry, Justin Guinney, Justin Starren, Karen Crowley, Katie Rebecca Bradwell, Kellie M Walters, Ken Wilkins, Kenneth R Gersing, Kenrick Dwain Cato, Kimberly Murray, Kristin Kostka, Lavance Northington, Lee Allan Pyles, Leonie Misquitta, Lesley Cottrell, Lili Portilla, Mariam Deacy, Mark M Bissell, Marshall Clark, Mary Emmett, Mary Morrison Saltz, Matvey B Palchuk, Melissa A Haendel, Meredith Adams, Meredith Temple-O'Connor, Michael G Kurilla, Michele Morris, Nabeel Qureshi, Nasia Safdar, Nicole Garbarini, Noha Sharafeldin, Ofer Sadan, Patricia A Francis, Penny Wung Burgoon, Peter Robinson, Philip RO Payne, Rafael Fuentes, Randeep Jawa, Rebecca Erwin-Cohen, Rena Patel, Richard A Moffitt, Richard L Zhu, Rishi Kamaleswaran, Robert Hurley, Robert T Miller, Saiju Pyarajan, Sam G Michael, Samuel Bozzette, Sandeep Mallipattu, Satyanarayana Vedula, Scott Chapman, Shawn T O'Neil, Soko Setoguchi, Stephanie S Hong, Steve Johnson, Tellen D Bennett, Tiffany Callahan, Umit Topaloglu, Usman Sheikh, Valery Gordon, Vignesh Subbian, Warren A Kibbe, Wenndy Hernandez, Will Beasley, Will Cooper, William Hillegass, and Xiaohan Tanner Zhang. The details of contributions are available online [75].

    Data Availability

    The prescription and COVID-19 datasets generated and analyzed during this study are not publicly available due to the National COVID Cohort Collaborative data sharing rules, but the X/Twitter data are available from the corresponding author on reasonable request.

    Authors' Contributions

    SAH, ASL, ISB, DS, and PMF were responsible for study conception and design. SAH, RMM, KAJ, and ISB conducted data acquisition. All authors were responsible for data analysis and interpretation of results. All authors discussed the results and wrote this paper. IAS is the Associate Editor for JMIR Infodemiology.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Tables on tweet sentiment and tweets per day.

    DOCX File , 57 KB
    1. Centers for Disease Control and Prevention. COVID Data Tracker. Atlanta, GA. US Department of Health and Human Services, CDC URL: https://covid.cdc.gov/covid-data-tracker [accessed 2024-10-10]
    2. CDC COVID-19 Response Team, Jorden MA, Rudman SL, Villarino E, Hoferka S, Patel MT, et al. Evidence for limited early spread of COVID-19 within the United States, January-February 2020. MMWR Morb Mortal Wkly Rep. 2020;69(22):680-684. [FREE Full text] [CrossRef] [Medline]
    3. Centers for Disease Control and Prevention. CDC museum COVID-19 timeline. URL: https://www.cdc.gov/museum/timeline/covid19.html [accessed 2024-10-10]
    4. Chen Q, Allot A, Lu Z. Keep up with the latest coronavirus research. Nature. 2020;579(7798):193. [CrossRef] [Medline]
    5. Ghosh P, Schwartz G, Narouze S. Twitter as a powerful tool for communication between pain physicians during COVID-19 pandemic. Reg Anesth Pain Med. 2021;46(2):187-188. [CrossRef] [Medline]
    6. Margus C, Brown N, Hertelendy AJ, Safferman MR, Hart A, Ciottone GR. Emergency physician Twitter use in the COVID-19 pandemic as a potential predictor of impending surge: retrospective observational study. J Med Internet Res. 2021;23(7):e28615. [FREE Full text] [CrossRef] [Medline]
    7. Swetland SB, Rothrock AN, Andris H, Davis B, Nguyen L, Davis P, et al. Accuracy of health-related information regarding COVID-19 on Twitter during a global pandemic. World Med Health Policy. 2021;13(3):503-517. [FREE Full text] [CrossRef] [Medline]
    8. Cinelli M, Quattrociocchi W, Galeazzi A, Valensise CM, Brugnoli E, Schmidt AL, et al. The COVID-19 social media infodemic. Sci Rep. 2020;10(1):16598. [FREE Full text] [CrossRef] [Medline]
    9. Kouzy R, Abi Jaoude J, Kraitem A, El Alam MB, Karam B, Adib E, et al. Coronavirus goes viral: quantifying the COVID-19 misinformation epidemic on Twitter. Cureus. 2020;12(3):e7255. [FREE Full text] [CrossRef] [Medline]
    10. Shahi GK, Dirkson A, Majchrzak TA. An exploratory study of COVID-19 misinformation on Twitter. Online Soc Netw Media. 2021;22:100104. [FREE Full text] [CrossRef] [Medline]
    11. Tasnim S, Hossain MM, Mazumder H. Impact of rumors and misinformation on COVID-19 in social media. J Prev Med Public Health. 2020;53(3):171-174. [FREE Full text] [CrossRef] [Medline]
    12. Guntuku SC, Buttenheim AM, Sherman G, Merchant RM. Twitter discourse reveals geographical and temporal variation in concerns about COVID-19 vaccines in the United States. Vaccine. 2021;39(30):4034-4038. [FREE Full text] [CrossRef] [Medline]
    13. Sule S, DaCosta MC, DeCou E, Gilson C, Wallace K, Goff SL. Communication of COVID-19 misinformation on social media by physicians in the US. JAMA Netw Open. 2023;6(8):e2328928. [FREE Full text] [CrossRef] [Medline]
    14. COVID-19 misinformation portal? A rapid response project from the social media lab. 2021. URL: https://covid19misinfo.org/ [accessed 2024-10-10]
    15. Brandwatch. 2022. URL: https://www.brandwatch.com/ [accessed 2024-10-10]
    16. Sentiment analysis. Brandwatch. 2022. URL: https://www.brandwatch.com/wp-content/uploads/brandwatch/Sentiment-Analysis.pdf [accessed 2024-10-10]
    17. Palmer G. News: Brandwatch and Twitter expanded strategic partnership. 2017. URL: https://www.brandwatch.com/blog/brandwatch-and-twitter-expand-strategic-partnership/ [accessed 2024-10-10]
    18. AminiMotlagh M, Shahhoseini H, Fatehi N. A reliable sentiment analysis for classification of tweets in social networks. Soc Netw Anal Min. 2023;13(1):7. [FREE Full text] [CrossRef] [Medline]
    19. Gonzalez JC. Accuracy measures in sentiment analysis: the precision of Meaningcloud's technology. 2021. URL: https://www.meaningcloud.com/blog/accuracy-in-sentiment-analysis [accessed 2024-10-10]
    20. Taylor N. Interview: the data science behind Brandwatch's new sentiment analysis. 2022. URL: https://www.brandwatch.com/blog/data-science-behind-brandwatchs-new-sentiment-analysis [accessed 2024-10-10]
    21. National Institute of Health (NIH) and National Center for Advancing Translational Sciences (NCATS). National COVID Cohort Collaborative Data Enclave Respository. Bethesda, MA. US Department of Health and Human Services, National Institutes of Health URL: https://covid.cd2h.org/ [accessed 2024-10-10]
    22. Self WH, Semler MW, Leither LM, Casey JD, Angus DC, Brower RG, National Heart‚ Lung‚Blood Institute PETAL Clinical Trials Network, et al. Effect of hydroxychloroquine on clinical status at 14 days in hospitalized patients With COVID-19: a randomized clinical trial. JAMA. 2020;324(21):2165-2176. [FREE Full text] [CrossRef] [Medline]
    23. Centers for Medicare and Medicaid Services. NADAC (National Average Drug Acquisition Cost). 2021. URL: https:/​/data.​medicaid.gov/​dataset/​d5eaf378-dcef-5779-83de-acdd8347d68e/​data?conditions[0][property]=ndc_description&conditions[0][value]=hydroxychloroquine&conditions[0][operator]=contains [accessed 2024-10-10]
    24. Granger CWJ. Investigating causal relations by econometric models and cross-spectral methods. Econometrica. 1969;37(3):424-438. [CrossRef]
    25. Chen Z, Hu J, Zhang Z, Jiang S, Han S, Yan D. Efficacy of hydroxychloroquine in patients with COVID-19: results of a randomized clinical trial. medRxiv. Preprint posted online on 2020. 2004. URL: https://www.medrxiv.org/content/10.1101/2020.03.22.20040758v3 [accessed 2024-10-10]
    26. Gautret P, Lagier J, Parola P, Hoang VT, Meddeb L, Mailhe M, et al. Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial. Int J Antimicrob Agents. 2020;56(1):105949. [FREE Full text] [CrossRef] [Medline]
    27. Crowley M, Thomas K, Haberman M. Ignoring expert opinion, Trump again promotes hydroxychloroquine. New York City. New York Times; 2020. URL: https://www.nytimes.com/2020/04/05/us/politics/trump-hydroxychloroquine-coronavirus.html [accessed 2024-10-10]
    28. US Food and Drug Administration. FDA cautions against use of hydroxychloroquine or chloroquine for COVID-19 outside of the hospital setting or a clinical trial due to risk of heart rhythm problems. URL: https:/​/www.​fda.gov/​drugs/​drug-safety-and-availability/​fda-cautions-against-use-hydroxychloroquine-or-chloroquine-covid-19-outside-hospital-setting-or [accessed 2024-10-10]
    29. Miller Z, Marchione M, Superville D. Trump says he's taking malaria drug to protect against virus. AP News. URL: https:/​/apnews.​com/​article/​virus-outbreak-malaria-donald-trump-ap-top-news-politics-0fbe485717c9a74e6d6c48aee7d208ae [accessed 2024-10-10]
    30. Beaubien J. WHO halts hydroxychloroquine trial over safety concerns. NPR News. URL: https:/​/www.​npr.org/​sections/​coronavirus-live-updates/​2020/​05/​25/​861913688/​who-halts-hydroxychloroquine-trial-over-safety-concerns [accessed 2024-10-10]
    31. US Food and Drug Administration. Coronavirus (COVID-19) update: FDA revokes emergency use authorization for chloroquine and hydroxychloroquine. 2020. 2020. URL: https://tinyurl.com/5cczudeh [accessed 2024-10-10]
    32. Tanni SE, Bacha HA, Naime A, Bernardo WM. Use of hydroxychloroquine to prevent SARS-CoV-2 infection and treat mild COVID-19: a systematic review and meta-analysis. J Bras Pneumol. 2021;47(5):e20210236. [FREE Full text] [CrossRef] [Medline]
    33. Gao J, Tian Z, Yang X. Breakthrough: chloroquine phosphate has shown apparent efficacy in treatment of COVID-19 associated pneumonia in clinical studies. Biosci Trends. 2020;14(1):72-73. [FREE Full text] [CrossRef] [Medline]
    34. Su Y, Ling Y, Ma Y, Tao L, Miao Q, Shi Q, et al. Efficacy of early hydroxychloroquine treatment in preventing COVID-19 pneumonia aggravation, the experience from Shanghai, China. Biosci Trends. 2021;14(6):408-414. [FREE Full text] [CrossRef] [Medline]
    35. de Wilde AH, Jochmans D, Posthuma CC, Zevenhoven-Dobbe JC, van Nieuwkoop S, Bestebroer TM, et al. Screening of an FDA-approved compound library identifies four small-molecule inhibitors of middle east respiratory syndrome coronavirus replication in cell culture. Antimicrob Agents Chemother. 2014;58(8):4875-4884. [FREE Full text] [CrossRef] [Medline]
    36. Keyaerts E, Vijgen L, Maes P, Neyts J, Van Ranst M. In vitro inhibition of severe acute respiratory syndrome coronavirus by chloroquine. Biochem Biophys Res Commun. 2004;323(1):264-268. [FREE Full text] [CrossRef] [Medline]
    37. Rome BN, Avorn J. Drug evaluation during the COVID-19 pandemic. N Engl J Med. 2020;382(24):2282-2284. [CrossRef] [Medline]
    38. Yu J, Meng S. Impacts of the internet on health inequality and healthcare access: a cross-country study. Front Public Health. 2022;10:935608. [FREE Full text] [CrossRef] [Medline]
    39. Ansari JAN, Khan NA. Exploring the role of social media in collaborative learning the new domain of learning. Smart Learn Environ. 2020;7(1):9. [CrossRef]
    40. Liu S, Zaigham GHK, Rashid RM, Bilal A. Social media-based collaborative learning effects on student performance/learner performance with moderating role of academic self-efficacy. Front Psychol. 2022;13:903919. [FREE Full text] [CrossRef] [Medline]
    41. Vosoughi S, Roy D, Aral S. The spread of true and false news online. Science. 2018;359(6380):1146-1151. [CrossRef] [Medline]
    42. Bowman S. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manag. 2013;10(Fall):1c. [FREE Full text] [Medline]
    43. Graber ML, Siegal D, Riah H, Johnston D, Kenyon K. Electronic health record-related events in medical malpractice claims. J Patient Saf. 2019;15(2):77-85. [FREE Full text] [CrossRef] [Medline]
    44. Twitter. COVID-19 Misinformation. URL: https://transparency.twitter.com/en/reports/covid19.html#2021-jul-dec [accessed 2024-10-10]
    45. Marik PE, Khangoora V, Rivera R, Hooper MH, Catravas J. Hydrocortisone, vitamin C, and thiamine for the treatment of severe sepsis and septic shock: a retrospective before-after study. Chest. 2017;151(6):1229-1238. [CrossRef] [Medline]
    46. Lamsal R. Coronavirus (COVID-19) tweets dataset. IEEE Dataport. 2020. URL: https://dx.doi.org/10.21227/781w-ef42 [accessed 2024-10-10]
    47. Joint statement by WHO U, UNICEF, UNDP, UNESCO, UNAIDS, Itu, UN Global Pulse, et al. IFRC. Managing the COVID-19 infodemic: promoting healthy behaviours and mitigating the harm from misinformation and disinformation. 2020. URL: https://tinyurl.com/2s2sf5js [accessed 2024-10-10]
    48. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA. 2002;287(20):2691-2700. [CrossRef] [Medline]
    49. Bora K, Das D, Barman B, Borah P. Are internet videos useful sources of information during global public health emergencies? A case study of YouTube videos during the 2015-16 Zika virus pandemic. Pathog Glob Health. 2018;112(6):320-328. [FREE Full text] [CrossRef] [Medline]
    50. Fung IC, Fu K, Chan C, Chan BSB, Cheung C, Abraham T, et al. Social media's initial reaction to information and misinformation on Ebola, August 2014: facts and rumors. Public Health Rep. 2016;131(3):461-473. [FREE Full text] [CrossRef] [Medline]
    51. Jin F, Wang W, Zhao L, Dougherty E, Cao Y, Lu C, et al. Misinformation propagation in the age of Twitter. Computer. 2014;47(12):90-94. [CrossRef]
    52. Pathak R, Poudel D, Karmacharya P, Pathak A, Aryal MR, Mahmood M, et al. YouTube as a source of information on Ebola virus disease. N Am J Med Sci. 2015;7(7):306-309. [FREE Full text] [CrossRef] [Medline]
    53. Sell TK, Hosangadi D, Trotochaud M. Misinformation and the US Ebola communication crisis: analyzing the veracity and content of social media messages related to a fear-inducing infectious disease outbreak. BMC Public Health. 2020;20(1):550. [FREE Full text] [CrossRef] [Medline]
    54. Seltzer E, Horst-Martz E, Lu M, Merchant R. Public sentiment and discourse about Zika virus on Instagram. Public Health. 2017;150:170-175. [CrossRef] [Medline]
    55. Sharma M, Yadav K, Yadav N, Ferdinand KC. Zika virus pandemic-analysis of Facebook as a social media health information platform. Am J Infect Control. 2017;45(3):301-302. [CrossRef] [Medline]
    56. Wood MJ. Propagating and debunking conspiracy theories on twitter during the 2015-2016 Zika virus outbreak. Cyberpsychol Behav Soc Netw. 2018;21(8):485-490. [FREE Full text] [CrossRef] [Medline]
    57. Bahk CY, Cumming M, Paushter L, Madoff LC, Thomson A, Brownstein JS. Publicly available online tool facilitates real-time monitoring of vaccine conversations and sentiments. Health Aff (Millwood). 2016;35(2):341-347. [CrossRef] [Medline]
    58. DeVerna MR, Pierri F, Truong BT, Bollenbacher J, Axelrod D, Loynes N, et al. CoVaxxy: a collection of English-language Twitter posts about COVID-19 vaccines. Proc Int AAAI Conf Web Soc Media. 2021;15(1):992-999. [CrossRef]
    59. Donzelli G, Palomba G, Federigi I, Aquino F, Cioni L, Verani M, et al. Misinformation on vaccination: a quantitative analysis of YouTube videos. Hum Vaccin Immunother. 2018;14(7):1654-1659. [FREE Full text] [CrossRef] [Medline]
    60. Mahoney LM, Tang T, Ji K, Ulrich-Schad J. The digital distribution of public health news surrounding the human papillomavirus vaccination: a longitudinal infodemiology study. JMIR Public Health Surveill. 2015;1(1):e2. [FREE Full text] [CrossRef] [Medline]
    61. Panatto D, Amicizia D, Arata L, Lai PL, Gasparini R. A comprehensive analysis of Italian web pages mentioning squalene-based influenza vaccine adjuvants reveals a high prevalence of misinformation. Hum Vaccin Immunother. 2018;14(4):969-977. [FREE Full text] [CrossRef] [Medline]
    62. Schmidt AL, Zollo F, Scala A, Betsch C, Quattrociocchi W. Polarization of the vaccination debate on Facebook. Vaccine. 2018;36(25):3606-3612. [CrossRef] [Medline]
    63. Al-Rakhami MS, Al-Amri AM. Lies kill, facts save: detecting COVID-19 misinformation in Twitter. IEEE Access. 2020;8:155961-155970. [FREE Full text] [CrossRef] [Medline]
    64. Chou WS, Oh A, Klein WMP. Addressing health-related misinformation on social media. JAMA. 2018;320(23):2417-2418. [CrossRef] [Medline]
    65. Dutta A, Beriwal N, Van Breugel LM, Sachdeva S, Barman B, Saikia H, et al. YouTube as a source of medical and epidemiological information during COVID-19 pandemic: a cross-sectional study of content across six languages around the globe. Cureus. 2020;12(6):e8622. [FREE Full text] [CrossRef] [Medline]
    66. Li HO, Bailey A, Huynh D, Chan J. YouTube as a source of information on COVID-19: a pandemic of misinformation? BMJ Glob Health. 2020;5(5):e002604. [FREE Full text] [CrossRef] [Medline]
    67. Memon S, Carley K. Characterizing COVID-19 misinformation communities using a novel Twitter dataset. arXiv. Preprint posted online on 2008. 2020:1-9.
    68. Pulido CM, Villarejo-Carballido B, Redondo-Sama G, Gómez A. COVID-19 infodemic: more retweets for science-based information on coronavirus than for false information. Int Sociol. 2020;35(4):377-392.
    69. Twitter. Legacy verification policy. 2023. URL: https://help.twitter.com/en/managing-your-account/legacy-verification-policy#categories [accessed 2024-10-10]
    70. American Medical Association. Social meda guidance for physicians taps timeless principles. 2018. URL: https:/​/www.​ama-assn.org/​delivering-care/​ethics/​social-media-guidance-physicians-taps-timeless-principles [accessed 2024-10-10]
    71. Nguyen BM, Lu E, Bhuyan N, Lin K, Sevilla M. Social media for doctors: taking professional and patient engagement to the next level. Fam Pract Manag. 2020;27(1):19-14. [FREE Full text] [Medline]
    72. Choo EK, Ranney ML, Chan TM, Trueger NS, Walsh AE, Tegtmeyer K, et al. Twitter as a tool for communication and knowledge exchange in academic medicine: a guide for skeptics and novices. Med Teach. 2015;37(5):411-416. [CrossRef] [Medline]
    73. Data Transfer Agreement Signatories. National Institutes of Health. URL: https://ncats.nih.gov/n3c/resources/data-contribution/data-transfer-agreement-signatories [accessed 2024-10-10]
    74. Haendel MA, Chute CG, Bennett TD, Eichmann DA, Guinney J, Kibbe WA, et al. N3C Consortium. The National COVID Cohort Collaborative (N3C): rationale, design, infrastructure, and deployment. J Am Med Inform Assoc. Mar 01, 2021;28(3):427-443. [CrossRef] [Medline]
    75. N3C Core Contributors. N3C. URL: https://covid.cd2h.org/contributors/ [accessed 2024-10-10]

    ‎
    MERS-CoV: Middle East respiratory syndrome
    N3C: National COVID Cohort Collaborative
    PCR: polymerase chain reaction
    SARS-CoV-1: severe acute respiratory syndrome coronavirus
    VAR: vector auto-regressive

    Edited by A Mavragani; submitted 23.01.24; peer-reviewed by K Lin; comments to author 06.09.24; revised version received 09.09.24; accepted 12.09.24; published 18.11.24.

    Copyright

    ©Scott A Helgeson, Rohan M Mudgalkar, Keith A Jacobs, Augustine S Lee, Devang Sanghavi, Pablo Moreno Franco, Ian S Brooks, National COVID Cohort Collaborative (N3C). Originally published in JMIR Infodemiology (https://infodemiology.jmir.org), 18.11.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Infodemiology, is properly cited. The complete bibliographic information, a link to the original publication on https://infodemiology.jmir.org/, as well as this copyright and license information must be included.