This research examined the portrayal of scientific evidence and uncertainty of 3 unproven COVID-19 therapeutics prior to the development of vaccines and compared its representation among the body (full text or video) of news reports with headlines and lead paragraphs. There are 2 widely understood and generally accepted paradigms of science communication that focus on the unidirectional delivery of information or active engagement with the public [25-27]. These paradigms can further be divided into 4 models of science communication, which are science literacy (also referred to as the deficit model), contextual, lay expertise, and public engagement or public participation [28]. The science literacy model is premised that information transmitted by experts aims to reduce the deficit of science knowledge among the public while the contextual model similarly transmits information, but situates information to specific audiences paying attention to culture, location, and language among other relevant contexts. Both, however, focus on the transmission of scientific information. The lay expertise model acknowledges the limitations of science and endorses “lay” knowledge as equal to scientific knowledge. The public engagement model aims to address science policy issues through active engagement and interaction with the public in a democratic process. These latter 2 models focus on improving engagement between science and society and recognize the limitations of science and refrain from giving science any more authority than other knowledge sources. Several scholars have conceptualized variations or additions to these models and outlined new goals of science communication [26,27,29,30]. However, these models remain largely theoretical and provide little practical direction toward the practice of science communication today, which involves an array of different sources (eg, internet, social media, and legacy media), actors (eg, general and science journalists, bloggers, influencers, and others), and approaches (eg, traditional journalistic norms and a plurality of practices among content contributors in the new media environment) [31]. In addition, the models provide little in terms of approaches to analyze contemporary news reports produced by traditional media outlets reported by journalists because there is no accepted conceptual framework on the public communication of science in the media and best practices to convey scientific evidence and uncertainty to the public [32,33]. Instead, our analytical approach was informed by adopting a lens of a changing science media communication landscape where public reporting of science has moved beyond conveying scientific facts and where journalists use frames to describe science accurately while maintaining audience engagement [31,34]. This, however, results in multiple challenges in accurately reporting science to the public including science hype [35]; errors of omission [36]; failure to report funding sources and conflicts of interest [37]; inadequately detailing methods, risks, and timelines [38-40]; improperly addressing scientific uncertainty; and situating science in particular contexts, for example, politics.

There is variability in the portrayal of scientific evidence and uncertainty in the news media. Some studies have shown less accurate portrayals where the results are discussed as certain and the benefits overemphasized [41-44], while other studies have demonstrated inaccuracies in reporting evidence when comparing press releases and media articles with scientific publications resulting in oversimplification, absence of quantification, lack of explicitly reporting data sources, and the absence of study limitations [45,46]. On the other hand, scientific uncertainty on climate change among other specific topics has been heightened in news media by journalists framing science as inconclusive or providing equal space for opposing viewpoints, even if one viewpoint is supported by substantial scientific consensus and evidence while the other lacks credible evidence [47-49]. News reports of scientific discoveries aiming to provide balanced perspectives may inadvertently cause a false balance and distort the public’s perception of consensus surrounding scientific evidence [50].

This research was also informed by the reality that the public often engages with snippets of information including a focus and evaluation of articles based on headlines [51,52]. Our focus on analyzing scientific evidence and uncertainty in headlines and lead paragraphs, and comparing it with the body of news reports was further informed by the primacy effect—a cognitive bias where people recall and place greater emphasis on initial pieces of information [53,54]—which may explain why headlines and lead paragraphs are especially important in shaping the audience’s understanding and judgment of news stories.

Finally, our conceptual framework was also informed by a media environment with a higher prevalence of COVID-19 health misinformation and an understanding of the efforts to correct news reports by health and scientific experts through media fact-checking and myth-busting approaches [55-58]. While several studies have shown the effectiveness of fact-checking and the correction of misinformation postexposure at altering beliefs [59-63], it remains unclear if such approaches would fully address concerns of reported health misinformation. This is partly due to differences in the focus fact-checkers and myth busters consider when aiming to correct scientific facts, the framing of science news stories by reporters, ambiguity and discordance in the interpretation of scientific facts, and that truth-telling may not always reach intended audiences [64-66]. Such approaches epistemologically rely on the assumption that facts, even in the context of scientific evidence, can always be clearly discerned from nonfactual information and that experts can identify and expose truths from falsehoods; such approaches may minimize or discount completely the complex interplay between beliefs, politics, and science [67]. Being mindful of this context, our analysis focused on how, where, and by whom scientific evidence surrounding unproven COVID-19 therapeutics was portrayed in the news stories themselves and we did not address the veracity of the scientific evidence presented in the news.

We chose to investigate 3 unproven products—hydroxychloroquine, remdesivir, and CP—based primarily on their popularity in US news among other factors that informed our decision. Several antivirals and immunomodulators were being considered as potential classes of therapeutics used to treat COVID-19 in the early 2020s [68]. An initial set of news database searches (discussed in the next section) identified hydroxychloroquine (4023 publications), remdesivir (2839 publications), azithromycin (416 publications), and CP (372 publications) as the most popular unproven products in US news compared to other products being considered including, lopinavir, interferon beta-1a, dexamethasone, and heparin or low molecular weight heparin. For feasibility reasons, we chose to evaluate 3 products and decided to include CP instead of azithromycin because after removing news reports discussing each product in conjunction with hydroxychloroquine, there were 344 articles discussing CP and only 57 articles discussing azithromycin. Additionally, the choice to examine hydroxychloroquine, remdesivir, and CP was supported because these products were being investigated in registered clinical studies in the United States (hydroxychloroquine, ClinicalTrials.gov NCT04714515; remdesivir, ClinicalTrials.gov NCT04257656; and CP, ClinicalTrials.gov NCT05578391) and permitted to be used to treat patients with COVID-19 by the US Food and Drug Administration (FDA; please see the following paragraphs for details).

News sources were identified using the Factiva database by the Dow Jones & Company [69]. Factiva is a commonly used research tool that provides full-text coverage of current and archival business and news information from traditional (eg, broadcast and newspaper) media and news available online. The specific news sources selected for our query were determined by popularity and coverage and were compiled based on 3 independent organizations that rank news based on circulation and internet traffic (see Table S1 in Multimedia Appendix 1) [70-72]. Having a diverse venue of sources is important since issues of unproven COVID-19 therapies have become politicized and studies have shown differing levels of depth and misinformation between news organizations.

After eliminating duplicate news sources listed by these organizations, we queried the terms hydroxychloroquine, Plaquenil, remdesivir, Veklury, and CP in the headline or lead paragraph of articles published from January 1, 2020, to July 30, 2020. The search dates were selected to ensure we captured the portrayal of scientific evidence of early unproven therapeutics to treat COVID-19 in US news. The start date of our search was chosen because a pneumonia-causing novel coronavirus was first announced on January 7, 2020, with the first confirmed US case on January 20, 2020 [73]. The end date was chosen based on clinical investigations and decisions made by the FDA on our 3 therapeutics of interest. Specifically, several clinical investigations examining the 3 potential therapeutics of interest were initiated during the first quarter of 2020, and scientific evidence about the safety and efficacy of the therapeutics was being collected and concluded [68,74,75]. Specifically, large clinical trials of hydroxychloroquine were initiated, paused, and eventually halted from late May to late June [74,76-78]. Based on clinical trial results, the FDA revoked emergency use authorization for hydroxychloroquine on June 15, 2020 [79]. Starting May 28, 2020, participants were being enrolled for a clinical trial on CP and on April 3, 2020, the FDA approved an expanded access protocol for Mayo Clinic to lead an investigation to understand the effects of CP [80]. The FDA also issued an emergency use authorization for remdesivir on May 1, 2020 [81]. Finally, the end date for our search was also chosen for practical reasons as data collection efforts began on July 31, 2020.

The search yielded 1136 reports including online and print articles, television transcripts, and videos. Application of inclusion or exclusion criteria was conducted by SW and TJB (primary coders). The coders matched the headline and news source of the web link obtained from Factiva for online media. Web-based reports with a video were considered 2 independent reports unless the video was verbatim or substantially similar to the associated article. Reports that were considered substantially similar must have had at least 1 area of similarity on the topic, main theme, concluding points, people quoted, or style (eg, debate). In instances of substantial similarity, only the most up-to-date report was included. Reports that were duplicative, unobtainable, or void of search terms were removed resulting in 1103 reports. If multiple therapeutics were discussed in a single report, the codebook was applied to each therapeutic. We randomly sampled a total of 550 news reports from our population to manage coder effort while allowing the capture of codes and themes among news reports. After random sampling, reports without substantial discussion of the COVID-19 therapeutic were excluded if they failed to satisfy codes 7-16 of the codebook (n=72). When a therapeutic was found only in an image caption, tweets, or graphics within the report, the report was also excluded (n=29). After applying exclusion criteria, a final data set of 479 individual discussions of a therapeutic was found in 449 news reports. See Figure S1 in Multimedia Appendix 1 for a flowchart providing an overview of the sampling strategy.

A comprehensive codebook was developed based on the literature described. The purpose of the codebook was to ensure that a consistent analytical framework was adopted by both coders and to reduce randomness in the interpretation of themes and codes. Upon developing an initial draft, the codebook was modified through the iterative analysis of 9 written articles (2 hydroxychloroquine, 2 remdesivir, and 5 CP) from different news sources with different word lengths. The draft codebook was then reviewed by a multidisciplinary team of experts in media analysis, bioethics, and health communication. Independent review of 50 reports by 2 coders (SW and TJB) showed good interrater reliability in coding for the full news report for theme and portrayal of scientific evidence. The median Cohen κ for these elements was 0.71 (IQR 0.58-0.74) with simple agreement ranging from 79% to 94%. Interrater reliability was initially poor when identifying sources of authority, with a median κ of 0.17 (IQR 0.06-0.23). Definitions were clarified to address omissions or ambiguities identified during this intercoder check. For example, evidence was defined as a formal demonstration of the effect of a treatment on the COVID-19 disease course, which includes symptoms, outcomes, side effects, or a lack thereof. In the same way, sample size or study design was added as an example of evidence details and a statement of small sample size or nonrandomized design was included in the codebook as an example of evidence limitations. Codes regarding sources of authority were modified to clarify that a subject matter expert who spoke on behalf of a government agency should not be coded as an “expert” but should be coded as a “government” source of authority. Due to poor initial reliability, modified definitions for sources of authority were formally retested by the same 2 coders on a new sample of 20 articles. Sources of authority showed good interrater reliability with a median κ of 0.73 (IQR 0.48-0.86) with simple agreement ranging from 91% to 100%. The general structure of the codebook captures metadata provided by the Factiva database (eg, word count and hyperlink), headline analysis, report analysis, scientific description, sources of authority, social context, qualitative description of video reports, and coder notes (see Multimedia Appendix 2 for final codebook with complete definitions of codes and examples).

The full text or video (body of news reports) of all discussions was analyzed for theme, scientific evidence, claims of safety or efficacy, and sources of authority. All terms are defined in the tables. The topical themes of news reports were identified inductively during iterative analysis of 9 articles as described above. An inductive approach permits the emergence of themes from the data and allows coders to remain open and exploratory [82]. This qualitative method is ideal for identifying themes when there is no prior knowledge about them [38,83,84]. In cases where more than 1 theme was reflected in the entire news report, the coder selected the theme that most closely resembled the news report. For text-based reports that discussed scientific evidence, details of evidence, or limitations of evidence, we also analyzed the presence of these codes within headlines and lead paragraphs. The lead paragraph was defined as the first paragraph of text with a minimum of 2 complete sentences excluding subtitles or alternate headlines just below the large print title. Coders independently analyzed approximately half the reports and jointly reviewed challenging reports and a random 10% of all reports as preplanned audits. Disagreements were resolved by consensus. Study data were recorded and managed using REDCap (Research Electronic Data Capture).

Different traits of reports (eg, discussion of evidence and discussion of specific side effects) were summarized for each therapeutic using descriptive statistics. Fisher exact tests were used to compare differences in trait frequencies. If 3×2 Fisher tests returned a P value of <.05, pairwise comparisons between therapeutics (ie, hydroxychloroquine - remdesivir, hydroxychloroquine-CP, and remdesivir-CP) were performed. No corrections were made for multiple comparisons. Analyses were performed in Excel (Microsoft Corp) and R (version 3.5.1; R Core Team).

This research study did not involve human subjects and was conducted using publicly available information; thus, ethics approval was not sought.

As reports spread of a novel coronavirus causing unprecedented hospitalizations and mortality, both medical specialists and everyday persons anxiously scrambled to learn as much and as fast as possible about prevention and potential treatments. Our analysis revealed that much coverage of the off-label use of hydroxychloroquine centered on politics and the safety and efficacy of the product. The politics theme dominated discourse surrounding hydroxychloroquine due to its endorsement by former President Trump as seen in other news media analyses [58,85]. During the months of March and April 2020, former President Donald Trump put forth hydroxychloroquine as a “game changer” after a study suggested efficacy in vitro. Propagation of statements surrounding the safety and efficacy of hydroxychloroquine had far-reaching effects, including increased sales by hospitals and the death of persons who took a similar product as prophylaxis [86]. While safety or efficacy discussions were also found in remdesivir and CP, limited knowledge of its antiviral properties may have played a significant role in shaping public discourse on the potential safety and efficacy of the drug. Conversations about CP instead focused on the human interest theme explicating the need for plasma donors and inspiring potential COVID-19 survivors to donate.

The results showed substantial heterogeneity in the sources of authority asserting claims of safety or efficacy of the COVID-19 therapeutics among the 449 reports. On many occasions, scientific claims of the safety and efficacy of a therapeutic were portrayed by prominent persons and were compared to the evidence discussed by experts. Contrary opinions on evidence of hydroxychloroquine delivered by politicians and experts may have inadvertently contributed to a false sense of scientific disagreement as news reports most frequently reported on claims made by a single prominent person but also mentioned the lack of scientific evidence supporting such claims. Presenting directly opposing viewpoints as equal when evidence or a scientific consensus is actually weighted is referred to as a false balance that can cause public uncertainty [87]. False-balance has been shown to distort public perceptions of consensus among experts resulting in people identifying greater disagreement and controversy than what is actually present [50]. Making such comparisons is problematic because nonscientific experts may interpret data toward a political advantage and their influence could have troubling downstream consequences when disseminated, for example, physicians prescribing unproven products and reduced intentions to adopt health behaviors [88-90]. Though the majority of news reports on hydroxychloroquine published the safety or efficacy claims made by prominent persons, very few reports actually portrayed hydroxychloroquine as efficacious, and safety warnings or specific side effects were frequently included.

While we found scientific evidence of therapeutics was portrayed in the body of news reports, mostly on hydroxychloroquine, limitations of evidence outlining scientific uncertainty were seldom seen, and rarely found in prominent locations (headlines and lead paragraphs) of news reports. With only about 40% of the public delving past headlines or lead paragraphs when reading articles [3,91], it is unlikely readers would grasp the scientific uncertainty associated with an unproven therapeutic. The dissemination of evolving science without expressing scientific uncertainty and the knowledge gap created by the iterative scientific process may have indirectly resulted in public confusion and minimized how large of a threat individuals believed the virus posed [92,93]. This confusion and doubt permitted some actors to promote disinformation surrounding the safety and efficacy of unproven therapeutics [8,21]. Journalists may be reluctant to express uncertainty of evidence in their reports out of fear their audience may react negatively toward such ambiguity [94] or the journalist may not be equipped to address a study’s findings, especially its limitations. Scientists on the other hand are reportedly reluctant to express uncertainty fearing reporters may lose interest and that the public might misinterpret the science or doubt scientists [95,96]. However, empirical studies have shown that neither conveying uncertainty nor presenting scientific limitations affect a person’s understanding of a topic, beliefs, or trust in science [97-99]. This data, along with our findings, suggest that uncertainty should be explained more frequently in news reports and placed in headlines and lead paragraphs to adequately support the public’s understanding of scientific evidence and uncertainty [30,87]. Further research comparing scientific evidence and expressions of uncertainty between the body of news reports versus headlines and lead paragraphs on different scientific topics is needed. Research examining key factors influencing journalists’ and news editors’ decision-making process regarding news headlines and lead paragraphs is important to inform updated journalistic guidelines and editorial policies aimed at reducing false balance. Research collaboration between journalists, scientists, and communication researchers to identify effective communication strategies and journalistic practices to ensure accurate and nuanced representation of scientific information in news reporting would be valuable.

The accurate reporting of scientific evidence and uncertainty is the collective responsibility of multiple professionals including reporters, editors, public health officials, and medical and scientific experts. While evidence from experts may be based on scientific facts, expert correspondents should make clear scientific uncertainty, avoid hyperbolic statements, and explain how the state of knowledge may change [22]. Medical doctors and scientists should not express unjustified certainty when forecasting the safety and efficacy of unproven therapeutics [100]. Despite the lack of consensus, reporters should offer greater context or altogether avoid framing unsubstantiated claims by prominent people alongside evidence-based claims from experts as it may provide a false impression of genuine scientific uncertainty. Editors should refrain from sensationalizing headlines or framing lead paragraphs that do not reflect evidence or study limitations. Instead, journalists and experts alike should focus on the accurate portrayal of the scientific studies presented in the report and aim to provide an accurate reflection of the safety and efficacy of unproven products to help the public understand why there is incomplete evidence and a lack of definitive recommendations. Prioritizing the accurate portrayal of science may help to rebuild public trust in experts, minimize risky behaviors, and ensure compliance with public health recommendations.

Our study has several limitations worth mentioning. First, the analysis included only news sources within the United States, all of which were written in English and thus may not cater to news outside of the country and written in different languages. Second, the analysis focused on traditional, mainstream news sources and we excluded social media. Although much COVID-19 information and misinformation were disseminated through social media contributing to an infodemic, traditional news remains the major form of information transfer, especially among older Americans, which is why we focused on evaluating traditional news sources [101]. Third, we chose to analyze approximately half the news reports in our population and the sample may not be representative of the entire news discourse within the full data set. Fourth, the data collection and analysis was restricted to the 3 most popular unproven COVID-19 therapeutics and other therapeutics during that period may be portrayed differently in US news. Finally, the results were not stratified based on different political leaning of news sources included in our analysis, which could affect how scientific evidence and uncertainty were portrayed in our data set.

News media plays a significant role in informing the public by serving as a crucial link between public health authorities interpreting scientific evidence and determining its implications and the public. Rapid publication of peer-reviewed and nonreviewed science allowed the media to bring breaking news about discoveries of COVID-19 therapeutics to the public and convey public health recommendations [18]. Prioritizing the accurate portrayal of science can help shape the impact of a public health emergency by influencing public perceptions and activities such as minimizing risky behaviors and encouraging compliance with public health recommendations. Publicizing that the news media source is likely to share frequent updates from scientific findings and communicating information as the best-known information at the time may help build public trust in scientific experts and decrease feelings of doubt and anxiety. The accurate reporting of scientific evidence and uncertainty is the collective responsibility of journalists and science communicators. Public trust in science can be strengthened by acknowledging evidentiary limitations, avoiding a false sense of disagreement, and portraying science as an iterative, self-corrective process that generates reliable knowledge.