The main sequence of study procedures is illustrated in Figure 1 and includes (1) a structured literature review, in which we assemble linguistic characteristics that were used in algorithms for distinguishing factual information and misinformation (phase 1); (2) data collection, which encompasses assembling cancer-related posts using the X application programming interface (API) and labeling them as misinformation and non-misinformation (phase 2); (3) identification of the linguistic characteristics in collected datasets using natural language processing tools (phase 3); and (4) conducting predictive modeling analysis to evaluate the effectiveness of linguistic characteristics in distinguishing social media posts with cancer misinformation (phase 4).

The study was institutional review board–approved by the University of North Carolina (IRB#21-2861). This was an analysis of publicly available data. As such, participants were not compensated and did not need to provide consent for the study, because the study did not involve any prospective data collection. To protect the confidentiality and anonymity of participants in this secondary data analysis, we reworded reported posts from X.

To identify linguistic characteristics, we developed a literature review protocol that included the search strategy and keywords. This process was informed by a collaboration with a health sciences librarian (CBS), who suggested an initial set of keywords referenced in several relevant reviews [17-21]. She also created an expanded title, abstract, and keyword search strategies for each of the following concepts: (1) text as a unit of analysis, (2) misinformation, (3) algorithms, (4) internet, and (5) linguistic features or characteristics. After the search was peer reviewed by a second health sciences librarian (CB), 5 databases were searched: ProQuest Central (ProQuest), which includes the arXiv repository; Scopus (Elsevier); IEEE Xplore (Institute of Electrical and Electronics Engineers); ACM Digital Library (Association for Computing Machinery); and Communication & Mass Media Complete (EBSCOhost). The keywords and search strategies are reported in Multimedia Appendix 1. Results were limited to citations published between January 2012 and December 2022. Within databases, results were limited to journal papers, conference proceedings, working papers, and book chapters.

Two reviewers (IF and DB) independently coded titles and abstracts in Covidence software (Veritas Health Innovation) [22] and resolved conflict in codes during research meetings. Papers were included if they focused on detecting misinformation and contained a “Methods” section describing an approach for algorithmically detecting misinformation (eg, reviews and viewpoints were excluded). Examples of the algorithms included supervised and semisupervised machine learning (eg, Bidirectional Encoder Representations from Transformers [BERT] classification) that was built on linguistic characteristics. Papers were excluded if they did not report specific linguistic characteristics, focused on misinformation in any language other than English, or used human coding but not algorithms. The detailed inclusion-exclusion criteria and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram are reported in Figure 2.

Upon identifying eligible papers, 2 team members (IF and DB) reviewed the full text and extracted the linguistic characteristics. Around 11% (10/90) of papers underwent double-coding. After reviewers reached an agreement, we continued with single coding. The linguistic characteristics were extracted based on the following criteria: observability, applicability, and generalizability. The observability criterion was related to whether readers could easily observe the linguistic characteristics within the text; for example, positive emotions could be easily observed while morale or cognitive language styles may be difficult to distinguish. The applicability criterion distinguished linguistic characteristics that readers could easily apply while reading the text. For instance, common characteristics such as the number of words required substantial effort from readers to evaluate and, therefore, were deemed nonapplicable. In contrast, readers could easily use citations and hashtags in their post evaluations as the mere presence of these characteristics was determined to be helpful in identifying misinformation. The third criterion, generalizability, was chosen to ensure that linguistic characteristics were not related to a specific context but could be applied across various contexts. Thus, characteristics that were based on specific words such as “COVID-19,” or “cure” were excluded.

Upon data collection and labeling, we used algorithmic approaches to model linguistic characteristics. First, we used an automated text search using regular expressions in Python [35] to capture digital numbers, hashtags, and URLs in the text.

Second, we used the Linguistic Inquiry and Word Count (LIWC) software [36]. LIWC calculates the proportion of the words in the posts associated with distinct psychological dimensions [37]. In this study, LIWC identified when authors of posts used certain, absolute, or tentative language.

Third, we leveraged the natural language processing tool, Name Entity Recognition [38], which was trained on human-labeled datasets to extract names from unstructured text. Using Name Entity Recognition, we were able to identify which posts contained personal names, organizational names, or locations identified from text.

Fourth, we experimented with several models for sentiment analysis and identified the DistilBERT algorithm as an optimal approach for its accuracy in handling health-related X data [39]. DistilBERT is a black-box algorithm that is trained on a large corpus of data and is based on multiple deep stack layers. The DistilBERT algorithm identified positive, negative, and neutral tones present in the posts. To evaluate the algorithm’s performance, we manually labeled 300 posts across the databases. On average, the DistilBERT algorithm achieved an 83% accuracy (82% for misinformation and 84% for the control database) in detecting the emotional tone within the posts.

Identified linguistic characteristics were used in an algorithm to test whether these could distinguish misinformation in posts. As shown in Figure 1, we conducted 2 experiments using tidymodels package in R software [40]. Using linguistic characteristics as predictors, we forecast the “misinformation” and “non-misinformation” labels in the datasets semimanually coded by researchers and BERT classification algorithm. Data were split 60:40 to enable evaluation of the predictive power of the model and reported the accuracy as a ratio of correctly classified posts to the total number of posts. We also reported area under the curve (AUC), which accounted for both false-positive and false-negative predictions. AUC value ranged from 0 to 1, where 0.5 indicated that the model performs no better than a random chance, and 1 was a perfect prediction. The model was built on the basis of Lasso (“Least Absolute Shrinkage and Selection Operator”) regression, which allowed variable selection by shrinking the coefficients of less important predictors to zero [41]. Bootstrapping procedure was applied to optimize and stabilize the selection of variables [42]. Lasso was chosen to address multicollinearity and overfitting issues in the regression model. More importantly, Lasso regression helped identify a set of linguistic characteristics that effectively distinguished posts containing misinformation. To evaluate the significance of specific linguistic characteristics, we computed importance scores, with higher scores indicating greater relevance in distinguishing posts containing misinformation. Importance scores, a common measure in predictive modeling, indicates to what extent individual predictors contribute to the overall model performance. The assessment involves permutating the characteristic values through shuffling and measuring the subsequent decline in model performance, effectively revealing the critical factors influencing predictions. Finally, we conducted a permutation statistical test (with 1000 permutations) to determine whether models with linguistic characteristics significantly outperformed random chance.

A total of 5677 citations were initially identified across all databases. After removing 1598 duplicates, we screened 4070 unique citations in Covidence. Subsequently, 3605 were excluded during the title and abstract review phase, leaving 464 papers for full-text review. Ultimately, we extracted linguistic characteristics from 88 full-text papers. These papers featured algorithmic approaches for identifying misinformation through automated text analysis, spanning various contexts, including politics, social issues, and computer science. Exclusion reasons are detailed in Figure 2, and additional information about the included papers can be found in Multimedia Appendix 3.

The extracted linguistic characteristics and corresponding literature are detailed in Table 1. Representative examples that contain each linguistic characteristic were chosen by selecting posts from the misinformation dataset. We used results from linguistic characteristic modeling to identify such posts. The first category of characteristics pertains to the sentiment and emotional expression in the text and includes positive, negative emotions, and neutral sentiments (absence of either). Some papers delved into more nuanced emotions such as anger, fear, surprise, and others. We excluded these emotions due to the potential difficulty for readers to detect nuanced emotions reliably in the text.

The next category comprises linguistic characteristics that pertain to psychological concepts. It is worth noting that some psychological concepts consist of a combination of linguistic characteristics, such as social processes including references to family, friends, other people, and verbs indicating interactions. Although algorithms frequently use such combinations, we decided to exclude the following psychological concepts that consisted of combinations of linguistic characteristics such as cognitive, perceptual, social processes, and morality or deception. The rationale behind this exclusion is that users are unlikely able to observe and combine linguistic characteristics for evaluations of the posts. We also excluded characteristics mentioned in fewer than 4 studies, such as gratitude, insight, causation, and persuasion. Following our 3 criteria, we included negations, tentativeness, profanity (as a proxy of informality), and words associated with absolutes and certainty.

Other categories that met our inclusion criteria were linguistic characteristics such as names of individuals, locations, and organizations, as well as categories related to the presence of URLs, hashtags, personal pronouns, and numbers. Readers can identify these characteristics without additional efforts (observability criterion) and use them for evaluation of the text (applicability) because the presence of these characteristics in social media has historically been a distinguishable factor in detecting misinformation. Furthermore, these characteristics were not context-dependent and, therefore, satisfy the generalizability criterion.

We collected a total of 45,791 posts related to unproven cancer therapies. Among these, 13,046 posts were labeled as misinformation (forming the misinformation dataset), while 32,745 posts were categorized as non-misinformation (comprising control dataset 1). Furthermore, we gathered 6782 posts from the profiles of comprehensive cancer centers, which were used as control dataset 2, as shown in Figure 1. The content description of both the misinformation dataset and the control dataset 1 is shown in Table 2. To illustrate the dataset in this study, we categorized the X posts into 9 distinct categories. The examples of the posts with misinformation are shown in Table 1.

As shown in Table 3, experiment 1 demonstrated that linguistic characteristics predicted misinformation with 60% accuracy. In experiment 2, they exhibited even stronger predictive power, achieving an accuracy of 77%. The importance scores for each linguistic characteristic are shown in Table 4.

Next, we selected linguistic characteristics with an impact score 0.05 and consistent predictive performance across experiments 1 and 2. These short-listed characteristics underwent further testing within the same experiments. In experiment 1, the short-listed characteristics achieved an accuracy rate of 50%, which did not significantly differ from random chance (P>.90). However, in experiment 2, these characteristics predicted misinformation with an accuracy rate of 73% and an AUC of 83. This performance was significantly better than random chance (McNemar ²₁=5.7 ×10⁷; P<.001). The importance scores for the short-listed characteristics are shown in Table 4. For a more detailed breakdown of the importance scores, we have summarized the percentage of posts containing these short-listed characteristics by dataset in Table 4 and the complete list in Multimedia Appendix 4.

We have identified linguistic characteristics that can help people affected by cancer detect cancer misinformation on social media platforms such as X. Linguistic characteristics that were likely to be present in posts with misinformation were related to certain, absolute language, and numbers. Certain language included phrases that reflected a “degree of bravado” or “boasting of certainty.” Examples of certain languages could be “I really believe,” “it is definitely helpful,” and similar others [36]. The absolute language referred to phrases that reflect black-and-white thinking and included words such as “none,” “all,” “never,” and others [36]. The number category encompassed any information reported with digits such as percentages, count of any units, years, and priorities. Notably, all 3 linguistic characteristics could be united under the umbrella of definite, confident language. Linguistic characteristics that were unlikely to be present in posts with misinformation encompassed URLs, hashtags, and location mentions. Each of these attributes could be considered as a form of citation or reference. URLs offered direct links to the original source or further information, hashtags connected posts to broader relevant discussions, while locations mentioned in posts provided context and a sense of origin to the information shared. Our findings are consistent with some of the suggestions provided by previous guidelines for identifying misinformation. For instance, the Food and Drug Administration recommends being vigilant if patients read confident statements such as a drug definitely “cures cancer” or “guarantees results” [124]. Other guidelines encouraged users to search for references and original sources of health-related information [12-14].

While consistent with previous recommendations, our findings make a unique contribution. Previous work has based the guidelines on theoretical assumptions, while our study is one of the first to provide some empirical evidence based on a large dataset to support the recommendations for users. Another contribution is that we outlined ineffective linguistic characteristics for detecting cancer misinformation. Despite a substantial body of research showing that social media posts with sentiments predicted fake news, we did not find these relationships. A potential explanation could be the algorithm’s limited efficiency in identifying emotions within cancer-related contexts. Furthermore, it is possible that authors express a limited range of emotions in cancer-related conversations, typically negative emotions toward cancer and both positive and negative emotions toward various treatments, including those that are unproven. These emotions may vary little across posts containing valid and nonvalid information, making emotions an unreliable factor for distinguishing misinformation.

Our work accumulates knowledge about misinformation detection from the literature covering a wide range of contexts—including political, social, and computer science—and translates this knowledge to the cancer context. The findings highlighted promising avenues for future research and could expedite the development of automated and augmented methods for identifying and verifying cancer-related misinformation on social media platforms. Finally, the robust labeled datasets developed by our research team are available to other researchers upon request to the corresponding author, thereby further supporting research on misinformation within the context of cancer and social media.

In practice, our work is at the forefront of customizing recommendations and contextualizing them for social network users. Our exploratory findings suggest a promising direction for studying linguistic characteristics that information users might apply when making quick judgments while scrolling through X feeds. Empowering users to stay vigilant in their initial evaluations could help reduce the spread of misinformation and the formation of erroneous beliefs. This is a crucial area for future research, which should explore how these findings apply in different cancer-related contexts and across various social networks.

All the studies included in our analysis exclusively originate from peer-reviewed journals and conference proceedings; however, we must exercise caution when considering the potential for publication bias. Furthermore, in accordance with our selection criteria for linguistic characteristics, we included only those papers that focused on text and excluded other forms of social media content, such as videos and images. We recommend that future research comprehensively explore social media, including multimedia content, as it could potentially provide additional insights for user-friendly recommendations.

In selecting linguistic characteristics, we prioritized observability, applicability, and generalizability. However, alternative criteria may be considered when users are open to a more thorough exploration of a post’s validity. For example, future research should explore the use of metadata, link content analysis, and hashtag meanings. As misinformation evolves and its authors adjust to societal changes, the linguistic characteristics that identify misinformation may also shift. A longitudinal analysis is necessary to understand how linguistic characteristics perform in predicting misinformation over time.

Algorithms used in our analysis operate with a certain level of accuracy. Specifically, the accuracy of label identification in the dataset reached 83%, indicating that approximately 17% of posts were labeled incorrectly. This means that in experiment 1 some proportion of misinformation is included in the non-misinformation group and vice versa, making further exploration less accurate in experiment 1. This degree of uncertainty is common in algorithmic performance. Therefore, it is important to interpret our results in light of the inherent imperfections in algorithmic performance.

Furthermore, we encountered that the short-listed linguistic characteristics did not significantly outperform random chance in identifying misinformation in experiment 1. This outcome underscores a potential boundary condition of the effectiveness of the linguistic characteristics. Notably, experiment 1 encompassed more homogeneous data in contrast to experiment 2. Based on these findings, it becomes plausible to speculate that linguistic characteristics might provide limited help when a reader assesses posts within a closely knit community.

In experiment 2, the control dataset 2 consisted of posts shared by cancer centers and was compared with the misinformation dataset comprising random posts. To address this limitation, we collected posts from cancer centers that contain words related to cancer therapies. This step was taken to ensure a similar context of discussion as the posts with unproven therapy. Next, we exclude linguistic characteristics that are likely displayed differences between datasets due to the distinct nature of the information within control dataset 2. For example, linguistic traits such as “the use of profanity” or “first-person pronouns” were discarded. Furthermore, we decided to focus our analysis solely on the text within the posts and omitted other accompanying metainformation that users might observe, such as the user’s name, location of the author, and posting time. This approach allowed us to assume that posts shared by cancer centers might be perceived more broadly, for instance, as posts shared by researchers, physicians, administrators, and patient advocates. Because of these measures, we anticipate that the linguistic characteristics identified in this research may help differentiate between health misinformation and factual posts on social media, irrespective of their sources. Despite our precautionary measures, we cannot fully guarantee that identified linguistics characteristics certainly distinguish between posts with misinformation and non-misinformation versus posts produced by the general public and posts by health experts from health care systems. However, there are factors that support the first conclusion more than the second. First, our findings are consistent with the previous theoretical and practical recommendations for identifying misinformation [12-14]. Second, the associated with misinformation linguistic characteristics, such as numbers and assertive language, are expected to be used by health experts. For instance, providers use numbers more confidently than the general public [125]. Professional guidelines for health providers encourage them to use numbers over verbal descriptions [126] as well as the use of assertive language in communication with patients [127,128]. Yet, our study associated these characteristics with misinformation shared by the general public on social media, which suggests that we might be finding more than just a mere distinction between the general public language and the health professional language. One study in and of itself is not yet a comprehensive body of evidence. Our findings will need to be validated and built upon via additional studies—including those that use posts from other types of entities and comparison groups.

Finally, our data were collected only on a single social network X. Many characteristics and customs of X are transferable to other social networks and our recommendations are likely to go beyond application on X, as demonstrated by the consistency of our recommendations with the recommendations of other researchers [12-14]. Given this limitation, our results need to be generalized cautiously, and further similar research is needed for different platforms (eg, Facebook, Pinterest, etc).

Our structured review synthesized knowledge from studies that used algorithmic approaches for text analysis to detect misinformation in social media. From this literature, we identified user-friendly linguistic characteristics that can assist individuals in distinguishing misinformation when they seek health-related information on social media. The linguistic characteristics, such as certainty, absolute language, and numbers, were positively associated with misinformation, while characteristics such as URLs, hashtags, and location mentions were negatively predictive of misinformation. Based on these findings, we suggested that users should be cautious of social media posts containing confident promises or specific numbers without proper references to the original information. According to our analysis, we expect that this approach will allow users to filter out two-thirds of posts with cancer-related misinformation. Yet, before drawing a definitive conclusion, further testing with different datasets is required.