We use the New Zealand data from Hess et al [7], which is a web-based stated choice survey on COVID-19 vaccines, undertaken on 20‐29 August, 2020 in the case of New Zealand. Additionally, we repeated the survey for the panel from 26 October to 18 November, 2020 (wave 2) and 23 March to 24 April, 2021 (wave 3). This element of the research design was important to be able to track changes over time. The panel was recruited using the Qualtrics survey company, and was initially representative of the New Zealand population of 18 years and older for age and gender (reweighting as needed for analysis, described at the end of this section). Qualtrics recruited into the survey from a pool of potential New Zealand-based participants who are available for the purpose of collecting representative data in web-based surveys, with a small financial incentive. We checked the survey responses across the panel, and there were no discernible signs of poor quality responses (eg, very short completion time).

The time period of the 3 waves captures a key moment for information gathering and preference formation regarding vaccines. Development and testing were being undertaken during waves 1 and 2. The Pfizer-BioNTech vaccine was approved by the United States Food and Drug Administration in December 2020, with Medsafe in New Zealand following on 3 February 2021. As the only vaccine initially offered in New Zealand, the rollout of Pfizer-BioNTech to a select group had begun by wave 3, with more advanced rollouts underway in other countries.

Additionally, over this time period, trust in the government increased and was high due to their strong lockdown policies, which were successful in eliminating community COVID-19 cases within New Zealand [30-32]. Demonstrating this high support, Prime Minister Jacinda Ardern’s Labor Party won re-election in October 2020, increasing their vote share to the highest ever for a single political party since New Zealand introduced its proportional representation electoral system in 1996, at 50% of the vote.

The main part of the survey is a stated choice survey, using a technique often called a discrete choice experiment [34]. Respondents were presented with 6 hypothetical, but realistic, choices regarding a COVID-19 vaccine. In each choice, respondents were asked to choose their preferred option out of 2 vaccines (each available as paid or free with a wait time) and no vaccine, giving them 4 vaccine choices and one no vaccine. The vaccines varied by risk of infection, serious illness, protection duration, mild or severe side effects, and population coverage. We used the Qualtrics web-based survey platform. It randomly assigned each respondent one of 6 blocks of choice sets. Respondents saw the same block of questions as initially assigned to them in each survey wave.

The choice sets were generated using a D-efficient design [35] from the NGene software package (ChoiceMetrics) [36]. They were calibrated such that an overall view of each respondent’s vaccine preferences over key vaccine attributes could be established, within credible ranges. Thus, they give us an overall view of COVID-19 vaccine preferences, rather than asking about specific vaccines that were in development. Each of the 4 question blocks is balanced such that a raw count of the number of vaccines chosen out of 6 is roughly comparable between respondents. For more details on the questions, see Hess et al [7].

Discrete, stated choice surveys are widely used in health due to their effectiveness in providing an accurate and fine-scale measure of individuals’ preferences [7,34]. While stated survey measures of sentiment, willingness, or intention to vaccinate are also valid approaches used in the literature (as covered in the introduction), the strengths of a discrete choice experiment mean it is highly suited to understanding vaccine preferences within a specific case study using hypothetical but realistic vaccine options. This is particularly important as the vaccines were still under development and their attributes were not clear. Hence, we could also not measure real vaccine uptake during this period [22].

For each wave, we sum the number of times respondent i selects a vaccine option across the 6 choice scenarios. We then classify individuals into 3 vaccine uptake categories, which we denote as resistant (vaccine chosen 0 times), hesitant (vaccine chosen 1 to 5 times), and provaccine (vaccine chosen 6 of 6 times). As noted in the previous paragraph, there may be some differences between the 4 choice blocks that could drive different counts between individuals, but these should not be significant and are unlikely to affect our classification of respondents into 3 categories (this classification approach reduces the influence of any minor measurement error across the blocks). Respondents also saw the same choice block in each wave, so differences between waves, within individuals, are not due to changes in questions.

In the New Zealand survey of Hess et al [7] in wave 3 only, we asked about the frequency of social media use, by platform, in the last 6 months. This period roughly equates with the time between the first and third survey waves, so that we can measure the association between use and changes in vaccination uptake preferences between survey waves. Platforms included are Facebook, Instagram, Twitter, and TikTok. We use dummy coding, with anyone using the platform at least once a week being coded as a user.

We asked participants about their trust in various sources for information about COVID-19 vaccines. We test their moderating effect on social media and vaccine uptake. They were measured on a 5-point Likert scale, and we coded them as dummy variables of trust (1) versus neutral or distrust (0).

We pool our data across the 3 waves, including social media use by type, a rich set of covariates, and time fixed effects. The original wave 1 sample included was representative by age and gender; however, due to attrition, the wave 3 sample needs some reweighting. We reweight for all analyses on the basis of age, gender, and ethnicity from the 2018 census [37,38]. We drop 110 observations with missing data on the variables we use in the modeling. As we collected social media use in wave 3 only, we are left with a balanced panel across the 3 waves of 257.

This project was granted ethics approval for research by the Waikato Management School Ethics Committee (application WMS 20/68). The panel was recruited using the Qualtrics survey company, and respondents were provided a small financial incentive by Qualtrics to participate in the sample. Participants provided their informed consent to participate in the study and had the option to withdraw their data at any time. All data were anonymized by Qualtrics before being sent to the research team to protect the privacy and confidentiality of our research participants.

We are interested in understanding three main points from the data as follows: (1) the association between social media use (by platform) and an individual becoming less (or more) provaccine over time; (2) the overall association of social media use (by platform) with likely vaccine uptake, and (3) the moderating influence of trust in information sources and vaccine uptake.

To understand the first point, we estimate simple Logit models for each transition between waves. This parsimonious modeling approach allows us to estimate the association between social media use and an increase/decrease in likely vaccine uptake over time, at an individual level. Our dependent variable is a binary indicator Lijk=1 for when individual i transitioned to a lower vaccine uptake group between waves j and k, and Hijk indicating if individual i transitioned to a higher vaccine uptake. We include the full set of control variables. We model the transitions separately for the different starting groups in wave j because these groups are fundamentally different from each other. For example, we model Lijk for provaccine individuals, for each wave. This allows us to understand whether social media is associated with staying provaccine or becoming less provaccine between waves j and k. If we had not separated these groups out, we would have implicitly treated transitions from being provaccine to hesitant the same as moving from hesitant to resistant.

For the latter 2 points, we estimate a PPO model [33] on the ordinal dependent variable of stated vaccine uptake of resistance through to provaccine. This model can give us an overall estimate of the association between social media platforms and vaccine uptake. However, it will not give us as clear a picture of transition over time as the Logit approach outlined above.

The PPO is a specific case of a generalized ordered logit (gologit). It allows us to take into account the ordinal nature of our outcome variable, but relax the proportional odds assumption of the more commonly used ordered logit, when needed [39,40]. The proportional odds assumption states that if we estimate a series of cumulative logit models by successively collapsing the ordinal variable into a binary variable (with different cutoffs), the odds ratios for each regressor will be equal across all models (within the limits of random error). When this assumption is violated, ordinal logit models may produce misleading and biased results [40]. Brant [41] developed a test to determine whether the proportional odds assumption holds for a set of variables. In our case, we find the proportional odds assumption to be violated for several covariates and the overall model. Hence, the PPO model allows us to relax the proportional odds assumption when needed, but keep it for the covariates where it is not violated.

An alternative model we could have estimated, which also does not require the proportional odds assumption to hold, is the multinomial logit model [39]. This model runs a series of logit regressions on every possible binary combination of the categorical variable (in our case, {1 vs 2, 1 vs 3, 2 vs 3}). However, this approach fails to account for the inherent ordering of Uit and computes a number of unnecessary parameters [40].

Using maximum likelihood, our PPO model estimates the probability that individual i in wave t is resistant, hesitant, or pro, represented by Uit∈{1,2,3}, respectively:

In our case, we can think of the PPO model as essentially 2 sets of logits, modeling the probability of being in categories 2 and 3 over 1 for the first set of coefficients (k=1), and category 3 over 1 and 2 for the second set of coefficients (k=2). Of course, both logit models are estimated simultaneously, and both are used to calculate predicted probabilities of being in any given category, as represented in the above equation. The coefficients β1k for covariates Xit vary over k. The coefficients β2 for covariates Zit do not vary over k when they do not violate the proportional odds assumption, reducing the number of coefficients to estimate. Covariates are placed into either Xit or Zit, depending on the results of Brant tests. These include time-invariant variables, such as social media use, and wave time fixed effects.

To investigate our aim (2) (stated above), we estimate a separate PPO for each social media platform, plus the full set of controls. For aim (3), we interact social media use with trust in different sources of information on vaccines (friends or family, the government, and social media).

We start by presenting summary statistics in Table 1 for the variables we use in our modeling (pooled across the 3 waves). In terms of social media use, most of the sample uses at least one social media platform out of the 4 we questioned them on (188/257, 73.2%). A majority of respondents use Facebook (177/257, 68.9%), with use of Instagram, Twitter, and TikTok being significantly lower at 30.4% (n=78), 15.2% (n=39), and 7% (n=18), respectively. Most respondents trust the government (199/257, 77.4%) and a minority trust their friends and family (84/257, 32.7%) as sources of vaccine information. As discussed in the Data section, our New Zealand sample is an informative context to study vaccine preferences, given the relatively high levels of trust in government, as COVID-19 vaccines were rolled out. Moreover, most respondents distrust social media generally (195/257, 75.9%) as a source of vaccine information. Across the 3 waves, we classify 76.1% (587/771) of responses as belonging to the provaccine uptake group, 18.3% (141/771) of responses as belonging to the vaccine-hesitant uptake group, and 5.6% (43/771) of responses as belonging to the resistant uptake group.

In Figure 1, we use a Sankey flow diagram to visually depict transitions between vaccine uptake groups across the 3 waves. Evidently, there are significant movements between vaccine uptake groups over time, particularly for the hesitant. Overall, we see movement away from hesitant by wave 3, towards provaccine and resistant. There is also a considerable proportion of respondents who remain stable in their vaccine uptake group membership across the 3 waves. Multimedia Appendix 1 contains the transition matrices.

In Table 2, we present the marginal effects at the means (MEMs) of social media use on the probability of provaccine individuals moving to a lower vaccine uptake group. These marginal effects show the change in probability for the dependent variable occurring when all other variables are set at their sample means. The first column shows the transition between waves 1 and 2, and the second column represents the transition between waves 2 and 3. Each row represents the MEMs from a model with a full set of controls, and one type of social media platform (or all social media platforms in the case of the first row). The full logit results, including controls, are in Multimedia Appendix 1.

We can see that Twitter and TikTok are associated with a significantly lower probability of provaccine individuals decreasing uptake between waves 1 and 2 (P<.01). Twitter users are 16.4 percentage points less likely to decrease uptake, and TikTok users are 15.1 percentage points less likely. The MEMs are similar in size for these 2 platforms for the waves 2 to 3 transition (however, the MEMs are only marginally significant). The MEMs are significantly stronger for the waves 2 to 3 transition for general social media, Facebook, and Instagram use. For the waves 2 to 3 transition, provaccine Facebook users are 15.6 percentage points less likely to decrease vaccine uptake (significant at the 10% level). We do not estimate this model for the vaccine-hesitant group moving to resistant, as there are too few such transitions.

We estimate the increase in uptake from hesitant in Table 3. We find statistically significant MEMs of social media use on the probability of positive changes in uptake for the waves 1 to 2 transition. Hesitant social media, Facebook, and Instagram users are 29.6, 23.4, and 45.3 percentage points more likely to increase uptake (respectively) between waves 1 and 2. We find no significant MEMs for the waves 2 to 3 transition. We do not include any other transition models due to sample size and rarity of transitions. For example, only 4 individuals went from hesitant to resistant between waves 1 and 2, making it impossible to model these transitions using our suite of covariates.

In Table 4, we present the coefficients for the full PPO results of our base models with pooled data, wave fixed effects, and no interactions. Each pair of columns shows a separate model, varying only by the social media platform dummy included in the model. The first set of columns, (1), is social media use of the specified platform. The left column, labeled 1 versus 2, 3, models being hesitant or provaccine, over being resistant. Hence, a positive coefficient indicates a more provaccine orientation (more likely to be hesitant or provaccine than resistant). The right column, labeled 1, 2 versus 3, models being provaccine, over being hesitant or resistant, with positive coefficients again indicating a more provaccine orientation. Where coefficients are missing, these have been restricted to being the same across both columns, as the proportional odds assumption is not violated. As described in the Methods section, our modeling approach allows us to relax the assumption that covariates have the same impact on preferences across different transitions. Moreover, our results in Table 4 show how social media use and a range of individual characteristics correlate with our novel vaccine preference groupings derived from discrete choice experiment data.

Looking across the models, we find that Instagram and Twitter use have a statistically significant positive relationship with vaccine uptake. On the other hand, at least one of the 4 types of social media, Facebook and TikTok use, did not have significant effects. We present the MEMs for social media of these models in Table 5. Here, we see highly statistically significant associations for Instagram and Twitter. Specifically, Instagram is associated with an 11.8 percentage point decrease in the likelihood of being resistant and a 20.3 percentage point increase in the likelihood of being provaccine. The direction is the same for Twitter, albeit with lower marginal effects. It is worth noting again that Twitter and TikTok use is relatively uncommon in our sample (Table 1).

In terms of demographic controls in Table 5, we see that income has a positive and significant association with vaccine uptake. University education, age, and gender do not have a statistically significant association with uptake level. Māori or Pacific ethnicity has a negative association with vaccine uptake, at the 5% or 10% level. Towards the bottom of Table 4, we see that the survey wave fixed effects have little predictive power on vaccine uptake. Wave 3 is associated with a lower chance of being vaccine resistant, with significance at the 10% level for 3 of the 5 models.

Next, we look at the trust variables. For the base models in Table 5, we see that the coefficient on trust in government for information about vaccines is positive and strongly statistically significant, as expected. Trust in family or friends for vaccine information has a negative but statistically insignificant relationship with uptake, except for the Instagram model, where it is negative and significant for the 1 versus 2, 3 component of the model. We exclude distrust in social media from the model as it leads to within-sample predictions of negative probabilities for a handful of observations (a drawback of PPO models [40]).

Next, we interact with trust in 2 key information sources (government, and friends and family) on vaccines, with social media type in our base PPO models. This analysis is to test whether trust in information sources has a mediating role in how social media use is associated with vaccine uptake.

We might expect that social media use will be associated with a higher likelihood of vaccine uptake for those who trust the government for vaccine information (compared with not trusting), given the use of social media by the government to inform the public about COVID restrictions. Finally, we expect social media users will be less likely to take the vaccine if they trust their friends or family for vaccine information (compared with not trusting), given that friends or family may share vaccine information on social media and are a less reliable source of information. To test these hypotheses, we follow the same approach as before and re-estimate the models from Table 5, with just the interaction of interest. We present the MEMs for these interactions in Table 6.

Let us first consider general social media use and trust in government, in the top left of the table. We see that social media use for those who trust the government is associated with a 4.7% lower probability of being resistant, which is statistically significant at the 5% level. However, none of the other marginal effects is significant, nor is the difference between the marginal effects for those who trust the government for vaccine information and those who do not. There is a similar result for Facebook users.

Instagram, Twitter, and TikTok users have higher probabilities of being pro and lower probabilities of being hesitant or resistant, for both individuals who trust the government and those who do not. The MEMs of social media use on those who trust the government and those who do not are counterintuitive. For example, Twitter use is associated with a 11.7 percentage point decrease in the probability of being resistant, for those who do not trust the government. On the other hand, Twitter use is associated with only a 3.9 percentage point decrease in being resistant for those who do trust the government. This is a statistically significant difference. Instagram and TikTok use shows largely similar patterns to the Twitter example mentioned here. Thus, Twitter, Instagram, and TikTok use is associated more positively with vaccine uptake for those who do not trust the government, compared with those who do trust the government.

The right side of Table 6 shows the MEMs for interactions between social media use and trust in friends and family. We might expect those who trust friends and family for vaccine information to be more susceptible to misinformation on social media. We see this for social media users in general in the top rows; social media users who trust friends and family are more likely to be resistant and hesitant, and less likely to be pro, at the 10% level. This finding is true for Facebook users as well. There are no significant differences for Instagram or TikTok users. However, we see the opposite for Twitter users. Twitter users who trust their friends and family are more likely to be pro and less likely to be hesitant compared with those who do not trust their friends and family, again at the 10% level of significance.