Publication bias
Following Nakagawa et al. (2022), we relied on two complementary approaches to assess small study effects, which may result from publication bias. First, we visualized the relationship between effect sizes and precision (SE) using funnel plots. To do this, we re-fitted selected models as random effect models and computed residual effect sizes conditional on experiment, observation, and factor level, for factors included as moderators in the main analyses. These conditional residuals have the advantage of taking some within-experiment non-independence into account, but they still make unlikely assumptions about sampling variances (Nakagawa et al. 2022).
We, therefore, complemented funnel plots with a two-step, modified Egger’s test for multilevel meta-analysis (Nakagawa et al. 2022). In the first step of this test, SE of effect sizes is included as the only moderator in a meta-regression with the same random effect structure as our main MLMA analyses. A significant slope of this moderator means that studies with low precision tended to report either more negative or more positive effects than studies with higher precision. Therefore, if the SE slope is different from zero, the second step of the test is to fit a meta-regression with the variance of effect sizes as the only moderator. The intercept of this second meta-regression is then a more appropriate estimate of the overall meta-analytic effect (Stanley & Doucouliagos 2014). Because we uncovered evidence consistent with publication bias in Q1 and Q2, we tested the robustness of the meta-analytic effects of moderators by fitting a multi-level meta-regression (MLMR) with variance in addition to the moderators of interest for each question in our study (see Supporting Material).