I have not been blogging as diligently lately, and one of the reasons for this is that I have been a bit preoccupied with writing resubmissions. As part of this process, I had to revisit the issue of treatment effect consistency which I discussed last summer. This post discusses an alternative way of assessing and quantifying whether treatment effect (in)consistency is present, using methods adopted from meta-analysis.
Thanks to the blocking design, each matched pair in ScotCET can be considered as a separate study in the analysis, and on these “studies” a meta-analysis can be carried out. STATA’s metaan package can conduct random-effect meta-analysis which attributes a certain weight to each matched pair according to their effect size and standard error. This method also posits that these measures are all related through some distribution (Kontopantelis and Reeves 2010). Thus, and as a first step, the effect sizes and standard errors need to be derived for each matched pair, the following way:
*Create a separate dataset, here it was named "metareg" *Set a working directory, so the "use" command could open the dataset *Procedural justice use scotcet_metareg, clear statsby _b _se, by(rpuc): regress pj treatment female age ed_attain breath_test marry merge 1:m rpuc using scotcet_metareg egen tag = tag(rpuc) rename _b_treatment pjb rename _se_treatment pjse keep pjb pjse save scotcet_metareg_metaan, replace *Normative alignment use scotcet_metareg, clear statsby _b _se, by(rpuc): regress na treatment female age ed_attain breath_test marry merge 1:m rpuc using scotcet_metareg egen tag = tag(rpuc) rename _b_treatment nab rename _se_treatment nase keep nab nase merge 1:1 _n using scotcet_metareg_metaan drop _merge save scotcet_metareg_metaan, replace
These commands first opened the dataset then specified a regression and estimated the regression coefficients and standard errors for each matched pair. These were selected and saved in a separate dataset. This has been carried out for both procedural justice and normative alignment with the police. Having derived this new dataset, the random effects meta-analysis could be carried out:
*Random effects meta-analysis with Maximum Likelihood and Forest Plots use scotcet_metareg_metaan metaan pjb pjse, ml forest metaan nab nase, ml forest
As with the previous case, forest plots are instructive regarding the effects. Here is the one for procedural justice:
And for normative alignment:
However, and in addition to the earlier approach, random-effects meta-analysis also derives two very useful statistics. The first is Cochran’s Q which is a statistical test for inconsistency, with the null hypothesis that all studies in the meta-analysis have the same magnitude of the effect. Another estimated measure is the I2 (I-squared), which quantifies the percentage of variation in the point estimates which can be attributed to between-study variation. Typically, values below 50% are considered as indicative of low inconsistency.
As shown by the two figures, the variation for procedural justice was 2.1% with a non-significant Cochran’s Q of χ2(8)=7.99. Similarly, for normative alignment Cochran’s Q was small (χ2(8)=10.1) and not significant, and the I2 was 8.8%. All in all, these results point to the same conclusions as before, but maybe a little more elegantly.
Kontopantelis, E., & Reeves, D. (2010). metaan: Random-effects meta-analysis. The Stata Journal, 10(3), 395–407.