ScotCET and implementation failure 5. – Treatment effect consistency in a blocking design with a demonstration in STATA 2.

I have not been blogging as diligently lately, and one of the reasons for this is that I have been a bit preoccupied with writing resubmissions. As part of this process, I had to revisit the issue of treatment effect consistency which I discussed last summer. This post discusses an alternative way of assessing and quantifying whether treatment effect (in)consistency is present, using methods adopted from meta-analysis.

Thanks to the blocking design, each matched pair in ScotCET can be considered as a separate study in the analysis, and on these “studies” a meta-analysis can be carried out. STATA’s metaan package can conduct random-effect meta-analysis which attributes a certain weight to each matched pair according to their effect size and standard error. This method also posits that these measures are all related through some distribution (Kontopantelis and Reeves 2010). Thus, and as a first step, the effect sizes and standard errors need to be derived for each matched pair, the following way:

*Create a separate dataset, here it was named "metareg"
*Set a working directory, so the "use" command could open the dataset

*Procedural justice
use scotcet_metareg, clear
statsby _b _se, by(rpuc): regress pj treatment female age ed_attain breath_test marry
merge 1:m rpuc using scotcet_metareg
egen tag = tag(rpuc) 
rename _b_treatment pjb
rename _se_treatment pjse
keep pjb pjse 
save scotcet_metareg_metaan, replace

*Normative alignment
use scotcet_metareg, clear
statsby _b _se, by(rpuc): regress na treatment female age ed_attain breath_test marry
merge 1:m rpuc using scotcet_metareg
egen tag = tag(rpuc) 
rename _b_treatment nab
rename _se_treatment nase
keep nab nase
merge 1:1 _n using scotcet_metareg_metaan
drop _merge
save scotcet_metareg_metaan, replace

These commands first opened the dataset then specified a regression and estimated the regression coefficients and standard errors for each matched pair. These were selected and saved in a separate dataset. This has been carried out for both procedural justice and normative alignment with the police. Having derived this new dataset, the random effects meta-analysis could be carried out:

*Random effects meta-analysis with Maximum Likelihood and Forest Plots
use scotcet_metareg_metaan
metaan pjb pjse, ml forest
metaan nab nase, ml forest

As with the previous case, forest plots are instructive regarding the effects. Here is the one for procedural justice:

And for normative alignment:

However, and in addition to the earlier approach, random-effects meta-analysis also derives two very useful statistics. The first is Cochran’s Q which is a statistical test for inconsistency, with the null hypothesis that all studies in the meta-analysis have the same magnitude of the effect. Another estimated measure is the I2 (I-squared), which quantifies the percentage of variation in the point estimates which can be attributed to between-study variation. Typically, values below 50% are considered as indicative of low inconsistency.

As shown by the two figures, the variation for procedural justice was 2.1% with a non-significant Cochran’s Q of χ2(8)=7.99. Similarly, for normative alignment Cochran’s Q was small (χ2(8)=10.1) and not significant, and the I2 was 8.8%. All in all, these results point to the same conclusions as before, but maybe a little more elegantly.


Kontopantelis, E., & Reeves, D. (2010). metaan: Random-effects meta-analysis. The Stata Journal, 10(3), 395–407.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.