ScotCET and implementation failure 3. – Treatment effect consistency in a blocking design with a demonstration in STATA

This post continues two earlier ones (this one and this one) and assesses the effect consistency across the matched pairs in the ScotCET dataset. Part of the redacted dataset can be downloaded at the bottom of this page which will allow you to carry out the same analysis presented below.

Due to the insufficient number of responses for the treatment wing of one of the matched pairs, only nine matched pairs can be analysed. In the dataset “treatment” refers to the binary treatment, “pj” to the factor score of procedural justice of the police, and “na” to the factor score of normative alignment with the police. The pre-treatment covariates include whether a breath test was conducted (“breath_test”), gender (“female”), age (“age”), marital status (“marry”), educational attainment (“ed_attain”), employment (“emp1”, “emp2”), and housing (“house1”, “house2”). Finally, the matched pairs of the road police units can be found under the variables starting with “rpu”.

First, a regression needs to be fitted for the overall model accounting for the clustering of the matched pairs with clustered robust standard errors. This is followed by fitting a regression for each of the nine matched pairs. For procedural justice the STATA code looks like this:

quietly eststo All: regress pj treatment breath_test female age marry ed_attain emp1 emp2 house1 house2, vce(cluster rpuc)
quietly eststo One: regress pj treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==1
quietly eststo Two: regress pj treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==2
quietly eststo Three: regress pj treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==3
quietly eststo Four: regress pj treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==4
quietly eststo Five: regress pj treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==5
quietly eststo Six: regress pj treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==6
quietly eststo Seven: regress pj treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==7
quietly eststo Eight: regress pj treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==8
quietly eststo Nine: regress pj treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==9

coefplot All || One || Two || Three || Four || Five || Six || Seven || Eight || Nine, ///
drop(_cons breath_test female age marry ed_attain emp1 emp2 house1 house2) xline(0) ///
xtitle(Treatment's effect on procedural justice) ytitle(Matched pairs) levels(95 67) ciopts(recast(. rcap)) ///
mlabel format(%5.0g) mlabcolor(gs0) mlabposition(12) horizontal bycoefs byopts(yrescale)

Which results in the following figure:

This figure shows that apart from the fourth, eighth, and ninth pairs’ point estimates, the rest are within the 95% confidence interval of the overall model, indicating that similar effects emerged across the different blocks. To test whether this applies to other constructs, normative alignment is presented here as well. The code mirrors to the one above:

quietly eststo All: regress na treatment breath_test female age marry ed_attain emp1 emp2 house1 house2, vce(cluster rpuc)
quietly eststo One: regress na treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==1
quietly eststo Two: regress na treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==2
quietly eststo Three: regress na treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==3
quietly eststo Four: regress na treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==4
quietly eststo Five: regress na treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==5
quietly eststo Six: regress na treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==6
quietly eststo Seven: regress na treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==7
quietly eststo Eight: regress na treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==8
quietly eststo Nine: regress na treatment breath_test female age marry ed_attain emp1 emp2 house1 house2 if rpuc==9

coefplot All || One || Two || Three || Four || Five || Six || Seven || Eight || Nine, ///
drop(_cons breath_test female age marry ed_attain emp1 emp2 house1 house2) xline(0) ///
xtitle(Treatment's effect on moral alignment) ytitle(Matched pairs) levels(95 67) ciopts(recast(. rcap)) ///
mlabel format(%5.0g) mlabcolor(gs0) mlabposition(12) horizontal bycoefs byopts(yrescale)

With the following figure:

This figure shows that most of the effects are within close range of each other, except for the third and fourth block’s which pull the overall effect size to the left. Regardless, by and large the effects are very similar across the board, most being close to zero.

Some might still find the outlier blocks as an evidence that the treatment was not consistent enough due to the issues surrounding ScotCET’s implementation. The approach taken above is very similar to one that might be taken in meta-analyses where several studies are evaluated to determine whether there is evidence in the literature of a certain effect (see for instance Weisburd et al. (2015) on broken windows policing). As with those studies, it is expected to find certain cases which appear to be outliers due to random sampling variability. In fact, those cases’ absence would be suspect and might indicate publication bias (Citkowicz and Vevea 2017). It is worth comparing the above figures to the logo of the Cochrane Collaboration which is famous for its systematic reviews of health studies. This shows that there is nothing startling about the differences between the effects found above:

In the next, final post I will evaluate the design and treatment effect heterogeneity as further tests whether the treatment effect can be attributed to the experimental design.

The dataset can be downloaded from here.

scotcet_blog

Bibliography

Citkowicz, Martyna and Jack L. Vevea. 2017. “A Parsimonious Weight Function for Modeling Publication Bias.” Psychological Methods 22(1):28–41.
Weisburd, David, Joshua C. Hinkle, Anthony a Braga, and Alese Wooditch. 2015. “Understanding the Mechanisms Underlying Broken Windows Policing: The Need for Evaluation Evidence.” Journal of Research in Crime and Delinquency 52(4):589–608.

Leave a Reply

Your email address will not be published. Required fields are marked *