Scotcet and implementation failure 4. – Treatment effect and design heterogeneity with a demonstration in R

This final post finishes the discussion of the previous three (you can find them here, here, and here) by looking at treatment effect heterogeneity and design heterogeneity.

First, it is worth discussing what these two concepts mean (for a more detailed discussion see: Gerber and Green, 2012, Chapter 9). Treatment effect heterogeneity usually refers to the potential interaction between the treatment and pre-treatment covariates. For instance, an interaction between female and the treatment would indicate that the treatment had a different impact depending on the gender of the participant. Nevertheless, finding that certain groups were more impervious to the treatment would question the effectiveness of the design and would contradict other evidence in the procedural justice literature (e.g., Mazerolle et al. 2013; Wolfe et al. 2015).

Furthermore, as discussed earlier, the current study employed blocking, which aimed to guarantee that the treated and control groups are identical with respect to the influential covariates, thus particular blocks should not influence the treatment effect either. This means that any interaction between the treatment and certain blocks would be an indication that the effect found exceeds the blocking, and that different blocks indeed received varying treatment. This is referred to as design heterogeneity.

Yet, specifying interactions between the treatment and certain covariates and/or blocks would not be very efficient. Luckily, there are ways to automate the process. Here the “FindIt” R package was used, which was developed my Imai and Ratkovic’s (2013). They recommend using their Squared Loss Support Vector Machine (L2-SVM) to assess this potential heterogeneity. For procedural justice and normative alignment the R code looks like this:

#Treatment effect heterogeneity for procedural justice
Finalpj <-FindIt(model.treat= pj ~ treatment,
model.main= ~ breath_test+female+age+marry+ed_attain+
emp1+emp2+house1+house2,
model.int= ~ breath_test+female+age+marry+ed_attain+
emp1+emp2+house1+house2,
data = scotcethet,
type="continuous",
treat.type="single",
search.lambdas=FALSE,
lambdas = c(-3.5375, -2.016))
summary(Finalpj)

#Design heterogeneity for procedural justice
FinalAllpj <-FindIt(model.treat= pj ~ treatment,
model.main= ~ rpuca+rpucb+rpucc+rpucd+rpuce+rpucf+rpucg+rpuch+rpuci+breath_test+female+age+marry+ed_attain+
emp1+emp2+house1+house2,
model.int= ~ rpuca+rpucb+rpucc+rpucd+rpuce+rpucf+rpucg+rpuch+rpuci+breath_test+female+age+marry+ed_attain+
emp1+emp2+house1+house2,
data = scotcethet,
type="continuous",
treat.type="single",
search.lambdas=FALSE,
lambdas = c(-3.5375, -2.016))
summary(FinalAllpj)

#Treatment effect heterogeneity for normative alignment
Finalna <-FindIt(model.treat= na ~ treatment,
model.main= ~ breath_test+female+age+marry+ed_attain+
emp1+emp2+house1+house2,
model.int= ~ breath_test+female+age+marry+ed_attain+
emp1+emp2+house1+house2,
data = scotcethet,
type="continuous",
treat.type="single",
search.lambdas=FALSE,
lambdas = c(-3.5025, -2.828))
summary(Finalna)

#Design heterogeneity for normative alignment
FinalAllna <-FindIt(model.treat= na ~ treatment,
model.main= ~ rpuca+rpucb+rpucc+rpucd+rpuce+rpucf+rpucg+rpuch+rpuci+breath_test+female+age+marry+ed_attain+
emp1+emp2+house1+house2,
model.int= ~ rpuca+rpucb+rpucc+rpucd+rpuce+rpucf+rpucg+rpuch+rpuci+breath_test+female+age+marry+ed_attain+
emp1+emp2+house1+house2,
data = scotcethet,
type="continuous",
treat.type="single",
search.lambdas=FALSE,
lambdas = c(-3.6495, -0.6765))
summary(FinalAllna)

No significant covariate-treatment or block-treatment interactions emerged for either procedural justice or normative alignment in either of the models. Moreover, the treatment effect didn’t show much of a change either when considering the different heterogeneities:

ATE

Design heterogeneity differences in ATEs Covariate heterogeneity differences in ATEs

Treatment-covariate interaction

Procedural justice -0.321 0.006 0.016 NA
Moral alignment -0.277 0.015 0.035 NA

The lack of heterogeneity found with regards to the treatment effect is further evidence to its robustness, which makes it very likely that it can be indeed attributed to the experimental design. It is worth pointing out that the design heterogeneity further limited the changes with regards to the treatment effect which implies that the blocking worked as intended.

References

Gerber, Alan. S. and Donald P. Green. 2012. “Field Experiments: Design, Analysis, and Interpretation.” W.W.Norton  & Company.

Imai, Kosuke and Marc Ratkovic. 2013. “Estimating Treatment Effect Heterogeneity in Randomized Program Evaluation.” Annals of Applied Statistics 7(1):443–70.

Mazerolle, Lorraine, Emma Antrobus, Sarah Bennett, and Tom R. Tyler. 2013. “Shaping Citizen Perceptions of Police Legitimacy: A Randomized Field Trial of Procedural Justice.” Criminology 51(1):33–63.

Wolfe, Scott E., Justin Nix, Robert Kaminski, and Jeff Rojek. 2015. “Is the Effect of Procedural Justice on Police Legitimacy Invariant ? Testing the Generality of Procedural Justice and Competing Antecedents of Legitimacy.” Journal of Quantitative Criminology.

Leave a Reply

Your email address will not be published. Required fields are marked *