Empirical results can be very dependent on model specification. How can analysts show model robustness in their research?

New working paper:

“Functional Form Robustness: Advancements in Multiverse Analysis.” With Sheridan Stewart

*

“We Ran 9 Billion Regressions: Eliminating False Positives Through Computational Model Robustness.” 2018. With John Muñoz. Sociological Methodology. Vol 48(1): 1-33.

Symposium article, with commentary by Bruce Western and Robert O’Brien.

Download the replication package

“Rejoinder: Can We Weight Models by Their Probability of Being True?” 2018. With John Muñoz. Sociological Methodology. Vol 48(1): 43–51.

“Model Uncertainty and the Crisis in Science.” 2018. Socius.

 

 Model Uncertainty and Robustness: A Computational Framework for Multi-Model Analysis.  Sociological Methods and Research. February 2017. Vol. 46(1): 3 – 40. (Lead article, with Katherine Holsteen)

Download: This do file installs our Stata program, loads in data sets, and replicates all the analyses in the paper.

 

OR: type “ssc describe multivrs” in Stata

 

 

The multiverse framework allows researchers to

(1) estimate thousands of regression models across combinations of control variables, functional forms, alternative variable definitions, and estimation commands;

(2) display the resulting modeling distribution of estimates; and

(3) discover which aspects of model specification have the most influence on an estimate of interest.

This allows analysts to clarify and demonstrate which modeling assumptions are essential to their empirical findings, and which are not.

 

This is part of an enduring theme in my research, focusing on robust results. My earlier research:

Model Uncertainty in Sociological Research: An Application to Religion and Economic Growth. American Sociological Review. June 2009.

 

New study, applying model robustness methods: Patients as Consumers in the Market for Medicine: The Halo Effect of Hospitality (with Xinxiang Chen)