Are social experiments being hyped (too much)?

Authors

DOI:

https://doi.org/10.14512/tatup.32.3.22

Keywords:

hype, social experiment, RCT, instrument constituencies, tools-to-theories heuristic

Abstract

Social experiments, also known as randomized controlled trials, are the subject of contentious discussions, giving rise to buzzwords such as ‘credibility revolution,’ ‘experimenting society,’ ‘global lab,’ or ‘empire of truth.’ While using exaggeration to illustrate opportunities and risks may well be justified, this research article analyzes to what extent the present debate is characterized by excessive hype. It finds that the transformative potential of social experiments is greatly overestimated, a judgment that applies to the reasoning of both proponents and critics.

References

ie – International Initiative for Impact Evaluation (2023): 3ie development evidence portal. Available online at https://developmentevidence.3ieimpact.org, last accessed on 14. 08. 2023.

Angrist, Joshua; Pischke, Jörn-Steffen (2010): The credibility revolution in empirical economics. How better research design is taking the con out of econometrics. In: Journal of Economic Perspectives 24 (2), pp. 3–30. https://doi.org/10.1257/jep.24.2.3 DOI: https://doi.org/10.1257/jep.24.2.3

Ball, Sarah; Head, Brian (2021): Behavioural insights teams in practice. Nudge missions and methods on trial. In: Policy & Politics 49 (1), pp. 105–120. https://doi.org/10.1332/030557320X15840777045205 DOI: https://doi.org/10.1332/030557320X15840777045205

Bédécarrats, Florent; Guérin, Isabelle; Roubaud, François (2019): All that glitters is not gold. The political economy of randomized evaluations in development. In: Development and Change 50 (3), pp. 735–762. https://doi.org/10.1111/dech.12378 DOI: https://doi.org/10.1111/dech.12378

Bothwell, Laura; Jones, David (2021): Innovation and tribulation in the history of randomized controlled trials in surgery. In: Annals of Surgery 274 (6), pp. e616–e624. https://doi.org/10.1097/SLA.0000000000003631 DOI: https://doi.org/10.1097/SLA.0000000000003631

Breslau, Daniel (1997): The political power of research methods. Knowledge regimes in U.S. labor-market policy. In: Theory and Society 26 (6), pp. 869–902. https://doi.org/10.1023/A:1006802628349 DOI: https://doi.org/10.1023/A:1006802628349

Case, Holly (2018): The age of questions. Or, a first attempt at an aggregate history of the eastern, social, woman, American, Jewish, Polish, bullion, tuberculosis, and many other questions over the nineteenth century, and beyond. Princeton, NJ: Princeton University Press. https://doi.org/10.23943/princeton/9780691131153.001.0001 DOI: https://doi.org/10.2307/j.ctt1zkjz56

Dedehayir, Ozgur; Steinert, Martin (2016): The hype cycle model. A review and future directions. In: Technological Forecasting & Social Change 108, pp. 28–41. https://doi.org/10.1016/j.techfore.2016.04.005 DOI: https://doi.org/10.1016/j.techfore.2016.04.005

Duflo, Esther; Kremer, Michael (2005): Use of randomization in the evaluation of development effectiveness. In: George Pitman, Osvaldo Feinstein and Gregory Ingram (eds.): Evaluating development effectiveness. World Bank series on evaluation and development, vol. 7. Piscataway, NJ: Transaction Publishers, pp. 205–231.

Faust, Jörg (2020): Rigorose Wirkungsevaluierung. Genese, Debatte und Nutzung in der Entwicklungszusammenarbeit. In: dms – der moderne staat – Zeitschrift für Public Policy, Recht und Management 13 (1), pp. 61–80. https://doi.org/10.3224/dms.v13i1.08 DOI: https://doi.org/10.3224/dms.v13i1.08

Fejerskov, Adam (2022): The global lab. Inequality, technology, and the experimental movement. Oxford: Oxford University Press. https://doi.org/10.1093/oso/9780198870272.001.0001 DOI: https://doi.org/10.1093/oso/9780198870272.001.0001

Fels, Katja (2022): Who nudges whom? Expert opinions on behavioural field experiments with public partners. In: Behavioural Public Policy, pp. 1–37. https://doi.org/10.1017/bpp.2022.14 DOI: https://doi.org/10.1017/bpp.2022.14

Gerber, Alan; Green, Donald; Kaplan, Edward (2014): The illusion of learning from observational research. In: Dawn Langan Teele (ed.): Field experiments and their critics. Essays on the uses and abuses of experimentation in the social sciences. New Haven, CT: Yale University Press, pp. 9–32. https://doi.org/10.12987/9780300199307-003 DOI: https://doi.org/10.12987/9780300199307-003

Gigerenzer, Gerd (1991): From tools to theories. A heuristic of discovery in cognitive psychology. In: Psychological Review 98 (2), pp. 254–267. https://doi.org/10.1037/0033-295X.98.2.254 DOI: https://doi.org/10.1037/0033-295X.98.2.254

Gisselquist, Rachel; Niño-Zarazúa, Miguel (2015): What can experiments tell us about how to improve government performance? In: Journal of Globalization and Development 6 (1), pp. 1–45. https://doi.org/10.1515/jgd-2014-0011 DOI: https://doi.org/10.1515/jgd-2014-0011

Godin, Benoît; Gingras, Yves (2002): The experimenters’ regress. From skepticism to argumentation. In: Studies in History and Philosophy of Science Part A 33 (1), pp. 133–148. https://doi.org/10.1016/S0039-3681(01)00032-2 DOI: https://doi.org/10.1016/S0039-3681(01)00032-2

Greenberg, David; Shroder, Mark; Onstott, Matthew (1999): The social experiment market. In: Journal of Economic Perspectives 13 (3), pp. 157–172. https://doi.org/10.1257/jep.13.3.157 DOI: https://doi.org/10.1257/jep.13.3.157

Intemann, Kristen (2022): Understanding the problem of ‘hype’. Exaggeration, values, and trust in science. In: Canadian Journal of Philosophy 52 (3), pp. 279–294. https://doi.org/10.1017/can.2020.45 DOI: https://doi.org/10.1017/can.2020.45

Kelly, Ann; McGoey, Linsey (2018): Facts, power and global evidence. A new empire of truth. In: Economy and Society 47 (1), pp. 1–26. https://doi.org/10.1080/03085147.2018.1457261 DOI: https://doi.org/10.1080/03085147.2018.1457261

Kvangraven, Ingrid (2019): Impoverished economics? Unpacking the economics Nobel Prize. In: openDemocracy, 18. 09. 2019. Available online at https://www.opendemocracy.net/en/oureconomy/impoverished-economics-unpacking-economics-nobel-prize, last accessed on 10. 10. 2023.

Leigh, Andrew (2018): Randomistas. How radical researchers are changing our world. New Haven, CT: Yale University Press. https://doi.org/10.12987/9780300240115 DOI: https://doi.org/10.12987/9780300240115

OECD – Organization for Economic Co-operation and Development (2021): Applying evaluation criteria thoughtfully. Paris: OECD. https://doi.org/10.1787/543e84ed-en DOI: https://doi.org/10.1787/543e84ed-en

Parkhurst, Justin (2017): The politics of evidence. From evidence-based policy to the good governance of evidence. London: Routledge. https://doi.org/10.4324/9781315675008 DOI: https://doi.org/10.4324/9781315675008

Parker, Ian (2010): The Poverty Lab. In: The New Yorker, 17. 05. 2010. Available online at https://www.newyorker.com/magazine/2010/05/17/the-poverty-lab, last accessed on 10. 10. 2023.

Picciotto, Robert (2012): Experimentalism and development evaluation. Will the bubble burst? In: Evaluation 18 (2), pp. 213–229. https://doi.org/10.1177/1356389012440915 DOI: https://doi.org/10.1177/1356389012440915

Ravallion, Martin (2009): Should the randomistas rule? In: The Economists’ Voice 6 (2), pp. 1–5. https://doi.org/10.2202/1553-3832.1368 DOI: https://doi.org/10.2202/1553-3832.1368

Simons, Arno; Voß, Jan-Peter (2018): The concept of instrument constituencies. Accounting for dynamics and practices of knowing governance. In: Policy and Society 37 (1), pp. 14–35. https://doi.org/10.1080/14494035.2017.1375248 DOI: https://doi.org/10.1080/14494035.2017.1375248

Taddese, Abeba (2021): Meeting policymakers where they are. Evidence-to-policy and practice partnership models. Washington, DC: Center for Global Development. Available online at https://www.cgdev.org/sites/default/files/meeting-policymakers-where-they-are-background-paper.pdf, last accessed on 10. 10. 2023.

Vinsel, Lee (2021): You’re doing it wrong. Notes on criticism and technology hype. In: STS-News, 01. 02. 2021. Available online at https://sts-news.medium.com/youre-doing-it-wrong-notes-on-criticism-and-technology-hype-18b08b4307e5, last accessed on 10. 10. 2023.

White, Howard (2014): Current challenges in impact evaluation. In: The European Journal of Development Research 26 (1), pp. 18–30. https://doi.org/10.1057/ejdr.2013.45 DOI: https://doi.org/10.1057/ejdr.2013.45

White, Howard (2019): The twenty-first century experimenting society. The four waves of the evidence revolution. In: Humanities & Social Sciences Communications 5 (1), pp. 1–7. https://doi.org/10.1057/s41599-019-0253-6 DOI: https://doi.org/10.1057/s41599-019-0253-6

WHO – World Health Organization (2023): Global observatory on health research and development. Number of clinical trials by year, location, disease, phase, age and sex of trial participants (1999–2022). Available online at https://www.who.int/observatories/global-observatory-on-health-research-and-development/monitoring/number-of-trial-registrations-by-year-location-disease-and-phase-of-development, last accessed on 10. 10. 2023.

Downloads

Published

2023-12-13

How to Cite

1.
Are social experiments being hyped (too much)?. TATuP [Internet]. 2023 Dec. 13 [cited 2025 Feb. 19];32(3):22-7. Available from: https://www.tatup.de/index.php/tatup/article/view/7078