Politics Foreign Affairs Culture Fellows Program

The Science of Policy

Innovation in business and politics depends on trial and error.
Illustration by Michael Hogue

Human beings crave certainty. Throughout history, assorted shamans, haruspices, auspices, astrologers, sibyls, kaballahists, pyromancers, Hegelians, Marxists, palmists, tarot-card readers, stock chartists, and computer modelers have made good livings off of the apparently limitless market demand for more certainty and reduced risk. But as Jim Manzi persuasively argues in his insightful and well-written new book, Uncontrolled, humanity is terrible at foresight, and trial-and-error is the chief way humans develop reliable knowledge.

Manzi begins with a telling example from the beginning of his business-consulting career. A retailer wanted to know if extensive plans to remodel its stores would result in enough profits to justify their costs. Young computer whiz Manzi crafted a complicated model taking factors like consumer research and competitive benchmarking into account and with great pride presented its output to a senior partner. The partner listened and then responded, “Okay, but why wouldn’t you just do it in a few stores and see how it works?” Manzi confesses, “This seemed so simple that I thought it couldn’t be right.” This encounter turned out to be the beginning of wisdom.

Uncontrolled presents a compact and lucid history of the development of the experimental method from its 17th-century promulgator Francis Bacon through David Hume to Karl Popper, Friedrich Hayek, and Thomas Kuhn. Experimentalists seek to identify causes by changing one possible cause while holding everything else constant and then carefully observing and measuring the results. The formalization of this seemingly simple trial-and-error procedure is a huge part of what has made the difference between modern wealth and health and Medieval poverty and plagues.

Manzi highlights the importance of the deep insight of philosopher Karl Popper that it must, in principle, be possible to falsify a theory for it to be scientific. Experiments can prove a theory false but never that it is true. While all scientific theories are conditional, they become more widely accepted as replicated experiments produce the same predicted results. “Science does not tell us whether theories are true, in the classic philosophical sense of accurately corresponding to reality, only that they are true in the sense of allowing us to make reliable nonobvious predictions,” explains Manzi. “In the end, sciences produce a body of engineering knowledge that lets us make practical predictions with tolerable reliability: an airplane of this design will fly; this vaccine will prevent smallpox; and so on.”

It is a sad fact that all too many practicing scientists will find Manzi’s claim that “the parallels between markets and science are striking” surprising. Yet is it so. “Both systems abjure absolute authority,” explains Manzi. He adds, “Both systems deploy trial-and-error learning and ruthlessly eliminate failures.” As Timothy Ferris, author of the superb book The Science of Liberty: Democracy, Reason and the Laws of Nature has put it: “Liberalism and science are methods, not ideologies.” Both science and markets advance in better understanding their subject matters by testing and falsifying asserted claims. And both often provoke populist backlashes because their psychologically counterintuitive processes and results are rejected by romanticizing reactionaries as “soulless” and “unnatural.”

Devising falsifiable experiments, while not a trivial problem, has been easier in the physical sciences than in social sciences. Even coming up with falsifiable experiments in biology, especially for therapeutic interventions, has proven harder to do. Why? Because of what Manzi calls “causal density,” in which the number and complexity of potential causes that give rise to a phenomenon increase dramatically, making it difficult even to identify all relevant contributing factors, much less to hold all but one constant.

Eventually researchers hit upon the technique of randomized controlled trials as a way to address the problem of increasing causal density. What Manzi more generally calls randomized field trials (RFTs) were developed as a way to evaluate and compare proposed therapeutic interventions. In RFTs researchers aim to measure an intervention’s effect by randomly assigning individuals to an intervention group or a control group. Any difference in outcomes between the two groups ideally represent the effect of the intervention.

“The RFT is a relatively new piece of technology—newer than the automobile or the airplane, and about the same age as color television or the electronic computer,” notes Manzi. RFTs combined with growing physical knowledge of biological pathways have helped guide researchers to many effective biomedical treatments.

Can this new technique be fruitfully applied to the sciences of human behavior? Causal density is even higher in the social and economic arenas in which policy specialists wish to intervene. As Manzi points out, “The maze of causation is now far beyond anything that physicists or biologists typically have had to address.” Consequently, social scientists try to use non-experimental methods to analyze data in an effort to unravel the tangled skein of causality.

Manzi shows that such non-experimental methods often prove less than successful. As examples, he eviscerates two widely publicized studies based on regression analyses, one claiming that since 1948 Republican presidents have increased income inequality and the other asserting that legalized abortion reduced the crime rate. Neither survived deeper scrutiny unscathed. Manzi then goes on to show that non-experimental analytical techniques also fueled various business strategy fads in the 1970s and 1980s that also did not pan out.

Then a conceptual breakthrough came two decades ago, when some entrepreneurs recognized that businesses could use experiments to test strategies in much the same way drug companies use them to evaluate therapeutics. One was the senior partner who had asked Manzi, why not experiment? In fact, that partner moved on to the credit card company Capitol One that now successfully runs thousands of experiments each year. A homely example: Would households respond at a higher rate to a solicitation in a blue envelope or a white one? Randomly mail out 50,000 of each and see what happens.

All sorts of companies now experiment in this way including Amazon, Google, and eBay. Manzi and his colleagues built their company, Applied Predictive Technologies, by providing the technology used to automate, design, and measure all sorts of business experiments. Of course, just like scientists, business experimenters need to replicate in order to validate their results.

Regarding the need for replication Manzi again gives an example how another famous one-off RFT was wildly overgeneralized, the jam experiment. One suspects that this experiment gained such wide acceptance because it flattered the confirmation biases of that class of influential intellectuals who disdain America’s consumer society. In the experiment shoppers were given a choice of tasting six jams or 24 jams and then given $1 jam-discount coupons. Considerably more people who had six jams to taste rather than 24 later purchased a jar using the coupon.

This study was popularized in the bestseller, The Paradox of Choice, which argued that consumers and citizens are overwhelmed with choices, implying that they might be better off with fewer options. Options, of course, that would no doubt be pre-selected by benevolent government officials guided by wise social scientists. But Manzi shows that subsequent randomized tests contradicted the jam experiment result and found that greater choice actually tended to increase consumption.

In reviewing the history of both physical-science and business experiments, Manzi garners two fundamental insights: innovative ideas rarely work, and when they do work they generally yield only small improvements. This holds true as well in the arena of public policy. Manzi considers three policy areas where some limited RFTs have been run: criminology, education, and social welfare. “Empirically, the vast majority of criminal justice, social welfare, and education programs fail replicated, independent well-designed RFTs,” he concludes.

In criminology only nuisance abatement of the sort described by the broken windows theory dependably reduced crime. Replicated social-welfare studies find that programs with mandatory work requirements are the only ones that reliably get people off of welfare. And the most consistent outcome of educational experiments is that student results improve, albeit marginally, if they can choose to go to a non-unionized school. One very preliminary insight that Manzi garners from the randomized policy experiment literature is that programs that focus on changing incentives and environment appear more likely to work whereas attempts to improve human behavior directly by raising skills or consciousness do not.

So we come to the heart of the book: can experimental science help identify policies whose benefits will outweigh their costs? Manzi cautiously thinks so. Manzi favors decentralization as a way to maximize the number of policy experiments. Thus he champions federalism, arguing that the federal government should grant waivers to states allowing them to experiment by changing almost any federal law or mandate—welfare eligibility, educational requirements, drug laws, etc.

States may be the laboratories of democracy, but recent history shows all too often that when one state tries out a new program, policymakers in other states or, even worse, those in Washington, D.C. rush to adopt the fad before its long-term results are in, e.g., Massachusetts’s experiment with mandatory health insurance. Still, Manzi’s stronger emphasis on federalism would likely be an improvement over the sort of one-size-fits-all policies regularly being imposed from Capitol Hill.

In keeping with his advocacy of the experimental method, Manzi proposes the creation of what would amount to a Federal Social Policy Experimentation Administration to oversee social-policy randomized experiments. He ambitiously wants to run 10,000 of them each year, too. But he acknowledges, “Naturally, Congress, presidential administrations, and everyone else with power or money at stake would attempt to manipulate the findings this agency produced.” Well, yes. I think he drastically underestimates how big a problem this would be. Already, most studies contracted by federal agencies from “independent” researchers find results that favor whatever policy the agency is promoting. There are reasons to doubt that this is just a happy coincidence.

In addition, Manzi fails to grapple with the problem that public-policy experimentation run by government does not benefit from the fierce discipline of profits and losses imposed by markets on businesses. It is a very rare thing for an agency ever to go out of business. Manzi clearly appreciates the institution of free markets as the critical arena for trial-and-error improvement of all types of technologies, products, and services. One way to improve the results of what is called public policy would be to make more of it private policy. While not all government services may be appropriately privatized, surely many would be improved by exposing them to the bracing experimentation and competition found in free markets. In any case, let’s run some RTFs and find out. Who needs a Federal Experimentation Agency if government services can be fruitfully moved to the private sector?

Manzi worries that the disruptive process of innovation fostered by free markets undermines social cohesion and produces resistance to change that can be exploited by those romanticizing populists mentioned above. Thus he favors maintaining a reformed version of the welfare state as way to buy off economic losers so that they will permit innovation to continue. “[A]s far as can be seen from history, the idea of a capitalist society without a welfare system is misplaced nostalgia—or more accurately, is an anachronism,” he asserts. But history’s verdict may not be in. The current economic crisis afflicting Western countries might more properly be thought of as part of a larger trial-and-error process indicating that the welfare state is not a viable long-term socio-economic model after all.

In Uncontrolled, Manzi provides an incisive and highly readable account of how trial-and error experimentation in science and free markets lessens human ignorance, uproots bias, and produces progress. Failure is a strict but effective instructor. Doubtless Manzi is right that deploying honest randomized trials would also improve the results of policymaking, but his sketch for how this might practically be done needs considerable fleshing out.

Ronald Bailey is the science correspondent for Reason.