When we debate affirmative action, it’s hard to get hard data. We can check how often identical resumes with white and black names get called back for interviews, but it’s hard to check whether a quota system changes those callback rates over time, or whether affirmative action creates a stigma against the people who are hired under the policy.

It’s pretty rare that public policies are ever tested properly (see Obama’s Innovation Centers, which “test” health policy without ever running controlled trials), but, due to a quirk in India’s election law, we have real-world data about the consequences of one case of quotas and affirmative action.

India set quotas for women’s representation for chief councillor on village councils. In other words, in a randomly selected subset of villages, only women were allowed to stand as candidates. A team of researchers conducted surveys before and after villages were governed by women and was able to isolate the effect of the quota system on villagers perception of women as leaders.

In the villages that had been forced to elect women, villagers were significantly more likely to rate women as competent leaders than in the villages that hadn’t been subject to the quota. The researchers were studying their reactions to women candidates generally, not attitudes about the specific woman leader elected, so it’s possible that individuals were still resented or felt singled out in the manner that Marjorie Romeyn-Sanabria and others describe. However, being exposed to a woman governing was enough to shift voters’ general attitude, even though she hadn’t won in a purely meritocratic election.

One reason the quota system may have produced results is that we never live in a pure meritocracy. There doesn’t need to be any malice aforethought or deliberate discrimination to derail meritocracy. After all, we’re still fumbling to find reliable measures of merit. Even rich, data-driven companies like Google have had to scrap their interview questions and GPA cutoffs when they found their measures of merit weren’t producing results. (A similar process occurs in college admissions, but given the incredible diversity of post-secondary education, and the vagueness of what a “good match” entails, I’ll stick to other domains for examples.)

When people make hiring decisions or try to choose a candidate to vote for, they’re often doing crude pattern matching: Does this person seem like the kind of person who’s succeeded in the past? In election years, this prompts a lot of bizarre analysis as we try to extend patterns (“Mitt Romney is the best nominee, because he’s taller, and taller challengers are more likely to win!”).

The same tortured analysis plays out in the business world, where Paul Graham, the head of YCombinator, a startup incubator, explained that one reason his company funds fewer women-led companies is because fewer of them fit this profile of a successful founder:

If someone was going to be really good at programming they would have found it on their own. Then if you go look at the bios of successful founders this is invariably the case, they were all hacking on computers at age 13.

The trouble is, successful founders don’t run through a pure meritocracy, either. They’re supported, mentored, and funded when they’re chosen by venture capitalists like Graham. And, if everyone is working on the same model of “good founders started at 13″ then a lot of clever ideas, created by people of either gender, might get left on the table.

Quota systems and affirmative action programs, at their best, are meant to give people a richer range of models to draw from. It’s not obvious, however, what levels of quotas should be set. Is the goal to use the quotas to force parity or just to have enough samples to give people a chance to encounter counter-examples?

The quota system in India reserved one-third of council leader positions for women, not fifty-fifty. We also don’t know whether changed attitudes persist, limiting the need for quotas, or whether they fade if, for example, the quota only applied to one election. Ultimately, the goal is not to keep substituting the judgment of the law for the judgment of the voters, but, if we’re mostly going to do quick and dirty pattern matching, to have populated our imagination with more than one image of “success” or “good fit” to draw upon.

But, because hiring teams are moving outside the models they know well, it’s hard to be motivated to take a chance or tweak a system that’s working well enough. It’s tempting to wait for other companies to take a risk and then watch for what works.

That’s how we end up with people looking for “the next Marissa Mayer” in the same way social media companies try to be “the next Facebook.” We see a new model succeed and then try to replicate it, instead of realizing that there might be a lot of other models that we’ve also been neglecting. Over the short to medium term, affirmative action and quota systems might be able to force us to refine our model for identifying merit by exposing us to new data.