I’ve been thinking a lot about college admissions since the publication of Ron Unz’s blockbuster critique of admissions process at Harvard. I’m convinced by Unz’s argument that Harvard’s standards are not strictly meritocratic, and systematically disfavor Asian-American applicants. I’m less convinced that college admissions should be strictly meritocratic–or even that we can imagine what that would mean.

The trouble is that “merit” can plausibly be understood in a number of ways, including past academic achievement, raw cognitive ability, talent in a particular field, unusual determination or work ethic, and chance to make it big after graduating. These qualities don’t necessarily overlap and are sometimes opposed. In my experience, for example, the most intellectually curious high school students often shine in their areas of interest, but just scrape by in other fields. On the other hand, class-president types can accumulate impressive resumes while remaining deadly dull.

So I have more sympathy for admissions officers than most of their conservative critics.  Even if we ignore non-academic interests in admitting trombone players, long-distance runners, and so on, the job of selecting the “best” 2000 or so applicants from a pool of over 30,000 is a thankless and probably impossible task. That’s why I’ve defended proposals for a lottery-based system, in which every student who meets a certain standard of competence is entered into a drawing for the appropriate number of winners. Over time, partly-randomized admissions should generate the diversity of abilities and interests that American colleges want while avoiding controversial judgments of merit.

But a lottery system won’t happen any time soon, partly because it threatens the myth that graduates of a fancy college who go on to run things deserve their success. In the meantime, is there any way to reduce the subjectivity involved in taking “soft” factors into account?

The Chronicle of Higher Education reports that some psychologists and administrators are developing assessments of “non-cognitive” or “metacognitive” abilities, including determination and finding creative solutions to problems. The idea is that there’s more to being a good student than just being smart enough–and that colleges would have more success in learning outcomes and retention if they could identify which applicants have the discipline and grit to succeed.

This is not a terrible idea in itself. Graduate schools routinely ask recommenders to provide evaluations of applicants’ abilities in areas that are not strictly academic. What’s more, the use of the new assessments appears to be limited to marginal cases. In other words, they identify students with borderline grades but other strengths. Finally, the assessments, which involve short answers to focused questions, provide a welcome alternative to the traditional admissions essay. Although I’ve never worked in an admissions office, I’ve read and written my share–and I’m prepared to assert that they are the most vapid literary form ever conceived.

But disadvantages of metacognitive assessment outweigh its advantages, at least in the version discussed at greatest length in the piece (the “Insight Resume” used by Oregon State University). Here are some of the problems:

1. The responses are based on self-reporting rather than evaluation by professionals (as in graduate admissions). As such, they provide information mostly about what students think of themselves rather than what they can do. This information isn’t totally useless: applicants may have something significant to report that doesn’t fit elsewhere in the application or is unknown to their teachers, bosses, or other experienced observers. But there’s no way of verifying whether it’s true or judging how it compares to the performance of students in similar positions.

2.  As far as I can tell, the actual assessment of applicants’ responses remains arbitrary. Unlike traditional psychological testing (such as the Rorschach), in other words, there doesn’t seem to be a standardized rubric for evaluating the answers. So “assessment” here really means the personal judgement of admissions officers. And that’s precisely the factor that external critics of the admissions system usually want to limit.

3. The specific questions in the Insight Resume, which are based on variables articulated by the education professor William E. Sedlacek, reflect the politically correct fascinations with discrimination and community service. The use of these questions suggests that metacognitive assessment may be used as a cover for racial preferences, which the Supreme Court may soon declare illegal at public universities.

Although they’re still under development, then, metacognitive assessments are unlikely to help resolve the dilemmas of meritocracy. The truth is, most of us want admissions decisions to be based on hard data and real achievements–except when we don’t. If a lottery system seems too radical, what about bringing back old-fashioned admissions tests, which tested mastery of disciplinary content? Such tests would favor students who happen to attend better schools, generally through no merit of their own. But at least we’d be confident that those who aced them already know something, rather merely than hoping they possess the potential to learn.

Follow Samuel Goldman on Twitter.