Voodoo Neuroscience?
It would take a different mind than mine (*cough*razib*cough*) to comprehend the intricacies, but this sort of thing certainly warms my anti-phrenological heart:
The studies in question have tended to claim astonishingly high correlations between localised areas of brain activity and specific psychological measures. For example, in 2003, Naomi Eisenberger at the University of California and her colleagues published a paper purporting to show that levels of self-reported rejection correlated at r=.88 (1.0 would be a perfect correlation) with levels of activity in the anterior cingulate cortex.
According to Hal Pashler and his band of methodological whistle-blowers, if Eisenberg’s study and others like it were accurate, this "would be a milestone in understanding of brain-behaviour linkages, full of promise for potential diagnostic and therapeutic spin-offs." Unfortunately, Pashler’s group argue that the findings from many of these recent studies are virtually meaningless.
[…]
Pashler and his team found that 54 per cent of the studies had used a seriously biased method of analysis, a problem that probably also undermines the findings of fMRI studies in other fields of psychology. These researchers had identified small areas of brain activity (called voxels) that varied according to the experimental condition of interest (e.g. being rejected or not), and had then focused on just those voxels that showed a correlation, higher than a given threshold, with the psychological measure of interest (e.g. feeling rejected). Finally, they had arrived at their published brain-behaviour correlation figures by taking the average correlation from among just this select group of voxels, or in some cases just one “peak voxel”. Pashler’s team contend that by following this procedure, it would have been nearly impossible for the studies not to find a significant brain-behaviour correlation.
Here (pdf) is an ungated preprint of the Vul et al paper; and thanks to Tyler Cowen for the link. I haven’t yet read the paper myself (though I do plan to), and am once again insufficiently schooled in the relevant methodological niceties to have anything more than a cursory grasp of the force of these criticisms, but at first glance they’re hardly that surprising.
Not because neuroscientists are inclined to make stuff up, though! The use of fMRI is a tricky business: the data it yields are almost essentially messy, and the methodology for dealing with them is being worked out very much on the fly. But if the verdict of Pashler and his colleagues (“a disturbingly large, and quite prominent, segment of social neuroscience research”, they write in their concluding remarks, “is using seriously defective research methods and producing a profusion of numbers that should not be believed”) is to be believed, then the present situation in social neuroscience at least is not an especially healthy one.
Anyway, I’m going to read the whole paper and report back if anything particularly interesting jumps out. Bonus points to readers not named Razib who can offer an eighth-grade-level refresher course on what in the world an r-squared value is again.
Elsewhere: Yale Mafioso and Culture11 contributor and blogger Will Wilson and I went back and forth on the varieties of modularity during a stint at James’s old blog, here and here and here and here and here.
(Image via Flickrer Jim Lindley.)
Filed under: philosophy, science/tech
You asked for it:
http://culture11.com/blogs/postmodernconservative/2009/01/16/phrenology-and-bayesianism/
Nicely done, Will! Now, what does the p stand for?
I’ll dovetail off of Will’s proper response, though one tiny correction: r-squared is not the correlation coefficient, it’s the coefficient of determination.
R-squared, in loose terms, is the % of variation in a response variable – y, or the dependent variable – that is associated with the variation in a dependent variable – x, or the independent variable. The relationship, as Will mentions, is only an association of correlation and not causality; in statistics, nothing can be causally associated.
The most significant problem with r-squared is that, for any model, this value increases as you increase the number of independent variables in your model. So if start by testing the theory that y = a*x(1), and I don’t get the r-squared I’d like, if I say that y = a*x(1) + b*x(2), I’ll get a higher r-squared, and if I say y = a*x(1) + b*x(2) + c*x(3), I’ll get an even higher r-squared, and so on…..until forever.
That’s where the Law of Parsimony comes into effect. Or as some of you philosophers know it: Ockham’s Razor. Keep it simple, and there’s your best answer.
The p-value is related to hypothesis testing. In nearly all inferential statistical tests, a researcher is trying to test an alternatehypothesis versus a null hypothesis. The p-value is the chance that you accept your alternate hypothesis over the null hypothesis, when in fact the null hypothesis is true. This is called Type 1 error, or Alpha.
Before any inferential test is conduced, the Alpha must be selected. The most frequently chosen Alpha is 0.05; we are setting up our test to require a p-value less than this Alpha. If we get a p-value less than 0.05, we then say that our finding was significant, and we accept our alternate hypothesis over our null hypothesis.
So, if we conduct a test on two groups of people A and B, and our null is that groups A and B are equal, and our alternate is that they are unequal, if we get a p-value that is less than 0.05, there is less than a 5% chance that we are wrong and that A and B are truly equal (again, Type 1 error). This is why in most studies you see p-values over and over again, because it’s used as evidence to support a conclusion on a hypothesis.
Obviously there’s a lot of math behind this, but p-values are only as good as the hypothesis it is testing, the experiment it comes from, and don’t necessarily represent true Type 1 error. The most common error in this regard is when hypothesis tests using tools such as Student t-tests are used on non-normal data (doesn’t fit the normal bell curve). They p-value may be significant, but it probably is misleading because the researcher used the wrong formula for the type of data he/she had.
The proliferation of statistical software in the modern age has had both positive and negative effects, the most negative being that any study can produce significant p-values or nice r-squared values if you know how to manipulated the data. It’s easy to do. We have far too many individuals who are not scientists designing experiments and attempting to validate them with statistics without ever being properly trained on methodology. It’s imperative, now more than ever, that those people who do have a deeper understanding of the science behind a study and the data analysis be vigilant in their critique and peer review.
I wish we could edit our posts! Sorry for some of the typos.
Here is our invited reply
http://www.scn.ucla.edu/pdf/LiebermanBerkmanWager(invitedreply).pdf