Wednesday, July 09, 2014

shouting into the wind

​I usually like io9, but this was pretty stupid, and yet it got a big response. It's a blurb on an essay, "On the Emptiness of Failed Replications", by Jason Mitchell, a social psychologist/neuroscientist, where he's criticizing some of the current discussion in the social sciences (and peripheral quarters). I've found other severely critical discussions of the essay, and none positive, although the negatives are (it seems) already entrenched in their opinions about how things *must* be done. Given my habitual uncertainty about everything, I think my take is fairly objective.
I haven't read most of the comments on the io9 article (just scanned the first page's worth), but it seems they are mostly agreeing with the negative post, and most of them don't discuss the essay *at all*. A lot of them are just bland 'social science is pseudoscience' stuff, but they're letting this blurb (assuming most of them didn't *read* the essay) feed their preconceptions. The comments on other blog posts are similar, though the quality of the posts themselves are generally better (if pedantic). I'm going to do some apologia here, since I'm avoiding working on a paper (or papers) of my own.
Mitchell's writing about the replication push, and explaining why, in basic philosophy-of-science terms (i.e. in terms of falsificationism - cf. previous post) replication as standard practice is not *scientific* practice. **Searching for null effects is meaningless**, he says, because there are vastly more ways to do an experiment wrong, and fail to detect an effect, than there are ways to do it properly. The scientific way to challenge a finding is not to try to reproduce it by following the steps in a methods section - it's by finding out how such a finding *could* arise, and explaining that in functional terms. The worst case is that the finding arises through fraud, and Mitchell goes pretty far in pointing out how a replication push can take the form of a witch-hunt.
Of course many science-minded folk are biased against the social sciences, in part because there have been a number of prominent frauds recently, but also because the methods are hard to discern. Social psychology, and most of experimental psychology at that, are different from the 'hard' sciences in important ways, but they are still scientific practices. It's not pseudoscience to say that you can measure a person's thoughts or perceptions or feelings or predispositions, although the measurements can be done well or done poorly.
There is one fundamental difference between the social sciences and the hard sciences (physics, chemistry, biology): in studying a human mind, you have to communicate with it, and every human mind is different - even a single human mind is different from day-to-day. Running a psychophysics or cognitive experiment effectively requires that you take this person, quickly figure out the contours of their personality (in a quick discussion or screening interview), and set their thought processes in such a way that you can *then do the experiment*. And furthermore, especially when it comes to in-depth studies like in real psychophysics, some people simply can't do the experiments; there are good psychophysics subjects and bad psychophysics subjects (in my experience, you might lump these into "people who are conscientious and have a high capacity for introspection", and "people who don't listen and who get bored by themselves"), and part of doing these experiments well is weeding out the bad ones. Usually it's pretty easy - a subject does some training blocks that produce crummy data - you try to explain to them how to improve their performance; they fail to improve, and you fire them. The foregoing steps are an art, the practice of communication and guidance and control, and some people are better at it than others. This is a relatively minor point in the essay, which is focused on replication, but it seems to have caught a lot of attention from the 'social science isn't science' folks.
I say all this as a social scientist, under general definitions. I'm a psychophysicist, a neuroscientist, and an experimental psychologist - most of my research is quantitative and computational and model-driven, with human beings (their behavior or their brains) as the source of my data. I completely recognize the situation that Mitchell is describing - I've had to explain these issues to juniors and colleagues many times over the years, though I don't know if I'm always convincing. I thought it was a great essay, and worth passing on. And, at the end of the day, I check the box 'psychologist', which puts me in the camp under attack, and so I feel I need to raise my tiny voice in defense of Dr Mitchell. Not that anyone comes here to read what I write, but I like to remember things, so.. meh.

No comments:

Post a Comment