[LINK] Fraud Case Seen as a Red Flag for Psychology Research

An article in the NYT's about everyone's favourite messy science, you know the one we sometimes rely on to provide a throwaway line as we pontificate wisely about biases? ;)

A well-known psychologist in the Netherlands whose work has been published widely in professional journals falsified data and made up entire experiments, an investigating committee has found. Experts say the case exposes deep flaws in the way science is done in a field, psychology, that has only recently earned a fragile respectability.

The psychologist, Diederik Stapel, of Tilburg University, committed academic fraud in “several dozen” published papers, many accepted in respected journals and reported in the news media, according to a report released on Monday by the three Dutch institutions where he has worked ...

In recent years, psychologists have reported a raft of findings on race biases, brain imaging and even extrasensory perception that have not stood up to scrutiny. Outright fraud may be rare, these experts say, but they contend that Dr. Stapel took advantage of a system that allows researchers to operate in near secrecy and massage data to find what they want to find, without much fear of being challenged. ...

In a prolific career, Dr. Stapel published papers on the effect of power on hypocrisy, on racial stereotyping and on how advertisements affect how people view themselves. Many of his findings appeared in newspapers around the world, including The New York Times, which reported in December on his study about advertising and identity.

In a statement posted Monday on Tilburg University’s Web site, Dr. Stapel apologized to his colleagues. “I have failed as a scientist and researcher,” it read, in part. “I feel ashamed for it and have great regret.” ...

Dr. Stapel has published about 150 papers, many of which, like the advertising study, seem devised to make a splash in the media. The study published in Science this year claimed that white people became more likely to “stereotype and discriminate” against black people when they were in a messy environment, versus an organized one. Another study, published in 2009, claimed that people judged job applicants as more competent if they had a male voice. The investigating committee did not post a list of papers that it had found fraudulent. ...

In a survey of more than 2,000 American psychologists scheduled to be published this year, Leslie John of Harvard Business School and two colleagues found that 70 percent had acknowledged, anonymously, to cutting some corners in reporting data. About a third said they had reported an unexpected finding as predicted from the start, and about 1 percent admitted to falsifying data.

Also common is a self-serving statistical sloppiness. In an analysis published this year, Dr. Wicherts and Marjan Bakker, also at the University of Amsterdam, searched a random sample of 281 psychology papers for statistical errors. They found that about half of the papers in high-end journals contained some statistical error, and that about 15 percent of all papers had at least one error that changed a reported finding — almost always in opposition to the authors’ hypothesis.

...

found that the more reluctant that scientists were to share their data, the more likely that evidence contradicted their reported findings.

...
“We know the general tendency of humans to draw the conclusions they want to draw — there’s a different threshold,” said Joseph P. Simmons, a psychologist at the University of Pennsylvania’s Wharton School. “With findings we want to see, we ask, ‘Can I believe this?’ With those we don’t, we ask, ‘Must I believe this?’

But reviewers working for psychology journals rarely take this into account in any rigorous way. Neither do they typically ask to see the original data. While many psychologists shade and spin, Dr. Stapel went ahead and drew any conclusion he wanted.

In any case this brought to my attention by a recent blog entry on iSteve.

Telling people what they want to hear

Steve Sailer thinks that what gets distorted the most in such a way is a matter of supply and demand. Which is obviously good signalling for him, but is also eminently plausible. One can't help but wonder especially on the interesting connections that exist between some of the "findings" of psychology of a certain period and place the obsessions and neurosis (heh) specific to that society.

An article in the NYT's about everyone's favourite messy science, you know the one we sometimes rely on to provide a throwaway line as we pontificate wisely about biases? ;)

A well-known psychologist in the Netherlands whose work has been published widely in professional journals falsified data and made up entire experiments, an investigating committee has found. Experts say the case exposes deep flaws in the way science is done in a field, psychology, that has only recently earned a fragile respectability.

The psychologist, Diederik Stapel, of Tilburg University, committed academic fraud in “several dozen” published papers, many accepted in respected journals and reported in the news media, according to a report released on Monday by the three Dutch institutions where he has worked ...

In recent years, psychologists have reported a raft of findings on race biases, brain imaging and even extrasensory perception that have not stood up to scrutiny. Outright fraud may be rare, these experts say, but they contend that Dr. Stapel took advantage of a system that allows researchers to operate in near secrecy and massage data to find what they want to find, without much fear of being challenged. ...

In a prolific career, Dr. Stapel published papers on the effect of power on hypocrisy, on racial stereotyping and on how advertisements affect how people view themselves. Many of his findings appeared in newspapers around the world, including The New York Times, which reported in December on his study about advertising and identity.

In a statement posted Monday on Tilburg University’s Web site, Dr. Stapel apologized to his colleagues. “I have failed as a scientist and researcher,” it read, in part. “I feel ashamed for it and have great regret.” ...

Dr. Stapel has published about 150 papers, many of which, like the advertising study, seem devised to make a splash in the media. The study published in Science this year claimed that white people became more likely to “stereotype and discriminate” against black people when they were in a messy environment, versus an organized one. Another study, published in 2009, claimed that people judged job applicants as more competent if they had a male voice. The investigating committee did not post a list of papers that it had found fraudulent. ...

In a survey of more than 2,000 American psychologists scheduled to be published this year, Leslie John of Harvard Business School and two colleagues found that 70 percent had acknowledged, anonymously, to cutting some corners in reporting data. About a third said they had reported an unexpected finding as predicted from the start, and about 1 percent admitted to falsifying data.

Also common is a self-serving statistical sloppiness. In an analysis published this year, Dr. Wicherts and Marjan Bakker, also at the University of Amsterdam, searched a random sample of 281 psychology papers for statistical errors. They found that about half of the papers in high-end journals contained some statistical error, and that about 15 percent of all papers had at least one error that changed a reported finding — almost always in opposition to the authors’ hypothesis.

...

found that the more reluctant that scientists were to share their data, the more likely that evidence contradicted their reported findings.

...
“We know the general tendency of humans to draw the conclusions they want to draw — there’s a different threshold,” said Joseph P. Simmons, a psychologist at the University of Pennsylvania’s Wharton School. “With findings we want to see, we ask, ‘Can I believe this?’ With those we don’t, we ask, ‘Must I believe this?’

But reviewers working for psychology journals rarely take this into account in any rigorous way. Neither do they typically ask to see the original data. While many psychologists shade and spin, Dr. Stapel went ahead and drew any conclusion he wanted.

In any case this brought to my attention by a recent blog entry on iSteve.

Telling people what they want to hear

found that the more reluctant that scientists were to share their data, the more likely that evidence contradicted their reported findings.

See also: climategate.

The article discusses the refusal of the University of East Anglia to release ”CRUTEM” data under the Freedom of Information Act – on this occasion the UK Information Commissioner ruled that the university would have to comply with requests to share its data:

Quote:

As a first comment on the University’s defence – in keeping with similar refusals of other requests, rather than focusing on their best line of argument,the practice of the UEA is to use a laundry list of exemptions – more or less throwing spitballs against the wall to see if any of them stuck. Many of the spitballs seem pretty strained, to say the least. In his ruling, the ICO picked each spitball off the wall and, in the process, established or confirmed a number of precedents that will hopefully encourage fewer spitballs in the future.

They attempted to use the following exemptions: s 6 – “information already publicly available” s 12(5)(a) – would have an adverse effect on “international relations, defence, national security or public safety” s 12(5)(c) – would have an adverse effect on “intellectual property rights” s 12(5)(f) – would have an adverse effect on the interests of the person who provided the information

As if it wasn’t already clear that this kind of behaviour (exhibited on a consistent basis) constitutes scientific misconduct, Wicherts and Bakker’s findings provide evidence that it should be so regarded.

Another interesting nugget I read about on Steve Sailer’s site – again referring to the research of Wicherts – concerned “stereotype threat”.

Quotes:

“In 1995, two Stanford psychologists, Claude Steele and Joshua Aronson, demonstrated that African-American college students did worse on tests of academic ability when they were exposed beforehand to suggestions that they were being judged according to their race. Steele and Aronson hypothesized that this effect, which they labeled stereotype threat, might explain part of the persistent achievement gap between white and black students. In the years since, this idea has spread throughout the social sciences.” [...]

A researcher, who doesn’t want his name or any potentially identifying information mentioned, for unfortunately obvious career reasons, recently attended a presentation at a scientific conference. Here is his summary of what he heard:

“One talk presented a meta-analysis of stereotype threat. The presenter was able to find a ton of unpublished studies. The overall conclusion is that stereotype threat does not exist. The unpublished and published studies were compared on many indices of quality, including sample size, and the only variable predicting publication was whether a significant effect of stereotype threat was found. …

“This is quite embarrassing for psychology as a science.” [...]

A meta-analysis of 55 published and unpublished studies of this effect shows clear signs of publication bias.”

Interestingly there are many credulous references to “stereotype threat” on lesswrong, but seemingly no skeptical postures (until now).

I have been aware of the case against stereotype threat for some time, but wouldn't want to post it under my regular handle (linked to my real name and high-karma). People who try to approach these topics in an even-handed way often lose their jobs or suffer other serious consequences. Mostly I just try to let the discussion die down, since those who might present non-PC evidence are laboring under differential burdens and constraints.

3NancyLebovitz15y

This is interesting, but not exactly what I'd call public evidence.

5Manfred15y

Interesting. So, say, NASA, which shares its data liberally, has gotten its conclusions right, while CRU, which resisted doing so, has gotten them wrong?

18

[LINK] Fraud Case Seen as a Red Flag for Psychology Research

18

18

18

[LINK] Fraud Case Seen as a Red Flag for Psychology Research

18

18