Author
|
Topic: High Error Rates Found In Published Scientific Research (Read 622 times) |
|
Blunderov
Archon
Gender: 
Posts: 3160
Reputation: 8.89
Rate Blunderov
"We think in generalities, we live in details"
  
|
 |
High Error Rates Found In Published Scientific Research
« on: 2007-09-16 05:49:46 » |
 |
[Blunderov] First the stunning revelation that 38% of people are not actually entitled to their opinion:
http://www.churchofvirus.org/bbs/index.php?board=33;action=display;threadid=39643
and now the disclosure that neither are an even greater percentage of scientists either.
More seriously, the problem seems to be with the peer review system which is probably insufficiently motivated to undertake the work with due diligence.
This is a big, big problem. Academics need to be able to trust, at least on a provisional basis, the work of their predecessors and contemporaries if there is not to be a great deal of time wasted in misbegotten and futile projects.
(Interesting that the research should emanate from an epidemiologist. When I first scanned the report I misread the worde as "epistemologist".  )
http://www.futurepundit.com/archives/004589.html
September 15, 2007
High Error Rates Found In Published Scientific Research
Epidemiologist John Ioannidis says most claimed research findings are wrong due to mistakes made by researchers in design and in analysis of results.
Dr. Ioannidis is an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Medford, Mass. In a series of influential analytical reports, he has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye.
These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."
The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.
Should government agencies that hand out research grants hand out parallel grants to independent groups to do parallel analyzes of data produced by funded researchers to find errors?
I suspect peer reviewers lack the time, data, and incentives needed to catch errors in research papers. Given the relative cheapness of data storage and data transmission technologies at least for some types of research independent parties should be given a crack at analyzing the same data to see if the results drawn by original researchers are warranted.
Another thought: Could some types of studies have agreed standard data formats with standard analyzes written ahead of time so that different studies could be compared automatically and so that bias would not enter as much into the analysis phase?
Ioannidis published a paper in Plos Medicine in August 2005 entitled Why Most Published Research Findings Are False.
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Step up a level and use the kind of thinking a business executive steeped in the wisdom of W. Edwards Deming would bring to a manufacturing quality problem: Huge amounts of labor goes into conducting scientific studies and therefore a high error rate represents a huge amount of waste of labor and supplies. Science needs more automation and other process improvements to raise quality control and reduce waste.
By Randall Parker at 2007 September 15 05:07 PM Policy Science | TrackBack
|
|
|
|
|
Hermit
Archon
Posts: 4287
Reputation: 8.93
Rate Hermit
Prime example of a practically perfect person
|
|
Re:High Error Rates Found In Published Scientific Research
« Reply #1 on: 2007-09-17 00:23:10 » |
|
It might be far more cost effective to offer grants covering only program costs, and offer prizes for papers adjudicated as valid and valuable by a TV/Web game show audience, openly available papers being chosen as potential winners by the web audience, where the necessary logical and judging skills are taught by peers in the first 80% of each program, with the judging taking 10 %, followed by a presentation of the value of the work to the same audience who might then base a prize value on the research's perceived value to society by society in the last 10%. Having a peer ready and prepared to spending the last 10% explaining why a paper didn't receive a prize might be beneficial to everyone involved and watching too.
I can imagine many academics screaming heresy, but the wisdom of large groups is well documented, and the fringe benefits of educating society could be great. The overall quality and applicability of research might be bumped up a notch too.
Now grant applications and papers not making the cut to the TV program could be adjudicated by audience members who had voted in say 6 of the previous 9 of the TV shows. Just imagine, a qualified franchise, Wikipedia like, cafeteria funding system. Whatever next :-)
Kindest Regards
Hermit
|
With or without religion, you would have good people doing good things and evil people doing evil things. But for good people to do evil things, that takes religion. - Steven Weinberg, 1999
|
|
|
|
Logged
