How reliable is "science"

You've just found the penis-shaped door to freedom. GET ON YOUR FUCKING FEET. Turn the tables on your masters. Light the entire world on fire. The time for sitting there like a little bitch is OVER.
Forum rules
This section is open to the public. Feel free to post questions, criticisms or comments. Thank you.
Post Reply
User avatar
Info
Dean of Beatdowns
Posts: 10144
Joined: Sat May 15, 2010 10:34 am
Contact:

Re: How reliable is "science"

Post by Info » Sun Sep 29, 2013 8:36 am

Widespread Academic Fraud In China
A flawed system for judging research is leading to academic fraud
Sep 28th 2013

DISGUISED as employees of a gas company, a team of policemen burst into a flat in Beijing on September 1st. Two suspects inside panicked and tossed a plastic bag full of money out of a 15th-floor window. Red hundred-yuan notes worth as much as $50,000 fluttered to the pavement below.

Money raining down on pedestrians was not as bizarre, however, as the racket behind it. China is known for its pirated DVDs and fake designer gear, but these criminals were producing something more intellectual: fake scholarly articles which they sold to academics, and counterfeit versions of existing medical journals in which they sold publication slots.

As China tries to take its seat at the top table of global academia, the criminal underworld has seized on a feature in its research system: the fact that research grants and promotions are awarded on the basis of the number of articles published, not on the quality of the original research. This has fostered an industry of plagiarism, invented research and fake journals that Wuhan University estimated in 2009 was worth $150m, a fivefold increase on just two years earlier.

Chinese scientists are still rewarded for doing good research, and the number of high-quality researchers is increasing. Scientists all round the world also commit fraud. But the Chinese evaluation system is particularly susceptible to it.

By volume the output of Chinese science is impressive. Mainland Chinese researchers have published a steadily increasing share of scientific papers in journals included in the prestigious Science Citation Index (SCI—maintained by Thomson Reuters, a publisher). The number grew from a negligible share in 2001 to 9.5% in 2011, second in the world to America, according to a report published by the Institute of Scientific and Technical Information of China. From 2002 to 2012, more than 1m Chinese papers were published in SCI journals; they ranked sixth for the number of times cited by others. Nature, a science journal, reported that in 2012 the number of papers from China in the journal’s 18 affiliated research publications rose by 35% from 2011. The journal said this “adds to the growing body of evidence that China is fast becoming a global leader in scientific publishing and scientific research”.

In 2010, however, Nature had also noted rising concerns about fraud in Chinese research, reporting that in one Chinese government survey, a third of more than 6,000 scientific researchers at six leading institutions admitted to plagiarism, falsification or fabrication. The details of the survey have not been publicly released, making it difficult to compare the results fairly with Western surveys, which have also found that one-third of scientists admit to dishonesty under the broadest definition, but that a far smaller percentage (2% on average) admit to having fabricated or falsified research results.

In 2012 Proceedings of the National Academy of Sciences, an American journal, published a study of retractions accounting for nation of origin. In it a team of authors wrote that in medical journal articles in PubMed, an American database maintained by the National Institutes of Health, there were more retractions due to plagiarism from China and India together than from America (which produced the most papers by far, and so the most cheating overall). The study also found that papers from China led the world in retractions due to duplication—the same papers being published in multiple journals. On retractions due to fraud, China ranked fourth, behind America, Germany and Japan.

“Stupid Chinese Idea”
Chinese scientists have urged their comrades to live up to the nation’s great history. “Academic corruption is gradually eroding the marvellous and well-established culture that our ancestors left for us 5,000 years ago,” wrote Lin Songqing of the Chinese Academy of Sciences, in an article this year in Learned Publishing, a British-based journal.

In the 1980s, when China was only beginning to reinvest in science, amassing publishing credits seemed a good way to use non-political criteria for evaluating researchers. But today the statistics-driven standards for promotion (even when they are not handed out merely on the basis of personal connections) are as problematic as in the rest of the bureaucracy. Xiong Bingqi of the 21st Century Education Research Institute calls it the “GDPism of education”. Local government officials stand out with good statistics, says Mr Xiong. “It is the same with universities.”

The most valuable statistic a scientist can tally up is SCI journal credits, especially in journals with higher "impact factors"—ones that are cited more frequently in other scholars’ papers. SCI credits and impact factors are used to judge candidates for doctorates, promotions, research grants and pay bonuses. Some ambitious professors amass SCI credits at an astounding pace. Mr Lin writes that a professor at Ningbo university, in south-east China, published 82 such papers in a three-year span. A hint of the relative weakness of these papers is found in the fact that China ranks just 14th in average citations per SCI paper, suggesting that many Chinese papers are rarely quoted by other scholars.

The quality of research is not always an issue for those evaluating promotions and grants. Some administrators are unqualified to evaluate research, Chinese scientists say, either because they are bureaucrats or because they were promoted using the same criteria themselves. In addition, the administrators’ institutions are evaluated on their publication rankings, so university presidents and department heads place a priority on publishing, especially for SCI credits. This dynamic has led some in science circles to joke that SCI stands for “Stupid Chinese Idea”.

Crystal unclear
The warped incentive system has created some big embarrassments. In 2009 Acta Crystallographica Section E, a British journal on crystallography, was forced to retract 70 papers co-authored by two researchers at Jinggangshan university in southern China, because they had fabricated evidence described in the papers. After the retractions the Lancet, a British journal, published a broadside urging China to take more action to prevent fraud. But many cases are covered up when detected to protect the institutions involved.

The pirated medical-journal racket broken up in Beijing shows that there is a well-developed market for publication beyond the authentic SCI journals. The cost of placing an article in one of the counterfeit journals was up to $650, police said. Purchasing a fake article cost up to $250. Police said the racket had earned several million yuan ($500,000 or more) since 2009. Customers were typically medical researchers angling for promotion.

Some government officials want to buy their way to academic stardom as well: at his trial this month for corruption, Zhang Shuguang, a former railway-ministry official, admitted to having spent nearly half of $7.8m in bribes that he had collected trying to get himself elected to the Chinese Academy of Sciences. Chinese reports speculated that he spent the money buying votes and hiring teams of writers to produce books. Widely considered to be a man of limited academic achievement, Mr Zhang ultimately fell just one vote short of election. Less than two years later, he was in custody.
social interaction is an interruption.

shape or be shaped.

User avatar
Info
Dean of Beatdowns
Posts: 10144
Joined: Sat May 15, 2010 10:34 am
Contact:

Re: How reliable is "science"

Post by Info » Fri Oct 18, 2013 6:01 am

Problems with scientific research
How science goes wrong
Oct 19th 2013

A SIMPLE idea underpins science: “trust, but verify”. Results should always be subject to challenge from experiment. That simple but powerful idea has generated a vast body of knowledge. But scientists are doing too much trusting and not enough verifying—to the detriment of the whole of science, and of humanity.

Too many of the findings that fill the academic ether are the result of shoddy experiments or poor analysis (see article). A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 “landmark” studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties.

What a load of rubbish
Even when flawed research does not put people’s lives at risk—and much of it is too far from the market to do so—it squanders money and the efforts of some of the world’s best minds. The opportunity costs of stymied progress are hard to quantify, but they are likely to be vast. And they could be rising.

One reason is the competitiveness of science. In the 1950s, when modern academic research took shape after its successes in the second world war, it was still a rarefied pastime. The entire club of scientists numbered a few hundred thousand. As their ranks have swelled, to 6m-7m active researchers on the latest reckoning, scientists have lost their taste for self-policing and quality control. The obligation to “publish or perish” has come to rule over academic life. Competition for jobs is cut-throat. Full professors in America earned on average $135,000 in 2012—more than judges did. Every year six freshly minted PhDs vie for every academic post. Nowadays verification (the replication of other people’s results) does little to advance a researcher’s career. And without verification, dubious findings live on to mislead.

Careerism also encourages exaggeration and the cherry-picking of results. In order to safeguard their exclusivity, the leading journals impose high rejection rates: in excess of 90% of submitted manuscripts. The most striking findings have the greatest chance of making it onto the page. Little wonder that one in three researchers knows of a colleague who has pepped up a paper by, say, excluding inconvenient data from results “based on a gut feeling”. And as more research teams around the world work on a problem, the odds shorten that at least one will fall prey to an honest confusion between the sweet signal of a genuine discovery and a freak of the statistical noise. Such spurious correlations are often recorded in journals eager for startling papers. If they touch on drinking wine, going senile or letting children play video games, they may well command the front pages of newspapers, too.

Conversely, failures to prove a hypothesis are rarely even offered for publication, let alone accepted. “Negative results” now account for only 14% of published papers, down from 30% in 1990. Yet knowing what is false is as important to science as knowing what is true. The failure to report failures means that researchers waste money and effort exploring blind alleys already investigated by other scientists.

The hallowed process of peer review is not all it is cracked up to be, either. When a prominent medical journal ran research past other experts in the field, it found that most of the reviewers failed to spot mistakes it had deliberately inserted into papers, even after being told they were being tested.

If it’s broke, fix it
All this makes a shaky foundation for an enterprise dedicated to discovering the truth about the world. What might be done to shore it up? One priority should be for all disciplines to follow the example of those that have done most to tighten standards. A start would be getting to grips with statistics, especially in the growing number of fields that sift through untold oodles of data looking for patterns. Geneticists have done this, and turned an early torrent of specious results from genome sequencing into a trickle of truly significant ones.

Ideally, research protocols should be registered in advance and monitored in virtual notebooks. This would curb the temptation to fiddle with the experiment’s design midstream so as to make the results look more substantial than they are. (It is already meant to happen in clinical trials of drugs, but compliance is patchy.) Where possible, trial data also should be open for other researchers to inspect and test.

The most enlightened journals are already becoming less averse to humdrum papers. Some government funding agencies, including America’s National Institutes of Health, which dish out $30 billion on research each year, are working out how best to encourage replication. And growing numbers of scientists, especially young ones, understand statistics. But these trends need to go much further. Journals should allocate space for “uninteresting” work, and grant-givers should set aside money to pay for it. Peer review should be tightened—or perhaps dispensed with altogether, in favour of post-publication evaluation in the form of appended comments. That system has worked well in recent years in physics and mathematics. Lastly, policymakers should ensure that institutions using public money also respect the rules.

Science still commands enormous—if sometimes bemused—respect. But its privileged status is founded on the capacity to be right most of the time and to correct its mistakes when it gets things wrong. And it is not as if the universe is short of genuine mysteries to keep generations of scientists hard at work. The false trails laid down by shoddy research are an unforgivable barrier to understanding.
social interaction is an interruption.

shape or be shaped.

User avatar
Info
Dean of Beatdowns
Posts: 10144
Joined: Sat May 15, 2010 10:34 am
Contact:

Re: How reliable is "science"

Post by Info » Fri Oct 18, 2013 6:04 am

Unreliable research
Trouble at the lab
Oct 19th 2013
Scientists like to think of science as self-correcting. To an alarming degree, it is not

“I SEE a train wreck looming,” warned Daniel Kahneman, an eminent psychologist, in an open letter last year. The premonition concerned research on a phenomenon known as “priming”. Priming studies suggest that decisions can be influenced by apparently irrelevant actions or events that took place just before the cusp of choice. They have been a boom area in psychology over the past decade, and some of their insights have already made it out of the lab and into the toolkits of policy wonks keen on “nudging” the populace.

Dr Kahneman and a growing number of his colleagues fear that a lot of this priming research is poorly founded. Over the past few years various researchers have made systematic attempts to replicate some of the more widely cited priming experiments. Many of these replications have failed. In April, for instance, a paper in PLoS ONE, a journal, reported that nine separate experiments had not managed to reproduce the results of a famous study from 1998 purporting to show that thinking about a professor before taking an intelligence test leads to a higher score than imagining a football hooligan.

The idea that the same experiments always get the same results, no matter who performs them, is one of the cornerstones of science’s claim to objective truth. If a systematic campaign of replication does not lead to the same results, then either the original research is flawed (as the replicators claim) or the replications are (as many of the original researchers on priming contend). Either way, something is awry.

To err is all too common
It is tempting to see the priming fracas as an isolated case in an area of science—psychology—easily marginalised as soft and wayward. But irreproducibility is much more widespread. A few years ago scientists at Amgen, an American drug company, tried to replicate 53 studies that they considered landmarks in the basic science of cancer, often co-operating closely with the original researchers to ensure that their experimental technique matched the one used first time round. According to a piece they wrote last year in Nature, a leading scientific journal, they were able to reproduce the original results in just six. Months earlier Florian Prinz and his colleagues at Bayer HealthCare, a German pharmaceutical giant, reported in Nature Reviews Drug Discovery, a sister journal, that they had successfully reproduced the published results in just a quarter of 67 seminal studies.

The governments of the OECD, a club of mostly rich countries, spent $59 billion on biomedical research in 2012, nearly double the figure in 2000. One of the justifications for this is that basic-science results provided by governments form the basis for private drug-development work. If companies cannot rely on academic research, that reasoning breaks down. When an official at America’s National Institutes of Health (NIH) reckons, despairingly, that researchers would find it hard to reproduce at least three-quarters of all published biomedical findings, the public part of the process seems to have failed.

Academic scientists readily acknowledge that they often get things wrong. But they also hold fast to the idea that these errors get corrected over time as other scientists try to take the work further. Evidence that many more dodgy results are published than are subsequently corrected or withdrawn calls that much-vaunted capacity for self-correction into question. There are errors in a lot more of the scientific papers being published, written about and acted on than anyone would normally suppose, or like to think.

Various factors contribute to the problem. Statistical mistakes are widespread. The peer reviewers who evaluate papers before journals commit to publishing them are much worse at spotting mistakes than they or others appreciate. Professional pressure, competition and ambition push scientists to publish more quickly than would be wise. A career structure which lays great stress on publishing copious papers exacerbates all these problems. “There is no cost to getting things wrong,” says Brian Nosek, a psychologist at the University of Virginia who has taken an interest in his discipline’s persistent errors. “The cost is not getting them published.”

First, the statistics, which if perhaps off-putting are quite crucial. Scientists divide errors into two classes. A type I error is the mistake of thinking something is true when it is not (also known as a “false positive”). A type II error is thinking something is not true when in fact it is (a “false negative”). When testing a specific hypothesis, scientists run statistical checks to work out how likely it would be for data which seem to support the idea to have come about simply by chance. If the likelihood of such a false-positive conclusion is less than 5%, they deem the evidence that the hypothesis is true “statistically significant”. They are thus accepting that one result in 20 will be falsely positive—but one in 20 seems a satisfactorily low rate.

Understanding insignificance
In 2005 John Ioannidis, an epidemiologist from Stanford University, caused a stir with a paper showing why, as a matter of statistical logic, the idea that only one such paper in 20 gives a false-positive result was hugely optimistic. Instead, he argued, “most published research findings are probably false.” As he told the quadrennial International Congress on Peer Review and Biomedical Publication, held this September in Chicago, the problem has not gone away.

Dr Ioannidis draws his stark conclusion on the basis that the customary approach to statistical significance ignores three things: the “statistical power” of the study (a measure of its ability to avoid type II errors, false negatives in which a real signal is missed in the noise); the unlikeliness of the hypothesis being tested; and the pervasive bias favouring the publication of claims to have found something new.

A statistically powerful study is one able to pick things up even when their effects on the data are small. In general bigger studies—those which run the experiment more times, recruit more patients for the trial, or whatever—are more powerful. A power of 0.8 means that of ten true hypotheses tested, only two will be ruled out because their effects are not picked up in the data; this is widely accepted as powerful enough for most purposes. But this benchmark is not always met, not least because big studies are more expensive. A study in April by Dr Ioannidis and colleagues found that in neuroscience the typical statistical power is a dismal 0.21; writing in Perspectives on Psychological Science, Marjan Bakker of the University of Amsterdam and colleagues reckon that in that field the average power is 0.35.

Unlikeliness is a measure of how surprising the result might be. By and large, scientists want surprising results, and so they test hypotheses that are normally pretty unlikely and often very unlikely. Dr Ioannidis argues that in his field, epidemiology, you might expect one in ten hypotheses to be true. In exploratory disciplines like genomics, which rely on combing through vast troves of data about genes and proteins for interesting relationships, you might expect just one in a thousand to prove correct.

With this in mind, consider 1,000 hypotheses being tested of which just 100 are true (see chart). Studies with a power of 0.8 will find 80 of them, missing 20 because of false negatives. Of the 900 hypotheses that are wrong, 5%—that is, 45 of them—will look right because of type I errors. Add the false positives to the 80 true positives and you have 125 positive results, fully a third of which are specious. If you dropped the statistical power from 0.8 to 0.4, which would seem realistic for many fields, you would still have 45 false positives but only 40 true positives. More than half your positive results would be wrong.



The negative results are much more trustworthy; for the case where the power is 0.8 there are 875 negative results of which only 20 are false, giving an accuracy of over 97%. But researchers and the journals in which they publish are not very interested in negative results. They prefer to accentuate the positive, and thus the error-prone. Negative results account for just 10-30% of published scientific literature, depending on the discipline. This bias may be growing. A study of 4,600 papers from across the sciences conducted by Daniele Fanelli of the University of Edinburgh found that the proportion of negative results dropped from 30% to 14% between 1990 and 2007. Lesley Yellowlees, president of Britain’s Royal Society of Chemistry, has published more than 100 papers. She remembers only one that reported a negative result.

Statisticians have ways to deal with such problems. But most scientists are not statisticians. Victoria Stodden, a statistician at Stanford, speaks for many in her trade when she says that scientists’ grasp of statistics has not kept pace with the development of complex mathematical techniques for crunching data. Some scientists use inappropriate techniques because those are the ones they feel comfortable with; others latch on to new ones without understanding their subtleties. Some just rely on the methods built into their software, even if they don’t understand them.

Not even wrong

This fits with another line of evidence suggesting that a lot of scientific research is poorly thought through, or executed, or both. The peer-reviewers at a journal like Nature provide editors with opinions on a paper’s novelty and significance as well as its shortcomings. But some new journals—PLoS One, published by the not-for-profit Public Library of Science, was the pioneer—make a point of being less picky. These “minimal-threshold” journals, which are online-only, seek to publish as much science as possible, rather than to pick out the best. They thus ask their peer reviewers only if a paper is methodologically sound. Remarkably, almost half the submissions to PLoS One are rejected for failing to clear that seemingly low bar.

The pitfalls Dr Stodden points to get deeper as research increasingly involves sifting through untold quantities of data. Take subatomic physics, where data are churned out by the petabyte. It uses notoriously exacting methodological standards, setting an acceptable false-positive rate of one in 3.5m (known as the five-sigma standard). But maximising a single figure of merit, such as statistical significance, is never enough: witness the “pentaquark” saga. Quarks are normally seen only two or three at a time, but in the mid-2000s various labs found evidence of bizarre five-quark composites. The analyses met the five-sigma test. But the data were not “blinded” properly; the analysts knew a lot about where the numbers were coming from. When an experiment is not blinded, the chances that the experimenters will see what they “should” see rise. This is why people analysing clinical-trials data should be blinded to whether data come from the “study group” or the control group. When looked for with proper blinding, the previously ubiquitous pentaquarks disappeared.

Other data-heavy disciplines face similar challenges. Models which can be “tuned” in many different ways give researchers more scope to perceive a pattern where none exists. According to some estimates, three-quarters of published scientific papers in the field of machine learning are bunk because of this “overfitting”, says Sandy Pentland, a computer scientist at the Massachusetts Institute of Technology.

Similar problems undid a 2010 study published in Science, a prestigious American journal (and reported in this newspaper). The paper seemed to uncover genetic variants strongly associated with longevity. Other geneticists immediately noticed that the samples taken from centenarians on which the results rested had been treated in different ways from those from a younger control group. The paper was retracted a year later, after its authors admitted to “technical errors” and “an inadequate quality-control protocol”.

The number of retractions has grown tenfold over the past decade. But they still make up no more than 0.2% of the 1.4m papers published annually in scholarly journals. Papers with fundamental flaws often live on. Some may develop a bad reputation among those in the know, who will warn colleagues. But to outsiders they will appear part of the scientific canon.

Blame the ref
The idea that there are a lot of uncorrected flaws in published studies may seem hard to square with the fact that almost all of them will have been through peer-review. This sort of scrutiny by disinterested experts—acting out of a sense of professional obligation, rather than for pay—is often said to make the scientific literature particularly reliable. In practice it is poor at detecting many types of error.

John Bohannon, a biologist at Harvard, recently submitted a pseudonymous paper on the effects of a chemical derived from lichen on cancer cells to 304 journals describing themselves as using peer review. An unusual move; but it was an unusual paper, concocted wholesale and stuffed with clangers in study design, analysis and interpretation of results. Receiving this dog’s dinner from a fictitious researcher at a made up university, 157 of the journals accepted it for publication.

Dr Bohannon’s sting was directed at the lower tier of academic journals. But in a classic 1998 study Fiona Godlee, editor of the prestigious British Medical Journal, sent an article containing eight deliberate mistakes in study design, analysis and interpretation to more than 200 of the BMJ’s regular reviewers. Not one picked out all the mistakes. On average, they reported fewer than two; some did not spot any.

Another experiment at the BMJ showed that reviewers did no better when more clearly instructed on the problems they might encounter. They also seem to get worse with experience. Charles McCulloch and Michael Callaham, of the University of California, San Francisco, looked at how 1,500 referees were rated by editors at leading journals over a 14-year period and found that 92% showed a slow but steady drop in their scores.

As well as not spotting things they ought to spot, there is a lot that peer reviewers do not even try to check. They do not typically re-analyse the data presented from scratch, contenting themselves with a sense that the authors’ analysis is properly conceived. And they cannot be expected to spot deliberate falsifications if they are carried out with a modicum of subtlety.

Fraud is very likely second to incompetence in generating erroneous results, though it is hard to tell for certain. Dr Fanelli has looked at 21 different surveys of academics (mostly in the biomedical sciences but also in civil engineering, chemistry and economics) carried out between 1987 and 2008. Only 2% of respondents admitted falsifying or fabricating data, but 28% of respondents claimed to know of colleagues who engaged in questionable research practices.

Peer review’s multiple failings would matter less if science’s self-correction mechanism—replication—was in working order. Sometimes replications make a difference and even hit the headlines—as in the case of Thomas Herndon, a graduate student at the University of Massachusetts. He tried to replicate results on growth and austerity by two economists, Carmen Reinhart and Kenneth Rogoff, and found that their paper contained various errors, including one in the use of a spreadsheet.

Harder to clone than you would wish
Such headlines are rare, though, because replication is hard and thankless. Journals, thirsty for novelty, show little interest in it; though minimum-threshold journals could change this, they have yet to do so in a big way. Most academic researchers would rather spend time on work that is more likely to enhance their careers. This is especially true of junior researchers, who are aware that overzealous replication can be seen as an implicit challenge to authority. Often, only people with an axe to grind pursue replications with vigour—a state of affairs which makes people wary of having their work replicated.

There are ways, too, to make replication difficult. Reproducing research done by others often requires access to their original methods and data. A study published last month in PeerJ by Melissa Haendel, of the Oregon Health and Science University, and colleagues found that more than half of 238 biomedical papers published in 84 journals failed to identify all the resources (such as chemical reagents) necessary to reproduce the results. On data, Christine Laine, the editor of the Annals of Internal Medicine, told the peer-review congress in Chicago that five years ago about 60% of researchers said they would share their raw data if asked; now just 45% do. Journals’ growing insistence that at least some raw data be made available seems to count for little: a recent review by Dr Ioannidis which showed that only 143 of 351 randomly selected papers published in the world’s 50 leading journals and covered by some data-sharing policy actually complied.


And then there are the data behind unpublished research. A study in the BMJ last year found that fewer than half the clinical trials financed by the NIH resulted in publication in a scholarly journal within 30 months of completion; a third remained unpublished after 51 months. Only 22% of trials released their summary results within one year of completion, even though the NIH requires that they should.

Clinical trials are very costly to rerun. Other people looking at the same problems thus need to be able to access their data. And that means all the data. Focusing on a subset of the data can, wittingly or unwittingly, provide researchers with the answer they want. Ben Goldacre, a British doctor and writer, has been leading a campaign to bring pharmaceutical firms to book for failing to make available all the data from their trials. It may be working. In February GlaxoSmithKline, a British drugmaker, became the first big pharma company to promise to publish all its trial data.

Software can also be a problem for would-be replicators. Some code used to analyse data or run models may be the result of years of work and thus precious intellectual property that gives its possessors an edge in future research. Although most scientists agree in principle that data should be openly available, there is genuine disagreement on software. Journals which insist on data-sharing tend not to do the same for programs.

Harry Collins, a sociologist of science at Cardiff University, makes a more subtle point that cuts to the heart of what a replication can be. Even when the part of the paper devoted to describing the methods used is up to snuff (and often it is not), performing an experiment always entails what sociologists call “tacit knowledge”—craft skills and extemporisations that their possessors take for granted but can pass on only through example. Thus if a replication fails, it could be because the repeaters didn’t quite get these je-ne-sais-quoi bits of the protocol right.

Taken to extremes, this leads to what Dr Collins calls “the experimenter’s regress”—you can say an experiment has truly been replicated only if the replication gets the same result as the original, a conclusion which makes replication pointless. Avoiding this, and agreeing that a replication counts as “the same procedure” even when it gets a different result, requires recognising the role of tacit knowledge and judgment in experiments. Scientists are not comfortable discussing such things at the best of times; in adversarial contexts it gets yet more vexed.

Some organisations are trying to encourage more replication. PLoS ONE and Science Exchange, a matchmaking service for researchers and labs, have launched a programme called the Reproducibility Initiative through which life scientists can pay to have their work validated by an independent lab. On October 16th the initiative announced it had been given $1.3m by the Laura and John Arnold Foundation, a charity, to look at 50 of the highest-impact cancer findings published between 2010 and 2012. Blog Syn, a website run by graduate students, is dedicated to reproducing chemical reactions reported in papers. The first reaction they tried to repeat worked—but only at a much lower yield than was suggested in the original research.

Making the paymasters care
Conscious that it and other journals “fail to exert sufficient scrutiny over the results that they publish” in the life sciences, Nature and its sister publications introduced an 18-point checklist for authors this May. The aim is to ensure that all technical and statistical information that is crucial to an experiment’s reproducibility or that might introduce bias is published. The methods sections of papers are being expanded online to cope with the extra detail; and whereas previously only some classes of data had to be deposited online, now all must be.

Things appear to be moving fastest in psychology. In March Dr Nosek unveiled the Centre for Open Science, a new independent laboratory, endowed with $5.3m from the Arnold Foundation, which aims to make replication respectable. Thanks to Alan Kraut, the director of the Association for Psychological Science, Perspectives on Psychological Science, one of the association’s flagship publications, will soon have a section devoted to replications. It might be a venue for papers from a project, spearheaded by Dr Nosek, to replicate 100 studies across the whole of psychology that were published in the first three months of 2008 in three leading psychology journals.

People who pay for science, though, do not seem seized by a desire for improvement in this area. Helga Nowotny, president of the European Research Council, says proposals for replication studies “in all likelihood would be turned down” because of the agency’s focus on pioneering work. James Ulvestad, who heads the division of astronomical sciences at America’s National Science Foundation, says the independent “merit panels” that make grant decisions “tend not to put research that seeks to reproduce previous results at or near the top of their priority lists”. Douglas Kell of Research Councils UK, which oversees Britain’s publicly funded research argues that current procedures do at least tackle the problem of bias towards positive results: “If you do the experiment and find nothing, the grant will nonetheless be judged more highly if you publish.”

In testimony before Congress on March 5th Bruce Alberts, then the editor of Science, outlined what needs to be done to bolster the credibility of the scientific enterprise. Journals must do more to enforce standards. Checklists such as the one introduced by Nature should be adopted widely, to help guard against the most common research errors. Budding scientists must be taught technical skills, including statistics, and must be imbued with scepticism towards their own results and those of others. Researchers ought to be judged on the basis of the quality, not the quantity, of their work. Funding agencies should encourage replications and lower the barriers to reporting serious efforts which failed to reproduce a published result. Information about such failures ought to be attached to the original publications.

And scientists themselves, Dr Alberts insisted, “need to develop a value system where simply moving on from one’s mistakes without publicly acknowledging them severely damages, rather than protects, a scientific reputation.”
social interaction is an interruption.

shape or be shaped.

User avatar
Dick Van Dyke
Hall Monitor
Posts: 1614
Joined: Thu Feb 16, 2012 11:59 pm

Re: How reliable is "science"

Post by Dick Van Dyke » Tue Nov 19, 2013 8:13 am

The MSM-Hyped “97% of The World’s Climate Scientists” Agree Global Warming Is Real Turns Out To Be Only 75 People, Not 2,500 As Claimed…

How do we know there’s a scientific consensus on climate change? Pundits and the press tell us so. And how do the pundits and the press know? Until recently, they typically pointed to the number 2500 – that’s the number of scientists associated with the United Nations Intergovernmental Panel on Climate Change. Those 2500, the pundits and the press believed, had endorsed the IPCC position.

To their embarrassment, most of the pundits and press discovered that they were mistaken – those 2500 scientists hadn’t endorsed the IPCC’s conclusions, they had merely reviewed some part or other of the IPCC’s mammoth studies. To add to their embarrassment, many of those reviewers from within the IPCC establishment actually disagreed with the IPCC’s conclusions, sometimes vehemently.

The upshot? The punditry looked for and recently found an alternate number to tout — “97% of the world’s climate scientists” accept the consensus, articles in the Washington Post and elsewhere have begun to claim.

This number will prove a new embarrassment to the pundits and press who use it. The number stems from a 2009 online survey of 10,257 earth scientists, conducted by two researchers at the University of Illinois. The survey results must have deeply disappointed the researchers – in the end, they chose to highlight the views of a subgroup of just 77 scientists, 75 of whom thought humans contributed to climate change. The ratio 75/77 produces the 97% figure that pundits now tout.

The two researchers started by altogether excluding from their survey the thousands of scientists most likely to think that the Sun, or planetary movements, might have something to do with climate on Earth – out were the solar scientists, space scientists, cosmologists, physicists, meteorologists and astronomers. That left the 10,257 scientists in disciplines like geology, oceanography, paleontology, and geochemistry that were somehow deemed more worthy of being included in the consensus. The two researchers also decided that scientific accomplishment should not be a factor in who could answer – those surveyed were determined by their place of employment (an academic or a governmental institution). Neither was academic qualification a factor – about 1,000 of those surveyed did not have a PhD, some didn’t even have a master’s diploma.

To encourage a high participation among these remaining disciplines, the two researchers decided on a quickie survey that would take less than two minutes to complete, and would be done online, saving the respondents the hassle of mailing a reply. Nevertheless, most didn’t consider the quickie survey worthy of response –just 3146, or 30.7%, answered the two questions on the survey:

1. When compared with pre-1800s levels, do you think that mean global temperatures have generally risen, fallen, or remained relatively constant?

2. Do you think human activity is a significant contributing factor in changing mean global temperatures?

The questions were actually non-questions. From my discussions with literally hundreds of skeptical scientists over the past few years, I know of none who claims that the planet hasn’t warmed since the 1700s, and almost none who think that humans haven’t contributed in some way to the recent warming – quite apart from carbon dioxide emissions, few would doubt that the creation of cities and the clearing of forests for agricultural lands have affected the climate. When pressed for a figure, global warming skeptics might say that human are responsible for 10% or 15% of the warming; some skeptics place the upper bound of man’s contribution at 35%. The skeptics only deny that humans played a dominant role in Earth’s warming.

Surprisingly, just 90% of those who responded to the first question believed that temperatures had risen – I would have expected a figure closer to 100%, since Earth was in the Little Ice Age in the centuries immediately preceding 1800. But perhaps some of the responders interpreted the question to include the past 1000 years, when Earth was in the Medieval Warm Period, generally thought to be warmer than today.

As for the second question, 82% of the earth scientists replied that that human activity had significantly contributed to the warming. Here the vagueness of the question comes into play. Since skeptics believe that human activity been a contributing factor, their answer would have turned on whether they consider a 10% or 15% or 35% increase to be a significant contributing factor. Some would, some wouldn’t.

In any case, the two researchers must have feared that an 82% figure would fall short of a convincing consensus – almost one in five wasn’t blaming humans for global warming — so they looked for subsets that would yield a higher percentage. They found it – almost — in those whose recent published peer-reviewed research fell primarily in the climate change field. But the percentage still fell short of the researchers’ ideal. So they made another cut, allowing only the research conducted by those earth scientists who identified themselves as climate scientists.

Once all these cuts were made, 75 out of 77 scientists of unknown qualifications were left endorsing the global warming orthodoxy. The two researchers were then satisfied with their findings. Are you?
coffee's for closers.

User avatar
Info
Dean of Beatdowns
Posts: 10144
Joined: Sat May 15, 2010 10:34 am
Contact:

Re: How reliable is "science"

Post by Info » Fri Jan 31, 2014 8:03 pm

satire with a grain of truth:

social interaction is an interruption.

shape or be shaped.

User avatar
Info
Dean of Beatdowns
Posts: 10144
Joined: Sat May 15, 2010 10:34 am
Contact:

Re: How reliable is "science"

Post by Info » Mon Feb 17, 2014 2:51 am



What 60 Minutes' Lesley Stahl learned about feminism's poisoning of scientific research.
social interaction is an interruption.

shape or be shaped.

User avatar
Info
Dean of Beatdowns
Posts: 10144
Joined: Sat May 15, 2010 10:34 am
Contact:

Re: How reliable is "science"

Post by Info » Wed Feb 19, 2014 12:07 am

just a simple joke to prove a point about the abuse of statistics/numbers:

Image

numbers don't lie; people do.
social interaction is an interruption.

shape or be shaped.

User avatar
Info
Dean of Beatdowns
Posts: 10144
Joined: Sat May 15, 2010 10:34 am
Contact:

Re: How reliable is "science"

Post by Info » Sat Mar 01, 2014 11:08 am

Alarming number of peer-reviewed papers are irreproducible
February 28, 2014

The ability to duplicate an experiment and its results is a central tenet of the scientific method, but recent research has shown an alarming number of peer-reviewed papers are irreproducible. In 2013, on the heels of several retraction scandals and studies showing reproducibility rates as low as 10 percent for peer-reviewed articles, the prominent scientific journal Nature dedicated a special issue to the concerns over irreproducibility.

"Too few biologists receive adequate training in statistics and other quantitative aspects of their subject," the editors wrote. "Mentoring of young scientists on matters of rigor and transparency is inconsistent at best."

The grade school maxim to "show your work" doesn't hold in the average introductory statistics class, said Mine Cetinkaya-Rundel, assistant professor of the practice in the Duke statistics department. In a typical workflow, a college-level statistics student will perform data analysis in one software package, but transfer the results into something better suited to presentation, like Microsoft Word or Microsoft PowerPoint.

Though standard, this workflow divorces the raw data and analysis from the final results, making it difficult for students to retrace their steps. The process can give rise to errors, and in many cases, the authors write, "the copy-and-paste paradigm enables, and even encourages, selective reporting."

As the use and analysis of big data becomes increasingly sophisticated, the team writes, the ability of researchers to retrace steps and achieve the same statistical outcomes will only grow in significance.
social interaction is an interruption.

shape or be shaped.

User avatar
Info
Dean of Beatdowns
Posts: 10144
Joined: Sat May 15, 2010 10:34 am
Contact:

Re: How reliable is "science"

Post by Info » Sat Apr 05, 2014 9:11 am

A whistleblower’s fight against fraudulent research
April 04, 2014

In July 2012, Paul Brookes, an associate professor of anesthesiology at the University of Rochester, launched Science Fraud, a website where whistleblowing life scientists could anonymously submit and discuss suspicious research in their field. The site lasted just six months before Brookes was outed and the site was shut down under a bombardment of legal threats.

Because of a high volume of submissions, Brookes still had 223 articles whose alleged problems he never had time to publish—not far from the 274 the site actually had managed to review. This gave him a unique opportunity, he realized. While opening his peers to public scrutiny angered many people, he could prove it made for better science, too.
Edit: oh look it's what we teach at Manhood 101 all the fucking time :dance:
In its short run, Science Fraud contributed to a burgeoning enthusiasm for self-policing in the sciences, which is increasingly easy thanks to blogging and social media. Little data exists to show whether or not self-policing actually leads to more corrections in the field, though. Does airing science’s dirty laundry actually compel journals and researchers to acknowledge bad data more than they would if only contacted in private?

That’s exactly what Brookes was able to find evidence for. Twenty-three percent of the problematic papers Science Fraud discussed among the scientists were either retracted from or corrected in journals, he reported yesterday in PeerJ. But only 3.1 percent of the papers the website never had time to critique were similarly addressed, even though the journals, funders, and authors’ institutions involved with these papers were still notified of potential errors. Public exposure appears to have made poor research seven times more likely to be fixed.

Where is the scientific integrity and self-correction we idealize? Sadly, Brookes’ report includes a handful of anonymous testimonials that highlight the frequent politics behind scientific corrections, such as:
"I reviewed a paper and found fabricated data. The journal rejected the paper, and subsequently it was published in a different journal with some problem data still present. The editor at the new journal knows about the previous rejection for reasons of data fabrication, but refuses to take up the matter with the authors unless I am willing to have my real name revealed as an accuser. I refused, because the lead author is on a panel that reviews my grant proposals."
New forums encouraging data sharing and post-publication scrutiny offer hope for improvement, according to Brookes, but many issues remain to be addressed as science’s corrective system evolves. “The jury is still out on exactly what the best system is,” he says in a press release, “who should be allowed to comment, will they be afforded anonymity, and of course who will pay for and police all this activity.”
social interaction is an interruption.

shape or be shaped.

User avatar
Info
Dean of Beatdowns
Posts: 10144
Joined: Sat May 15, 2010 10:34 am
Contact:

Re: How reliable is "science"

Post by Info » Fri Apr 18, 2014 10:07 am

Updated Review: Tamiflu Is a Bust
After finally getting their hands on full clinical study reports, independent reviewers say the antiviral drug is ineffective.
April 10, 2014

Governments have spent billions of dollars stockpiling the antiviral medication Tamiflu. Earlier reviews of the drug called into question just how effective it was, and the latest analysis, published today (April 10) in the British Medical Journal (BMJ), concludes that the money has been going “down the drain.”

An international team found that while Tamiflu might reduce the duration of flu symptoms by half a day, there’s no evidence that it reduces hospital admissions or complications of an infection. On top of that, the antiviral’s side effects include nausea and vomiting. “There is no credible way these drugs could prevent a pandemic,” Carl Heneghan, one of authors of the review and a professor at Oxford University, told reporters.

The data for this most recent review came from full study reports—data generated by clinical trials that are usually not open for scrutiny by independent researchers. Efforts by the BMJ and the research team convinced drugmaker Roche, which markets Tamiflu, to release the reports.

Fiona Godlee, an editor at BMJ, said that the picture of Tamiflu was previously much more positive than after the full study reports were disclosed. “Why did no one else demand this level of scrutiny before spending such huge sums on one drug?” she said at a press briefing. “The whole story gives an extraordinary picture of the entrenched flaws in the current system of drug regulation and drug evaluation.”

Roche stands by the utility of Tamiflu. “We fundamentally disagree with the overall conclusions” of the review, the company told MedPage Today. And others have said that the results don’t necessitate an end to stockpiling the drug. Sabrina Spinosa of the European Medicines Agency (EMA), which approved the use of Tamiflu in 2002, told Nature that the agency had reviewed the same clinical trial reports. “The review does not raise any new concerns,” she said, adding that the EMA maintains its position on the risks and benefits of Tamiflu.
social interaction is an interruption.

shape or be shaped.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest