Pressure to Publish Incentivizes Bad Science -- With Consequences for Us All

The Wired Word for the Week of October 2, 2016

In the News

The inner workings of the field of science and research may seem a long way from the everyday concerns of most of us, but consider how often you make decisions about diet, health, lifestyle, product purchases and other matters based on the results of scientific studies.

Now imagine that some of the findings you rely on have been made public not because they are as correct as rigorous testing can demonstrate, but because the findings have likely been published due to their novelty or the level of surprise or excitement they generate.

That, says a study published September 21 by researchers at the University of California, is what often happens when scientists compete for academic prestige and jobs.

In our day, the upward trajectory of scientists' careers depends on publishing as many papers as possible in the most prestigious journals. Those who succeed are the ones most likely to receive grants and jobs and to be held in high esteem.

Regarding this trend, Ed Yong, who covers science for The Atlantic, writes (see article in links list below), "Now, imagine you're a researcher who wants to game this system. Here's what you do:.Run many small and statistically weak studies. Tweak your methods on the fly to ensure positive results. If you get negative results, sweep them under the rug. Never try to check old results; only pursue new and exciting ones.

"These are not just flights of fancy," Yong continues. "We know that such practices abound. They're great for getting publications, but they also pollute the scientific record with results that aren't actually true. As Richard Horton, editor of The Lancet once wrote, 'No one is incentivized to be right. Instead, scientists are incentivized to be productive.'"

That scenario, of course, means the scientist involved is deliberately going for publication success rather than true results. But Paul Smaldino, one of the researchers of the just-released study, says that a similar outcome can happen "even if individuals aren't trying to maximize their metrics."

For the study, Smaldino and his research colleague Richard McElreath created a mathematical model which simulated labs competing with each other and "evolving" -- that is, as researchers varied their methods to achieve success, they passed those practices on to the students they were training, meaning that over time, the very culture of science is changed by natural selection.

This becomes a vicious cycle in which studies that get published because their results seem remarkable help the researcher get grants to conduct more research.

"There will always be researchers committed to rigorous methods and scientific integrity," Smaldino has written. "But as long as institutional incentives reward positive, novel results at the expense of rigor, the rate of bad science, on average, will increase."

There is indication that the problem has already reached significant levels. Recently, researchers found that only 36 percent of psychology studies examined could be reproduced. A major cancer research organization was unable to replicate over 90 percent of the "landmark"” cancer studies evaluated. (See articles in links list.)

It is hoped that studies such as the Smaldino one -- which is itself novel and attention-grabbing -- will raise awareness of this problem and lead to solutions, including changing the incentives to put a premium on transparency. After its results have been replicated, of course.

One solution being employed by almost 40 journals is something called "registered reports." In such cases, journal teams evaluate proposed experimentation ahead of time on the basis of its ideas and plans, before any actual work is launched. Then, for the plans they accept, the journal commits to publish the results, no matter what those findings are. It "moves the focus away from eye-catching results and towards solid, reliable methods," writes Yong. There are also moves to tie grants to registered reports.

More on this story can be found at these links:

The Inevitable Evolution of Bad Science. The AtlanticStudy Warns That Science as We Know It Is Evolving Into Something Shoddy and Unreliable. Science AlertIn Cancer Science, Many "Discoveries" Don't Hold Up. Reuters
The Natural Selection of Bad Science. Royal Society Open Science
Only a THIRD of Scientific Studies Can Be Replicated .... Daily Mail (UK)

The Big Questions

1. Regarding information, which fields or institutions are you most likely to trust? Why? Which ones are you most likely to be skeptical of? Why?

2. What gives any institution or field of inquiry authority and trustworthiness?

3. Regarding the spiritual life, to what do you grant authority? Why?

4. Is knowledge of truth in any absolute, objective sense even possible when declared by human beings? Or is truth always entangled in personal motivations that inherently distort it, whether deliberately or unintentionally? Or something else?

5. Can there be the genuine trust necessary for right understanding when there is not openness? Explain your answer.

Confronting the News With Scripture and HopeHere are some Bible verses to guide your discussion:

Deuteronomy 19:15A single witness shall not suffice to convict a person of any crime or wrongdoing in connection with any offense that may be committed. Only on the evidence of two or three witnesses shall a charge be sustained. (For context, read 19:15-21.)

This verse points out that within the judicial system of ancient Israel, witnesses not only provided testimony but also pressed charges against others. In order to enforce truthfulness in such proceedings, at least two and preferably three witnesses were needed to sustain a charge. This suggests a recognition that one witness alone could easily have ulterior motives or be mistaken.

How does this apply to the bad-science story? In good science, the results of a study should be reproducible by others, meaning that there would be at least two studies supporting a new scientific claim. But under the current system, there's not a lot of incentive to spend one's time redoing experiments others have already done and reported.

The biblical idea of two witnesses in court, it seems, is one that science -- and perhaps other disciplines -- would do well to adopt. There are, in fact, some scientists who focus on replicating past studies to see if their results hold, but, said Smaldino, "there are way more results than can possibly be replicated."

Better to have two witnesses or more to start with.

Questions: Do you think there would be more truthfulness and "full disclosure" in our society if people were convinced there would be real consequences for failures in this area? Does the very difficulty of proving skewed results serve in some ways to encourage it?

Ecclesiastes 1:12-14
I, the Teacher, when king over Israel in Jerusalem, applied my mind to seek and to search out by wisdom all that is done under heaven; it is an unhappy business that God has given to human beings to be busy with. I saw all the deeds that are done under the sun; and see, all is vanity and a chasing after wind. (For context, read 1:12-17.)

With this statement, the author of Ecclesiastes -- "the Teacher" -- seems to be talking about researching life insofar as he was able. While he never states what methodology he used, his "findings" are that "all is vanity and a chasing after wind."

Questions: Given his conclusion, do you think his research brought him to an accurate conclusion? If not, what might have skewed his work? Would others looking at the same phenomenon as this teacher reach the same findings? Why or why not? Why do you think Ecclesiastes was included in the scriptures?

Proverbs 24:3-4
By wisdom a house is built, and by understanding it is established; by knowledge the rooms are filled with all precious and pleasant riches. (No context needed.)

This old proverb praises both wisdom and knowledge, but of course, one needs accurate knowledge -- not skewed findings -- for wisdom to result.

Questions: Why do you think the Bible speaks of both knowledge and wisdom? Is there a biblical view of science? If so, what is it?

Matthew 6:1Beware of practicing your piety before others in order to be seen by them; for then you have no reward from your Father in heaven. (For context, read 6:1-6, 16-18.)

Here, and in the accompanying context verses, Jesus points out the dangers of doing seemingly good things for "reward." So he counsels his hearers to give alms without calling attention to the gift, to pray behind closed doors and to fast without telling others you are doing so. God will reward you eventually, but working for earthbound rewards subtracts from the power of your deed.

Questions: What is the biblical principle here? In what ways, if any, might this principle be applied to human fields of endeavor that ask for our trust and belief?

Colossians 3:23
Whatever your task, put yourselves into it, as done for the Lord and not for your masters .... (For context, read 3:18-25.)

Questions: How is this instruction from Paul applicable to those who work in scientific research? in banking? in journalism? in ministry? where you work?

For Further Discussion

1. Name some information you once relied on to guide how much of a certain food you consumed, and how you now have different information about the effects of that food on your health. What caused the change in information?

2. The idea behind reliability in research is that any significant results must be more than a one-off finding and be inherently repeatable. Other researchers must be able to perform exactly the same experiment, under the same conditions, and generate the same results. How might this apply, if at all, to the spiritual life?

3. Respond to the following: The Smaldino and McElreath simulation only examined only two incentives -- publication rates and replication rates. One TWW team member, who is a physicist, has noticed several other sources of bias over the course of his career:

• Desire to produce what a team leader wants. He writes, "I recall getting screamed at (no exaggeration) by my boss when my experiments, computer model results and recent theoretical analysis by another group of scientists all indicated that a pet idea of his would not work."
• Pressure to conform to what the sponsor (funding source) wants. Many people distrust studies sponsored by big corporations, such as tobacco companies or pharmaceutical companies, believing that scientists will yield to pressure to produce the "right" results. Even greater pressure can be exerted by federal funding agencies, many of which have a definite direction they want studies to indicate. This is particularly a problem when a field of study has only one or a few sources of funding.
• Insertion of one's own biases into the analysis. Scientists are human, and often see what they expect to see.
• Sometimes they just make mistakes. One recent paper in the Journal of Political Science reported that those with several psychoticism traits tended to be associated with "conservative" political views. In reality, they had swapped labels, and have since published a retraction, stating that the psychoticism was associated with "liberal" or "progressive" political views. (They had also swapped ideologies in other areas, originally reporting that neuroticism and social desirability were more associated with "liberal" views when actually they were more associated with "conservative" views.)

Responding to the News

Consider whether something you are struggling with spiritually would benefit from the input of another faithful Christian. While it can be hard for us to know, all on our own, whether something is true or whether God is calling us along a certain path, another Christ-follower -- or two or three -- might be able to affirm or challenge our "findings" and, in so doing, help us discern God's will in a given situation.

Prayer

Grant us discernment, O Lord, that we be neither eager consumers of questionable data that bolster our own positions nor eager deniers of questionable data that challenge our positions. And grant us the wisdom to know when we are confronted with truth. In Jesus' name. Amen.