Category: scientific method

I haven’t been interested in the work of John Ioannidis because it seems unrelated to discovery. Ioannidis says too many papers are “wrong”. I don’t know how the fraction of “wrong” papers is related to the rate of discovery. For example, what percentage of “wrong” papers produces the most discovery? Ioannidis doesn’t seem to think about this. Yet that is the goal of science — better understanding. Not “right” papers. Continue reading “The Trouble with Critics of Science, Such as John Ioannidis”

A year ago, Eric Lander, who identified himself as “President and Founding Director of the Broad Institute of Harvard and MIT [and] one of the principal leaders of the Human Genome Project, directing the largest center in the international project” did a Reddit AMA (Ask Me Anything). One of the questions did not go as expected:

Question As an advisor to the President, what is being done or do you think will be done to increase the attractiveness of students finishing PhD programs in science?

Lander We need to shorten the time for getting a PhD and for a first faculty job. Young people should get out into the scientific world early, when they have lots of fresh ideas. We should encourage grants to young scientists and should encourage them to take big risks. When you’re taking big risks, science is amazingly fun.

The response to this answer was very negative.

With all due respect, this is a ludicrous statement. . . The true problem is the way in which you fund science. You fund projects and proposals. In order to get these projects funded, the preliminary data has to be essentially the whole project being done. Then you fund at a 6% percent line. It leads to cronyism in the peer review process and a general sense of despair in scientists. How about you radically change the funding system for PIs?

I too am disappointed with Dr. Lander’s response to possibly THE most important question here regarding training basic scientists.

Do you truly believe this? . . . There is no reason to encourage more students to go into science if there is not enough government funding to support their careers.

Alas, this is not important. It just pleased me that someone questioned Dr. Lander’s absurd claims, which he makes often. “We should encourage young scientists to take big risks”. Yes, I agree, does he really believe this? Do he really believe that someone coming up for tenure should take big risks?

In a recent post, I said a well-functioning system will produce both optimality and complexity. I meant important systems like our bodies, economies, and formal education. If you look at the nutrition advice provided by the United States Department of Agriculture — the food pyramid, the food plate, the recommended daily allowances, and the associated reports — you will find nothing that increases the complexity of metabolism inside our bodies (in particular, the diversity of metabolic pathways). The advice is all optimality — for example, the best amounts of various micronutrients. The people behind the USDA advice, reflecting the thinking of the best nutrition scientists in the world, utterly fail to grasp the importance of complexity. Half of nutrition research — or more than half, since the topic has been so neglected — should be about how to increase internal complexity. In practice, almost none of it is. It’s obvious, I think, that the microbes within us are very important for health. They are mostly in our intestines and must be heavily influenced by what we eat. How did they get there? How can their number be increased? How can their diversity be increased?

The absence is especially striking because the point is so simple. To solve actual problems, you need both optimality and complexity. Showers — what we use to take a shower — provide an example. You want to adjust the water temperature. If you try to do this while taking a shower, it can be hard because of the delay between changing the hot/cold water proportions and feeling the effects. It is better to use the bathtub (lower) tap to set the temperature (measuring it with your wrist) and only after you’ve optimized the temperature, shift the water to the shower head. The bathtub tap produces simple output (a single stream of water) that is easy to optimize. The shower head produces more complex output that is harder to optimize but does a better job of washing (an actual problem). You need both bathtub tap (for optimization) and shower head (for complexity) to do a good job solving the problem. Likewise, we need both an understanding of necessary nutrients (Vitamin A, etc.), which can be optimized, and an understanding of microbes, which cannot be optimized but can be made more complex, to make good decisions about what food to eat. Ordinary food is the hardware, you might say; and microbes are the software.

 

Recently I interviewed two job candidates for an assistant professor position at Tsinghua. I asked both of them: “Why did you decide to study this?” (this = their field of research). One had no answer at all. The other had an answer that didn’t make sense. I didn’t mean it as a tough question. If they had said “because that’s what they were doing where I got a postdoc” I would have been perfectly happy. If that were the answer, I might have asked “why does your advisor study it?” — to which “I don’t know” would have been perfectly acceptable. Of course, there are better answers.

When I was a graduate student, I read Adventures of a Mathematician by Stanislaw Ulam (a very good well-written book). One of the book’s comments impressed me: That John Von Neumann was able to distinguish the main lines of growth of the tree of mathematics from the branches. My research was about how rats measure time. The relevance to big questions in the psychology of learning wasn’t obvious. I wondered: Am I studying something important? Or something that will be irrelevant in twenty years? My advisor didn’t seem to have thought about this. 

When I interviewed for jobs at various universities, no one asked me why do you study this? But it was still a question worth answering. As a grad student I had no choice. But eventually I would have a choice: I could continue to study how rats measure time. Or I could study something else. (Eventually I did change — to studying what controls variation in behavior.)

Here’s what I would say now about how to choose a research topic.

What’s best is a new method. If you can use a new method to answer questions in your field, do that. The cheaper, easier and more available the method, the better. As a graduate student, I developed a new way to study how rats measure time, which I called the peak procedure. It made it easier to determine if an experimental treatment affected an animal’s internal clock.

What’s second best is a new experimental effect. Discovering a new way to change something of interest. The bigger, cheaper, newer, and more surprising the effect, the better. Using the peak procedure, my colleagues and I discovered a large and surprising effect (at a certain time during the peak procedure, the variability of bar-press duration — how long a rat holds down the bar when pressing it — became much larger). When I first saw the result, I assumed it was due to a software mistake. It turned out to be a window in what controls the variability of behavior — an easy way of studying that. In that sense it was also a new method.

I don’t know if the two job candidates I interviewed were doing either of these two things. Maybe not. My broader point is that if you don’t have a good understanding of how to choose a research topic you will have to retreat to studying something simply because others are studying it. Which is exactly the wrong thing to do if you want to be an innovator and a leader.