On Scientific Method

When I visited George Mason University recently, I asked Tyler Cowen, “What’s the secret of a successful blog?” Cowen and Tabarrok’s Marginal Revolution is the most successful blog I know of.

His answer: “Three elements: 1. Expertise. 2. Regularity. 3. Recurring characters, like a TV show.” By regularity he meant at least 5 times/week.

I saw I had considerable room for improvement. Since then, I’ve tried to post at least twice/week. With this post I am adding scientific method to the subtitle, which I hope makes me appear more expert. A Berkeley philosophy professor named Paul Feyeraband wrote a book that I thought is called On Method but that I see is actually called Against Method. He was at Berkeley when I arrived. I remember two things about him: 1. He gave all his students A’s. 2. He ate at Chez Panisse every night.

CIA Fun Facts

Tonight, at a panel discussion at UC Berkeley that was part of The New Yorker College Tour, I learned two things about Central Intelligence Agency headquarters in Langley, Virginia:

1. There are scales in the bathrooms (according to Lawrence Wright).

2. There is a gift shop that sells CIA golf balls and the like. By the register is a notice: “If you are a covert operative, don’t use your credit card” (according to Jeffrey Goldberg).

The big shock, however, was neither of these. It was, as Hilary Goldstine pointed out, that there were almost no undergraduates in the audience. Which speaks volumes about UC Berkeley. It was a great discussion. Jane Mayer was the third discussant and Orville Schell the moderator.

The Trouble With Rigor

This is an easy question: When writing down numbers, when is it bad to be precise? Answer: When you exceed the precision to which the numbers were measured. If a number was measured with a standard error of 5 (say), don’t record it as 150.323.

But this, apparently, is a hard question: When planning an experiment, when it is bad to be rigorous? Answer: When the effort involved is better used elsewhere. I recently came across the following description of a weekend conference for obesity researchers (December 2006, funded by National Institute of Diabetes & Digestive & Kidney Diseases):

Obesity is a serious condition that is associated with and believed to cause much morbidity, reduced quality of life, and decreased longevity. . . . Currently available treatments are only modestly efficacious and rigorously evaluating new (and in some cases existing) treatments for obesity are clearly in order. Conducting such evaluations to the highest standards and so that they are maximally informative requires an understanding of best methods for the conduct of randomized clinical trials in general and how they can be tailored to the specific needs of obesity research in particular. . . . We will offer a two-day meeting in which leading obesity researchers and methodologists convene to discuss best practices for randomized clinical trials in obesity.

Rigorously evaluating new treatments”? How about evaluating them at all? Evaluation of new treatments (such as new diets) is already so difficult that it almost never occurs; here is a conference about how to make such evaluations more difficult.

This mistake happens in other areas, too, of course. Two research psychiatrists have complained that misguided requirements for rigor have had a very bad effect on finding new treatments for bipolar disorder.

Too Few Riders, Too Many Stolen Bases

I heard two excellent talks last week. Bent Flyvbjerg, a professor of Planning at Aalborg University, Aalborg, Denmark , spoke on “Survival of the Unfittest: Why the Worst Megaprojects [subways, airports, bridges, tunnels] Get Built.” Why? Because of false claims. Cost estimates turn out to be much too low and benefit estimates (such as ridership) much too high. Boston’s Big Dig, for example, has already cost more than three times the original estimate. Cost estimates were too low in 90% of projects, Flyvbjerg said. The tools used to make those estimates have supposedly improved a great deal over the last few decades but their accuracy has not improved. Lovallo and Kahneman have argued that the underlying problem is “optimism bias“; however, Flyvbjerg believes that the problem is what he now calls strategic misrepresentation — when he used the term lying people got upset. The greater the misrepresentation, the more likely the project would be approved — or rather the greater the truth the more likely the project would not be approved. That is a different kind of bias. An everyday example is me and my microwave oven. Sometimes I use my microwave oven to dry my clothes. I’ve done this dozens of times but I continue to badly underestimate how long it will take. I guess that a shirt will take 8 minutes to dry; it takes 15 minutes. I know I underestimate — but I keep doing it. This is not optimism bias. Microwaving is not unexpectedly difficult or unpredictable. The problem, I think, is the asymmetry of the effects of error. If my guess is too short, I have to put the shirt back in the microwave, which is inconvenient; if my guess is too long the shirt may burn — which corresponds to the project not being approved.

Incidentally, Flyvjberg has written a paper defending case studies and by extension self-experimentation. He quotes Hans Eysenck, who originally dismissed case studies as anecdotes: “Sometimes we simply have to keep our eyes open and look carefully at individual cases — not in the hope of proving anything but rather in the hope of learning something.” Exactly.

The other excellent talk (“Scagnostics” — scatterplot diagnostics) was by Leland Wilkinson, author of The Grammar of Graphics and developer of SYSTAT, who now works at SPSS. He described a system that classifies scatterplots. If you have twenty or thirty measures on each of several hundred people or cities or whatever, how do you make sense of it? Wilkinson’s algorithms measure such properties of a scatterplot as its texture, clumpiness, skewness, and four others I don’t remember. You use these measures to find the most interesting scatterplots. He illustrated the system with a set of baseball statistics — many measurements made on each of several hundred major-league baseball players. The scatterplot with the most outliers was stolen bases versus age. Stolen bases generally decline with age but there are many outliers. Although a vast number of statistical procedures assume normal distributions, Wilkinson’s tools revealed normality to be a kind of outlier. In the baseball dataset, only one scatterplot had both variables normally distributed: height versus weight. These tools may eventually be available with R.

David Jenkins on the Shangri-La Diet

David Jenkins, a professor of nutrition at the University of Toronto, invented the glycemic index, probably the most important nutritional innovation of the last thirty years. The glycemic index helped me permanently lose 6 pounds (see Example 7 of this paper). While preparing her CBC piece about the Shangri-La Diet, Sarah Kapoor interviewed Jenkins. Here is a partial transcript of what he said.

The Writing Cure

I wonder how many bloggers know about this — research about the beneficial effects of journal writing. James Pennebaker, a professor of psychology at the University of Texas Austin, has done a lot of research in this area. Here is a list of studies. This article sums it up nicely: “Writing about important personal experiences in an emotional way for as little as 15 minutes over the course of three days brings about improvements in mental and physical [!] health. This finding has been replicated across age, gender, culture, social class, and personality type.”

I’m guessing this research started as a search for the crucial ingredients of psychotherapy. What happens during psychotherapy that helps people? Early research found that the therapist’s training made no detectable difference. This suggested that just telling one’s story was therapeutic. Journal writing is another step in the same direction: You tell your story without anyone listening. Next step: studying the health effects of blogging.

The Annotated Woman’s World article

The next issue (Oct 3) of Woman’s World, already available many places, has a lovely cover story (pp. 18-19) about the Shangri-La Diet with the funny title “Instant Willpower!” The article is very accurate and reasonable but I have a few comments.

“Lose 7 lbs a week!” (cover and p. 19). Average weight loss is 1-2 lbs/week.

“Makes your body release stored fat!” (cover). Clever. I would have said something plodding like “lose body fat.”

“Roberts says refined walnut oil and light olive oils are best” (p. 18). Refined walnut oil is hard to find. I buy Spectrum refined walnut oil at Whole Foods. The Spectrum Organics store locator will find stores that carry Spectrum products but not all carry refined walnut oil. In Berkeley, most don’t. You may want to call ahead.

“When reading scientific journals to prepare for a lecture, Roberts had a eureka moment. . . Turning this interesting idea into practical weight-loss advice took lots of trial and error. . . . In short order, he was 35 pounds slimmer” (pp. 18-19). I lost 35 pounds using sugar water, not oil. It took three months. The turning point in going from theory to practice was a strange experience in Paris, described in the book. Also crucial was Emily Mechner’s observation that if my theory was correct, flavorless oils should work as well as sugar water. All in all, though, this is a good summary.

[to make this plan work even better] “Stick with your normal foods” (p. 19). No, I think the diet works better if you start eating foods that are new to you — foods with unfamiliar flavors.

“Avoid flax, unrefined walnut and extra virgin olive oils, which have strong flavor, says Roberts” (p. 19). You can drink these oils if you close your nose (using a noseclip for example) while drinking them. That will eliminate the flavor.

Seymour Benzer (crippling medical school research)

In an interview, Seymour Benzer, the great Caltech biologist, told a story that I think explains a lot about medical-school research, including UCLA medical school professor John Ford’s complaint about The Shangri-La Diet:

Harold Brown [president of Caltech 1969-1977] made himself quite conspicuous by . . . trying to develop a medical school relationship. . . . His idea was for Caltech to pair up with UCLA to make a medical school. We would do the first two years of basic education of the medical students, and afterwards they would be guaranteed two more years of clinical experience at UCLA. And then they could be doctors. . . . In the Biology Division, it went over like a lead balloon: Why should we be knocking ourselves out teaching these guys, and then they go away elsewhere and don’t even do research — they become doctors? What’s in it for us?

Some things are hard to learn by reading. Saul Sternberg, now a professor of psychology at Penn, once spent a quarter at Berkeley and was around when Stanford grad students and faculty in cognitive psychology came up to Berkeley to present their research. One of the grad students told Sternberg about a reaction-time experiment she had done about mental something or other (mental rotation?) in which the conditions compared varied in what the subject saw. Sternberg pointed out that it would be better to keep constant what the subject sees. This is the beginning of wisdom in the design of cognitive psychology experiments, but you won’t find it written down anywhere.

Seymour Benzer (crippling the Salk Institute)

One of the most fascinating stories in Benzer’s oral history interview is about construction of the Salk Institute in La Jolla, California:

Benzer: Louis Kahn [the architect] asked Salk, “How much money have you got to put into the building?” And Salk said, “Ten million for endowment” — this was all from the commitment from the March of Dimes — “and a million dollars a year for operating expenses in perpetuity.” So Kahn went home and designed a building for $20 million. In fact, he bragged about this at some dinner he had in La Jolla. He talked about other buildings he had designed and he said it was always his policy to make the building for twice as much as the money available, because you could always count on the fact that people scurry around to find the extra money.

Salk went for that idea on the argument that later on it would cost much more to build it. That was absolutely true. But at the time it had the effect of liquidating the endowment. And everything suffered from then on. The institute . . . was always worrying about where the next buck was coming from.
. . .
Interviewer: So they liquidated their entire endowment to construct a more expensive building?

Benzer: Yes.

Kahn knew a general principal about human nature that I do not. Why do backers reliably “scurry around to find the extra money”? Something powerful is at work here.

Seymour Benzer (part 1)

I found a long interview with Seymour Benzer, a biologist at Caltech, who is one of my favorite scientists — lots of creative and important work. I was pleased to learn he is a foodie. During a 1956 trip to Japan he had sushi for the first time. “One of the greatest things about the trip,” he said in 1990 (when the interview took place), presaging a future in which every upscale American supermarket sells sushi. (For dinner tonight I made salmon tartare.) When I was a student at Caltech I knew the other students liked him, but I never met him.

Benzer began the use of fruit flies to study behavior. At Woods Hole I took a course called Neural Systems and Behavior with a fruit-fly segment taught by Laurie Tompkins. She had met Benzer at a party. When she told him she studied fruit-fly mating, Benzer asked if they have orgasms. Very early in his work on behavior, he gave a talk to Roger Sperry’s lab about his plans. After his talk there was a lot of debate about it. Some people thought it was very promising; others thought it was nonsense.

Interviewer: Why were people so skeptical?

Benzer: Why? A lack of imagination.

Excellent answer. I would have said: People are always skeptical.

Spices: A New Kind of Vitamin?

For Shangri-La dieters who randomly spice their food, the current issue of the American Journal of Clinical Nutrition (July 2006) has very good news. Spices are a better source of antioxidants than any other food group, according to a survey of popular American foods.

There are many reasons to think antioxidants are beneficial. Oxidative damage, which antioxidants reduce or prevent, seems to play a role in many major diseases, including heart disease. Yet large trials in which people were given a few antioxidants, such as alpha-tocopherol and beta-carotene, did not find health benefits. Maybe the reason for these failures is that you need a suite of antioxidants; maybe antioxidants, “which cooperate in an integrated manner in plant cells [to reduce oxidative damage], also cooperate in animal cells,” the authors write. “A network of antioxidants with different chemical properties may be needed for proper protection against oxidative damage.” A very plausible idea.

To test this idea, it would help to know the antioxidant content of everyday foods. This is what the researchers tried to find out. They used a chemical assay to measure the total antioxidant content of 1113 popular American foods, chosen based on a careful national survey.

Here are the top ten foods by antioxidant content (per gram): cloves, oregano, ginger, cinnamon, tumeric, walnuts, basil, mustard, curry powder, pecans. Here are the next ten: baking chocolate, parsley, molasses, pepper, artichokes, dark chocolate, blackberries, whole-grain cereal, cranberries, chocolate pudding mix. Chocolate is also high in antioxidants — more good news. Red wine was #30. (White wine was low.)

Lowest on the list were animal products. “In general, plant and plant products in the diet have a much higher antioxidant content than do animal products,” the authors wrote. Oils, such as canola oil and olive oil, were higher than animal products, but less than other plant products. Cooking (heating) increased the antioxidant activity of plant foods such as carrots, tomatoes, and spinach.

The end of the paper describes evidence that higher intake of antioxidants is associated with lower risks of stomach cancer and lung cancer.

Could vitamins plus fiber plus spices provide most of the health benefits of fruits and vegetables? It is entirely possible. If so, it would be a major nutritional advance. Spices would be a new kind of vitamin. Good nutrition would include at least one heavily-spiced meal per day.

Berkeley Public Library Watch:The Shangri-La Diet, 3 holds on 5 copies. The Omnivore’s Dilemma by Michael Pollan, 128 holds on 29 copies. Website Watch: Distinct hosts served at sethroberts.net, latest 24-hour period: 832. One month ago: 539. Distinct hosts served is close to the number of different visitors.

The Ecology of New Ideas

A curious feature of the Shangri-La diet is how much its spread has been helped by things that did not exist a few years ago.

First, open access. My article with the data and ideas behind the diet was published in an “open-access” journal and stored in an “electronic repository.” Thus anyone with Internet access could read the article. The repository now has about 12,000 articles; mine was Number 117.

Second, blogs. Interest in this article was greatly amplified by blogs. My friend Andrew Gelman blogged about it. His post was read by Alex Tabarrok, who wrote about it at Marginal Revolution. His post was read by Stephen Dubner and led to a Freakonomics column in the New York Times — a great way to get book publishers’ attention. After the column (sadly eclipsed by Hurricane Katrina), a few blogs focussed on the diet and helped me weave a fuller view of its effects into the book I soon got a contract to write. When the book was published, quite a few bloggers had already heard about its main idea, which rendered its very strange concept slightly less strange, i.e., more acceptable. Now it is being discussed and tried in several blogs (see Blogroll)

Third, forums — the Shangri-La diet forums at sethroberts.net. At a talk about user interfaces a few years ago, I heard a famous designer say that new devices went through three stages of use: (a) hobbyist; (b) expert; and (c) mass market. Departments of electrical engineering, he said, were good at providing products for the first two stages, but were poor at making mass-market products. As far as the Shangri-La diet is concerned, this is what the sethroberts.net forums have done so well: made the diet acceptable to almost anyone. They have made the oil easier to drink, answered all sorts of common questions, and provided reassurance (it may sound crazy but it works), expert advice, and support.

Recently I heard Yochai Benkler, a professor at Yale Law School, speak on “The wealth of networks: How social production transforms markets and freedom” at an MIT symposium. This example supports his general point that new network-related products (such as open access) are empowering the little guy — the little guy here being me, who never got a large grant to support this research.

Why did these three new things (open access, blogs, and forums) all start at roughly the same time? Of course all of them were made possible by the growth of the Internet but so were a billion other things that haven’t yet come to pass. I have been working on a theory of human evolution that says language evolved because single words helped people trade. I think the growth of the Internet has been caused by the modern version of just that — better connection of buyer and seller. But open access, blogs, and forums have nothing to do with commerce. I think all three arose from another basic human tendency: a desire to share our enthusiasms. During the early days of electronic discussion groups (called bulletin boards), I was greatly disappointed that not one was devoted to Spy magazine. Why did we evolve this basic tendency? Because it led to the beginning of science — the intertwined growth of knowledge. So it makes quite a bit of sense that these three new things together acted in a kind of scientific way, bringing an effective weight-loss method out of darkness.

Berkeley Public Library Watch:The Shangri-La Diet, 4 holds on 5 copies. The Omnivore’s Dilemma by Michael Pollan, 116 holds on 7 copies. Website Watch: Distinct hosts served at sethroberts.net, latest 24-hour period: 1494. One week ago: 888. Distinct hosts served is close to the number of different visitors.