How Bad is The Secret?

Not as bad as you think. The Secret, of course, is the huge best-seller (now #2 on Amazon) that claims you can get what you want by applying “The Law of Attraction” — namely, that if you think about something it will come to you. According to Wikipedia, “there have been no widely recognized studies demonstrating that the [Law of Attraction] actually works.” The book has been — not to put too fine a point on it — ridiculed, for example by the author Barbara Ehrenreich.

I learned about The Secret last July from my friend Sarah Kapoor, who made a CBC segment about the Shangri-La Diet. She told me about nine YouTube spots (Parts 1-9), each 10 minutes long, that together made a movie. I watched only Part 1 (now unavailable). It wasn’t enjoyable. It seemed like a parody of a film about science, and not a funny one. Sarah said it was growing like wildfire but at the time the segments had received only a few thousand views so I wondered what she was talking about. Time has proven her correct.

Is The Secret complete nonsense? It sounds like complete nonsense, the writers of Wikipedia apparently think it’s complete nonsense (“no widely-recognized studies . . . “), Barbara Ehrenreich thinks it’s complete nonsense (she calls it “mass delusion”), but I don’t think it actually is complete nonsense. Around 1980, Robert Cialdini, a psychology professor at Arizona State, and Kathleen Carpenter, an undergraduate, did a remarkable study. They gave Tempe residents one of two paragraphs to read about the benefits of cable TV (a new thing at the time). One was a dry statement of the benefits; the other asked the reader to imagine partaking of the benefits (“take a moment and think of how . . . you will be able to spend your time at home, with your familly, alone, or with your friends”). A month later, these residents were offered the choice of whether to get cable TV or not. Of those given the dry information, about 20% subscribed; of those given the “take a moment” statement, about 50% subscribed. A huge difference, with nontrivial monetary consequences, from what seems like a tiny treatment. The title of the published article, which appeared in Journal of Personality and Social Psychology (1982, vol. 43, pp. 89-99) was “Self-relevant scenarios as mediators of likelihood estimates and compliance: Does imagining make it so?” Imagining did make it so, in a surprising way. The effect is much too large to be dismissed. I don’t think it has been repeated, although I’m not sure.

I learned about this study from the excellent new book Made to Stick by Chip Heath and Dan Heath.

Life is Complicated

Yesterday morning I listened to Ira Glass. Yesterday evening I listened to Bill McKibben. And I reflected:

1. Bill McKibben wrote a whole book, The Age of Missing Information (1992), about the malign influence of TV. He spent a year watching a single day’s output of the 100-odd channels of one cable company. TV makes people self-centered, he decided.

2. Ira Glass said we are living in a Golden Age of Television and listed a handful of current shows — including The Wire, The Daily Show, Colbert, Friday Night Lights, Project Runway, Entourage, House, and “anything with Ricky Gervais” — in support of his claim. He has just spent a year starting a TV version of This American Life.

3. Bill McKibben wrote an article (in The Nation) praising This American Life to the skies.

I think of McKibben and Glass as the two Boy Geniuses of American intellectual life. (Curiously I cannot think of any Girl Geniuses.) Both of them did great work while really young. When McKibben was in his twenties, he wrote a long series of editorials in The New Yorker that were inspiring. (They were unsigned. I found out who wrote them by writing to the magazine.) His first book, The End of Nature (1989), about global warming, was prophetic. I think it was the very first general-audience book on the subject. As for Glass, This American Life was terrific right from the start, twelve years ago. He was 36 when it started.

Charles Murray vs. Charles Murray

The Bell Curve (1994) by Richard Herrnstein and Charles Murray, which argued that IQ is destiny, was the most IQ-glorifying book since . . . well, ever. Now Mr. Murray has taken a big step away from his position in that book, yet he continues to glorify IQ.

In today’s Wall Street Journal, Mr. Murray wrote an op-ed piece (“What’s wrong with vocational school?”) with which I mostly agree. His main point is that for most students, college is a waste of time. As a college teacher (at Berkeley), I have seen that all too clearly. Mr. Murray has an unfortunate way of stating his position. “A four-year college education teaches advanced analytic skills and information at a level that exceeds the intellectual capacity of most people.” I’d put it differently: A four-year college education teaches analytic skills and information at a level that exceeds the interest of most people. I am sure that if my students or anyone’s students were more interested in the material, they would learn it better. That most college students are not interested in the same things as most college professors is a good thing, economically speaking. A healthy economy is a diverse economy; a diverse economy requires a wide range of skills and knowledge, much wider than the narrow skills and knowledge possessed and taught by college teachers. But it is a bad thing for the students and teachers, who are trapped. They have to be there. I feel worse for the students, of course — they are paying to be there.

It isn’t complicated: IQ tests were designed to predict school performance. They do. People with higher IQs do better in school. To believe in the value of IQ is to believe in the school system it reflects. To glorify one is to glorify the other. Now Mr. Murray has taken a step away from one (the school system) but not the other (IQ). Well, nobody’s perfect.

Were I grading The Bell Curve, I would give it a B. The sad truth is that its basic conclusion, that a high IQ is really helpful, is entirely correct. A better book would have replaced the wacky genetic chapter with an attempt to understand why IQ matters so much. In a world where we place less weight on successful completion of college — the world that Murray now advocates — IQ will matter less.

In The Nature of Economies, Jane Jacobs pointed to the stultifying effects of discrimination. “Macho cultures typically have pitiful, weak economies,” she wrote. “Half their population, doing economically important types of work, such as cooking and food processing . . . are excluded from taking initiatives to develop all that work [e.g., open a restaurant] — and nobody else does it, either.” IQ discrimination is also stultifying. If our society did a better job of helping students who are not good at college — helping them find jobs where their abilities shine, instead of wasting four precious years of their lives — the entire economy would benefit.

Methodological Lessons from Self-Experimentation (part 4 of 4)

6. Curiosity helps — because it provides a wide range of knowledge. Pasteur made a similar point when he said luck favors “the prepared mind” by which he meant the well-stocked mind. To come up with my theory of weight control you needed to know both obesity research and animal learning because the theory is based on basic facts about weight control and basic facts about Pavlovian conditioning. I knew the weight control facts because I had taught introductory psychology and lectured on weight control. I knew the basic facts about Pavlovian conditioning because my graduate training was in animal learning. It was unusual to know both sets of facts. Few obesity researchers knew much about animal learning; few animal-learning researchers knew much about weight control. The same thing happened with my mood research: Facts that I had learned from teaching introductory psychology showed me that my findings made sense and were important. I had taught introductory psychology because I was curious about psychology.

These two examples (weight control, mood) surprised me. I may have heard this point made a few times but I didn’t know any examples. Since then, however, I have come across examples not involving me that make the same point. Luca Turin is a biophysicist who has come up with a far better explanation of how the nose works than any previous theory. His recent book The Secret of Scent tells the story. “In order to solve the structure/odor problem,” he wrote, “you need to know at least three things: (a) biology, (b) structure and (c) odor. Each of these three things taken individually is not difficult” (p. 166). The problem had gone unsolved because no one before Turin knew all three.

7. Publish in open-access journals. Because my long self-experimentation paper was published in an open-access journal, anybody could read it within minutes. My friend Andrew Gelman blogged about it, which caused Alex Tabarrok at Marginal Revolution to mention it. This brought it to the attention of Stephen Dubner, who with Steven Levitt wrote about it in their Freakonomics column in the New York Times. That led to a contract to write two books — one about weight loss, the other about self-experimentation in general. That anyone could download my paper made it spread much faster. In the old days, with photocopies and libraries and mailed reprints . . . no talk tonight.

A summing-up, if you want to figure something out via data collection: 1. Do something. Don’t give up before starting. 2. Keep doing something. Science is more drudgery than scientists usually say. 3. Be minimal. 4. Use scientific tools (e.g., graphs), but don’t listen to scientists who say don’t do X or Y. 5. Post your results.

Read Part 1, Part 2, and Part 3. You no longer need to register to comment. My talk Tuesday night (tomorrow Jan 9) 7:30 pm at PARC (Palo Alto) is open to the public.

The Decline of Harvard

In high school, I learned a lot from Martin Gardner‘s Mathematical Games column in Scientific American. I read it at the Chicago Public Library on my way home from school while transferring from one bus line to another — thank heavens transfers were good for two hours. In college, it was long fact articles in The New Yorker. Now it’s Marginal Revolution, where I recently learned:

Harvard has also declined as a revolutionary science university from being the top Nobel-prize-winning institution for 40 years, to currently joint sixth position.

The full paper is here.

What should we make of this? Clayton Christensen, the author of The Innovator’s Dilemma (excellent) and a professor at the Harvard Business School, has been skeptical of Harvard’s ability to maintain its position as a top business school. He believes, based on his research and the facts of the matter, that it will gradually lose its position due to down-market competitors such as Motorola University and the University of Phoenix, just as Digital Equipment Corporation, once considered one of the best-run companies in the world, lost its position. A few years ago, in a talk, he described asking 100 of his MBA students if they agreed with his analysis. Only three did.

How would we know if Harvard was losing its luster? Christensen asked a student who strongly disagreed with him. Harvard business students (except Christensen’s) are taught to base their decisions on data. So Christensen put the question like this: If you were dean of the business school, what evidence would convince you that this was happening and it was time to take corrective action?

When the percentage of Harvard graduates among CEO’s of the top 1000 international companies goes down, said the student.

But by then it will be too late, said Christensen. His students agreed: By then it would be too late to reverse the decline.

Christensen’s research is related to mine, oddly enough — we both study innovation. For explicit connections, see the Discussion section of this article and the Reply to Commentators section of this one.

Web Trials

Thanks to Rey Arbolay, at the Shangri-La Diet forums, the eternal question “will this help?” is being answered in a new way. The specific question is “will the Shangri-La Diet help me lose weight?” The new way of answering it is that people are posting their results with the diet in the Post Your Tracking Data Here section of the forums. What they post is standardized and numerical enough that ordinary statistical methods can be used to learn from them. I’ll call this sort of thing a web trial.

It’s a lot better than nothing or a series of individual cases studied separately. I learned a lot from my most recent analysis — for example, that people lose at a rate of about 1 pound per week after Week 5. I couldn’t have done a good job of predicting where any of the fitted lines on the scatterplots would be or the size of the male/female difference. Nor could I have done a good job predicting the variability — the scatter around the lines.

It’s a lot worse than perfection. It would be much better if a comparison treatment (in the case of SLD, a different way of losing weight) was being tested in the same way. Then results from the two treatments could be compared and you would be closer to answering the practical question “what should I do?” (That modern clinical trials — very difficult and expensive — still use placebo control groups although placebos are not serious treatment options is a sign of . . . something not good.)

I can imagine a future in which people with a health problem (acne, insomnia, etc.) go to a website and enroll in a web trial. They told about several plausible treatments: A, B, C, etc., all readily available. They are given a choice of (a) choosing among them or (b) being randomly assigned. They post their results in a standardized format for a few weeks or months. Then someone with data analysis skills analyzes the data and posts the results. As for the participants, if the problem hasn’t been solved they could enroll again. This would be a way that anyone with a problem could help everyone with that problem, including themselves. The people who set up the trials and analyze the results would be like the book industry or Wikipedia insiders — people with special skills who help everyone learn from everyone.

Books Were the First Open-Source Software

Here is Aaron Swartz on Wikipedia:

When you put it all together, the story becomes clear: an outsider makes one edit to add a chunk of information [to a Wikipedia entry], then insiders make several edits tweaking and reformatting it. In addition, insiders rack up thousands of edits doing things like changing the name of a category across the entire site — the kind of thing only insiders deeply care about. As a result, insiders account for the vast majority of the edits. But it’s the outsiders who provide nearly all of the content.

(Correcting Wikipedia’s founder, by the way.) When I visited my editor, Marian Lizzi, at Penguin, I realized that book publishing is exactly the same: Outsiders write the books, insiders edit them.

The curious thing about book publishing is similar to what Swartz noticed in a different realm: The content, the crucial stuff, is entirely from amateurs. No other industry, with the possible exception of craft shows, is like this. If I run a deli, I buy supplies and food from people who make their living selling supplies and food. If I make clothes, I buy my cloth from professional cloth makers. If I make cheese, my milk comes from professional farmers. Only book publishers endlessly deal with amateurs.


Science Versus Human Nature

Last weekend I saw the writer Thomas Cahill on Book TV. He mentioned his book How The Irish Saved Civilization. The real contribution of the Irish, he said, wasn’t that they saved the sacred texts, it was that they brought humor to their study. “They brought irreverence to reverence,” he said. “That was entirely new.”

This reminded me of Brian Wansink‘s comments about cool data. His notion that research designs should be judged on their coolness was entirely new to me. I’m not the only one; the Wikipedia entry for scientific method says nothing about it. Using cool and research design in the same sentence is quite a bit like bringing irreverence to reverence. Once somebody says it, though, it makes sense. I remember being thanked after an interview; I replied that there’s no point doing the research if no one ever learns about it. Coolness obviously plays into that — influences the chance that other people will learn about it.

I think most scientists will agree with Wansink, that coolness matters. I think you don’t find his idea in books and articles about scientific method not only because there is so little written about research design (at least compared to the amount written about data analysis) but also because it appears undignified. “I’m important, I shouldn’t have to worry about being cool” is the (very human) unspoken attitude.

Brian Wansink on Research Design

An experiment in which people eat soup from a bottomless bowl? Classic! Or mythological: American Sisyphus. It really happened. It was done by Brian Wansink, a professor of marketing and nutritional science in the Department of Applied Economics and Management at Cornell University, and author of the superb new book Mindless Eating: Why We Eat More Than We Think (which the CBC has called “the Freakonomics of food”). The goal of the bottomless-soup-bowl experiment was to learn about what causes people to stop eating. One group got a normal bowl of tomato soup; the other group got a bowl endlessly and invisibly refilled. The group with the bottomless bowl ate two-thirds more than the group with the normal bowl. The conclusion is that the amount of food in front of us has a big effect on how much we eat.

There are many academic departments (called statistics departments) that study the question of what to do with your data after you collect it. There is not even one department anywhere that studies the question of what data to collect — which is much more important, as every scientist knows. To do my little bit to remedy this curious and unfortunate imbalance, I have decided to ask the best scientists I know about research design. My interview with Brian Wansink (below) is the first in what I hope will be a series.

SR: Tell me something you’ve learned about research design.

BW: When I was a graduate student [at the Stanford Business School], I would jog on the school track. One day on the track I met a professor who had recently gotten tenure. He had only published three articles (maybe he had 700 in the pipeline), so his getting tenure surprised me. I asked him: What’s the secret? What was so great about those three papers? His answer was two words: “Cool data.” Ever since then I’ve tried to collect cool data. Not attitude surveys, which are really common in my area. Cool data is not always the easiest data to collect but it is data that gets buzz, that people talk about.

SR: What makes data cool?

BW: It’s data where people do something. Like take more M&Ms on the way out of a study. All the stuff in the press about psychology — none of it deals with attitude change. Automaticity is seldom a rating, that’s why it caught on. It’s how long they looked at something or how fast they walked. That’s why I’ve been biassed toward field studies. You lose control sometimes in field studies compared to lab studies, but the loss is worth it.

The popcorn study is an example. We found that people ate more popcorn when we gave them bigger buckets. I’d originally done all that in a lab. So that’s great, that’s enough to get it published. But it’s not enough to make people go “hey, that’s cool.” I found a movie theatre that would let me do it. It became expensive because we needed to buy a lot of buckets of popcorn. Once you find out it happens in real theatres, people go “cool.” You can’t publish it in great journal because you can’t get 300 covariates; we published it in slightly less prestigious journal but it had much greater impact than a little lab study would have had.

One thing we found in that study was that there was an effect of bucket size regardless of how people rated the popcorn. Even people who hated the taste ate more with the bigger bucket. We asked people what they thought of the popcorn. We took the half of the people who hated the popcorn the most — even they showed the effect. But there was range restriction — the average rating in that group was only 5.0 on a 1-9 scale — not in the “sucky” category. Then we used old popcorn. The results were really dramatic. It worked with 5-day-old popcorn. It worked with 14-day-old popcorn — that way I could say “sitting out for 2 weeks.” That study caught a lot of attention. The media found it interesting. I didn’t publish the 5-day-old popcorn study.

I’m a big believer in cool data. The design goal is: How far can we possibly push it so that it makes it a vivid point? Most academics push it just far enough to get it published. I try to push it beyond that to make it much more vivid. That’s what [Stanley] Milgram did with his experiments. First, he showed obedience to authority in the lab. Then he stripped away a whole lot of things to show how extreme it was. He took away lab coats, the college campus. That’s what made it so powerful.

SR: A good friend of mine, Saul Sternberg, went to graduate school with Milgram. They had a clinical psychology class together. The professor was constantly criticizing rat experiments. This was the 1950s. He said that rats were robot-like, not a good model for humans. One day Milgram and my friend brought a shoebox to class. In the box was a rat. They put the box on the seminar table and opened it, leaving the rat on the table. The rat sniffed around very cautiously. Cautious and curious, much more like a person than like a robot. It was a brilliant demonstration. My friend thinks of Milgram’s obedience experiments as more like demonstrations than experiments. But you are right, they are experiments consciously altered to be like demonstrations. Those experiments were incredibly influential, of course — it supports your point.

BW: When we first did the soup bowl studies, we refilled the soup bowls so that we gave people larger and smaller portions than they thought had. We heated the soup up for them but gave them 25% more to see if they would eat more than they thought. You could put that in an okay journal. The bottomless soup bowl would be more cool. Cool data is harder to get published and it’s much more of hassle to collect the data, but it creates incredible loyalty among grad students, because they think they are doing something more exciting. It’s more of military operation than if they are just collecting some little pencil-and-paper thing in the lab. It makes research more of an adventure.

Another thing: field experiments are difficult. There’s a general tendency in research to be really underpowered with things [that is, to not have enough subjects]. Let’s say you’re doing the popcorn bucket study. Is the effect [of bucket size] going to come out? Rather than having too many cells and not get significance, it’s a good idea to have fewer cells — replace a manipulated variable with one or two measured variables. For example, instead of doing a two-by-two between-subjects design we might have a design when one factor is measured rather than manipulated. If the measured factor doesn’t come out you haven’t lost anything; you still have all the power. With the popcorn study we knew the study would work with the big bucket [that is, we knew there would be an effect of bucket size] but we didn’t know if there would be an effect of bucket size if we gave them [both good corn and] bad corn [thereby doing a two-by-two study] and only 200 people showed up [leaving only 50 people per cell]. So when we did the field study for the first time, we gave them all popcorn 5 days old. We measured their taste preference for popcorn then used it as a crossing variable. We used scores on that measure to divide the subjects into two groups.

SR: Let’s stop here. This is great stuff.


Most of the VCP-310 professionals advice against PMI-001 and suggest going for EX0-101 instead, proceeding to CCIE-LAB finally.

Amazon Rank: The Poor Man’s BookScan

Calling all authors!

If you have written a book, you have probably wondered: What’s the connection between amazon rank and number of books sold? Well, wonder no more. Below is a graph based on The Shangri-La Diet. The copies-sold information is from Nielsen BookScan. Their website says:

Most of the nation’s major retailers for books are included in our panel of reporting book outlets: Borders and Walden, Barnes & Noble Inc., Barnes &, Deseret Book Company, Hastings, Books-A-Million, Tower Music and Books, Follett College stores, and Weekly sales information is also tracked from Mass merchandisers like Target, Kmart and Costco, along with smaller retail chains and hundreds of general independent bookstores.

The graph shows that the relationship between books sold and amazon rank is linear on a log-log scale (as so many things are — the physicist Per Bak wrote a whole book about such relationships). Each point is a different week. To illustrate the formula of the line,

ln(copies sold/week) = 9.67-0.53*ln(amazon rank),

the amazon rank of Send In The Idiots: Stories From the Other Side of Autism by Kamran Nazeer, a masterpiece about the adult lives of autistic children, is now 35,758. Predicted sales is 61 books/week.

Four Great Modern Books (part 1: description)

Last week I read Send In The Idiots by Kamran Nazeer. It was so good — so fresh, clear, and moving — it made me wonder how it came to be. In other words, where do great books come from? Asking what several great books have in common should suggest answers to this question.

Among books published in the last 40-odd years these are the best I have read:

The Economy of Cities
(1969) by Jane Jacobs. Why and how cities grow or fail to grow. How new goods and services arise. They almost always begin in cities — the book starts with a discussion of how agriculture began. Cities and the Wealth of Nations (1984) by Jacobs is also great but too similar to Economy to be worth separate discussion.

Totto-Chan (1981) by Tetsuko Kuroyanagi. A memoir of the author’s primary school days at a progressive Tokyo private school. (Mentioned in The Shangri-La Diet.)

The Man Who Would Be Queen (2003) by J. Michael Bailey. What scientists, especially Bailey (a psychology professor at Northwestern University), have learned about the causes and effects of male homosexuality. One chapter is about male transsexuals, who are not always homosexual.

Send In The Idiots (2006) by Kamran Nazeer. The beautiful subtitle is Stories From the Other Side of Autism. When he was a child, Nazeer attended a school for autistic children. This book is about the adult lives of several of his classmates.

In later posts I will explain why I like these books so much and try to ferret out the secrets of their greatness.