I learned of The Human Factor: Inside the CIAâ€™s Dysfunctional Intelligence Culture by Ishmael Jones (a pseudonym) from an interview on the New Yorker website. This comment by the author interested me:
I could say much the same about science: Once it became a place to get rich (or at least get large grants), effective science became a lot less common. A great deal of science is done by drug companies. They pay a lot. Some of their scientists are surely brilliant but their talents are wasted by the need to find solutions that will be highly profitable. My self-experimentation found solutions that cost nothing and make far more intellectual sense. I was able to do something that didn’t produce a lot of publications because it wasn’t my job.
Many skills make good full-time jobs. Science doesn’t. There is too much pressure for short-term results. Without short-term results, you may lose your job or your grant. (Or, in China, most of your income.) Nor is science a good source of status. If you want your science to provide your status, you will be under great pressure to conform. Yet for practically all scientists, it’s their full-time job and their main source of status. This may not make it impossible for them to do good work but I suspect it comes close to doing so. My self-experimentation was effective not only because it was fast and cheap (per experiment) but also because I could be slow (per publication) and do something low-status.