Hi, I'm Richard Nisbett. The course is Critical Thinking for the Information Age. This is lesson number four on experiments. Last week we talked about correlations, how to find relationships among variables. But correlations, as it may have become clear in that lesson, have a great weakness. They normally can't tell us about causality. Usually, we have to conduct an experiment in order to be sure about causality. The definition of an experiment is, it's a scientific study in which at least one variable is manipulated, and at least one variable is measured. For example, if we take an experiment that was done decades and decades ago, very important one, give some people iodine and others no iodine and examine the incidence of thyroid problems, which used to be a very serious problem. It turns out if you give people iodine they don't have thyroid problems so much. And, as a result, when you buy salt it's going to say iodized. And that's to protect your thyroid. Another example from psychology might be, pay some children for drawing with Magic Markers. And then examine the extent to which they draw with them a couple of weeks later when they're not paid. What you'll find is that if you pay kids for doing something like playing with magic markers, they're less interested in doing it. You've turned it into work by paying them for doing it. Both of these experiments are examples of the so-called gold standard of research, which is the randomized control design. The experimenter assigns people, or things, at random to the experimental condition versus the control condition. And if you don't have this requirement met, you don't have an experiment, basically. You just have, you're just fiddling around, because assignment at random means that, on average, you've got the same batch of people in your experimental condition or the same batch of things, as you do in your control condition. The treatment is the independent variable. The control condition receives a different treatment, or no treatment at all. The things that are measured constitute the dependent variables. There can be many different experimental conditions, and many different control conditions. Doesn't have to be just one of each. Let's look at some examples of questions that can be examined either with correlational research, or experimental research. A lot of health research is of that kind. Multiple, multiple correlational studies of a particular health practice and, typically, a very few experimental studies, because it's a lot easier to do correlational studies. You ask people, do you eat fish? How often? And then you see what their cardiovascular condition is. You can do that kind of study without ever leaving the armchair. And then you compare people who happen to eat one type of food with people who don't eat that type of food. You can compare people who get exercise with people who don't get exercise. You compare people who take a particular supplement with those who take no supplement. Or, you can carry out an experiment in which we have randomly selected people eat a certain type of food for a period of time and have others not eat that type of food. The same for supplements. You might think that correlational research and experimental research would normally reach the same results. You can say an awful lot of sentences that are the same about those two types of studies. But, in fact, it's surprisingly often the case that it's not true that the correlation result is the same as the experimental result. So let's look at the case of Vitamin E and prostate cancer. Vitamin E, in the correlational studies, turns out to be correlated with a lower likelihood of prostate cancer. So, naturally, we're beginning to think, maybe Vitamin E lowers the likelihood of prostate cancer. But, there are all kinds of differences between people who take Vitamin E and those who don't. Sounds like a health thing to do, take that vitamin, and people with more money are more likely to be taking the Vitamin E. And more money is associated with lower mortality for almost everything. People who are more educated, paying attention to what the health journals are recommending are more likely to do that. More educated people tend to take better care of themselves, have better diets and so on. So, it's really very problematic to just say, people who take Vitamin E are less likely to have prostate cancer. because there's so much else that could be different, so much in the way of a confound, like we talked about in the last lesson. What you have to do is to apply the Gold Standard of Research, which is the randomized controlled experiment. You flip a coin, proverbially, some times these days you would use a random generator device to see who gets the experimental treatment and who gets the control treatment. And then you measure the outcome. Now, the beauty of the experiment is that we know that people in the experimental condition are essentially the same as the people who are in the control condition. Because they had an equal chance of being in one condition versus the other. So, we're going to have equal social class, on average, equal number of rich people between experimental and control conditions. Identical diets, on the average. going to, just people who have a particularly good diet, they're going to be equally like you to be in either condition. So when you do this experiment, you get rid of all the possible confounds, and what you find is that Vitamin E makes prostate cancer more likely. This kind of pattern happens quite often. For example, with hormone replacement therapy for menopausal women, couple three decades ago a doctor started giving women hormone replacements as their hormone status changed, and it was found that women who had the hormone replacement therapy had less cardiovascular disease. So hundreds of thousands of women, partly on the basis of that finding, were given hormone replacement therapy. When you do the experiments it turns out hormone replacement therapy increases cardiovascular illness. The same kind of story's told for supplements of all kinds. Selenium for prevention of cancer. You know how the people who take selenium just because they thought it was a good idea are less likely to have cancers of various kinds, but the experiments show it's useless. The same thing is true is for glucosamine for arthritis, Vitamin C for prevention of the common cold, Vitamin B12 for cognitive function in the elderly and so on. In fact, most of the supplements that have ever been tested, you find effects in the correlational studies but not in the experiments. And when you do find effects in the experiments, they're usually much larger in the correlational studies. This generalization, by the way, all of these generalizations that I've just been making, apply to healthy people with an adequate diet. If you've got a selenium deficiency, you're going to be better off if you get some selenium. But if you're just an ordinary healthy person with an ordinary healthy diet, very few supplements are going to do you any good. There are actually 50,000 supplements on the market. Hardly any of them are tested. Our government does not require testing of supplements. It doesn't even require you to put selenium, or whatever, in your pill. You can put saw dust, and indeed, there some major chain stores where you're going to get saw dust rather than selenium. And here's a fairly exhaustive list of the ones that have actually been found effective. Probiotics for gut health, zinc for shortening duration of colds, CoQ10 for cardiovascular health, fish oil for cardiovascular health and various other purposes, zinc for a number of purposes, Vitamin D3 for prevention of fractures in the elderly. If your supplement is not on this list, I suggest you go to Cochrane Reviews or some other medical website which can tell you what would be a good supplement to be taking. This applies not just to supplements, but every kind of health practice. You may have heard that it's very important to have breakfast, because you're healthier that way and less likely to be obese. And the theory about that is that, if you don't have breakfast, you're starving later in the day, so you over eat, you over shoot what you should have. And, in fact, all the correlational studies show that. But the experimental studies show there's no difference. So if you want to skip breakfast, go ahead. It will not make you fatter. But anyway, where do you suppose the idea came from that it's very healthy to eat breakfast and very important for keeping weight down? The idea of John Harvey Kellogg, who wrote this hypothesis, or actually, he wrote it as a finding, in his health journal. Mr. Kellogg was the inventor of Corn Flakes. So there may have been a connection with why he felt it important to tell people that breakfast is a big deal. So what's going on? Why are the correlation studies finding one thing and the experimentals another? Well, Vitamin E is confounded with social class, with education, with better health practices and with all that goes with those things. The problem here is called self-selection. People choose their level on the independent variable and they bring along with that their level on a host of other variables. So, I take Vitamin E, but that's, I'm paying attention, it sounds healthy. I can afford vitamins. I'm the sort of person who's very concerned about health. I eat a very healthy diet. I do exercise and so on. In fact, this bias is given the name, healthy user bias. Anything that people think is healthy is going to be associated with health, because the people who care about health, and can afford to be healthy, are doing it. If Science Magazine were to announce next month that people who eat brandied pretzels for breakfast live longer, in ten years, it would be true. The people who ate brandied pretzels for breakfast would live longer, because they read Science Magazine, or the New York Times, which reports that to them. You can even speak of self-selection for things. It sounds a little bit weird, but you can talk about that Ford pickup as self-selecting its driver. The Volvo wagon as self-selecting its driver. And they're self-selecting different kinds of drivers. The people who did those correlational studies, by the way, probably didn't think that the experiments were necessary because they didn't look just at one correlation. It's not quite as simple as the way I described them. They didn't just look at people who took Vitamin E and looked to see whether they had prostate cancer. They did what's called controlling for possible confounding variables by using multiple regression analysis. Multiple regression analysis examines the association of each of a number of variables with the target independent variable, X, with a target dependent variable, Y. The additional variables are controls. Multiple regression analysis looks at the correlation of X and Y, controlling for all variables that are correlated with both X and Y. It does this by subtracting out of the correlation between X and Y the correlation between each of the other independent variables with both the X and the Y variable. There are huge problems with multiple regression analysis. Typically you can't identify all the possible variables that might be correlated with the target independent variable and the target dependent variable. You often can't measure some variables you really assume are important. And often you measure them with such poor reliability or validity that the results are distorted. If you measure a variable with zero validity, so that it has no real meaning at all. If you control for it, you'll find it doesn't make any difference, so you control for that variable. Well you didn't control for it if it has no validity. And many controls have very low validity in practice. And it's meaningless to say you control for variables with missing variables, which is often the case. How many old ladies are driving pickups? How many tough young guys are driving powder blue Volvo station wagons? I want to stress two things about multiple regression analysis. Most science and health writers don't understand the extreme weakness of multiple regression analysis. And they don't know when to alert you as to whether the method used in this study was dubious, namely based on multiple regression analysis, or solid, based on the goal standard experiment. And as a matter of fact, many scientists don't understand the weakness of multiple regression analysis. They do it feeling like they have done their controls and they feel confident that the result is meaningful when it often isn't. You're constantly being told by the media about research which is meaningless or worse. Ask yourself about a reported finding, is this correlational or experimental? If the report doesn't say which it is, probably it's correlational. And if the finding is correlational, it's quite likely to be wrong. At least when the finding concerns biological research, or research on human behavior.
