In the previous few discussions, we've talked about vision as a computational problem. We've looked at phenomena like recognizing objects in the world, at distinguishing the edges, the boundaries of objects, at depth perception. There's much more to say about vision. If I say, we're only touching on a few subjects in vision. But we've looked at some of the ways in what we think of as the automatic processes of vision can be illuminated. Perhaps, we can gain more understanding of them by looking at them as computational problems as things that we can try to write programs to do in a computer. We may not get all the answers about how human beings accomplished vision but the computational approach to vision, is very rich in that it enables us to gain more insight into how hard or easy certain elements of what we think of as an automatic task might be, we have to do vision virtually every second of the day or at least our waking day. By trying to re-interpret the vision problem computationally, we can learn a great deal. Today, we're going to talk about something related to vision but not exactly the same. This is the issue of mental imagery, and mental imagery it's not only a fascinating topic in its own right. But historically, it represents a tremendously important source of early controversies in the field of cognitive science in general. I don't know how much detail I'll be able to go into about that. But suffice it to say that in the early days of cognitive science, there wasn't a whole lot of attention paid to trying to make computational models of mental imagery because it ran against the grain of the programming that people did in those early years of cognitive science. Again, I'm not sure how much detail to go into about this, but the programs that cognitive scientists were most comfortable doing in those early years matched up much better with our discussions of problem-solving and search than they do with mental imagery. So what is mental imagery? Let's pause. I mean we have a colloquial or intuitive understanding of what mental imagery is. As far as we're concerned, we think of it as the ability to conjure up pictures in our heads, images of things that we are not in fact perceiving. So for example, I could ask you to form an image of a unicorn. You're not looking at a unicorn, you've never seen a unicorn, but you should most likely be able to form a mental image of one, a picture of one in your mind. Then you could even answer questions about it. You could answer questions like is the unicorns horn longer than its leg or something like that? I could pose questions to you about your mental image which you could then give consistent answers to. So mental imagery is a very interesting phenomenon, seems to be this cognitive ability. I'm speaking here very loosely intuitively, but that we can form pictures that we can look at in our heads, inspect, and then answer questions about. That talent has proven to be an extremely challenging subject for the cognitive sciences and for the computational representation of mind in general. We'll see how some of the controversies around mental imagery have played out. But first, before we even get into that, let's just talk about some of the historically, some of the ways in which people have thought about mental imagery. So here's some quotes. The subject of mental imagery has been a venerable subject in the philosophy of mind. So people who have talked about the philosophy of mind have often appealed to this ability that we seem to have of forming pictures in our head. In fact, for some of the philosophers of mind, mental imagery is the bedrock, it's the foundation or at least one major foundation of human thinking. So here's Aristotle. "The nature of memory and its process has now been explained as the persistent possession of an image in the sense of a copy of the thing to which the image refers." Before we go on, note that Aristotle was saying that the image is a copy, we see something and then we form a copy of it in our heads and then we're able to inspect that copy. There's a little bit of a difference here with the views of Thomas Hobbes. Thomas Hobbes, the seventeenth-century philosopher and what he was writing is, "After the object is removed or the eye is shot, we still retain an image of the thing seen, though more obscure than when we see it." This is from his famous book, Leviathan. Now, notice that he's adding something to Aristotle's description of mental imagery. He's saying that, we have a mental image of something, though it's a little more obscure than when we actually saw it. That's an interesting point and we'll elaborate on that as we go along. Finally, here's John Locke, later British philosopher who's saying the ideas of the nurse and mother are well-framed in their, i.e. children's minds. Like pictures of them, represent only those individuals. So throughout the philosophy of my throughout history of philosophy of mind, people have appealed to mental imagery as a recurring aspect and often a fundamental aspect of human thinking. I'm particularly interested in mental imagery in my own research because I'm interested in mathematics and science education, and there are repeated references from working scientists and mathematicians to the essentially testifying to the fact that they make heavy use of mental imagery in their own work. That is they often they will say things they think in images, or they mix images and words, or images and symbols as they're thinking. Jacques Hadamard, French psychologist and mathematician. He wrote quite interesting book in the middle of the 20th century. It's been translated and sometimes the title is a little different but in my translation, it's called the Psychology of Invention in the Mathematical Field. He interviewed, he talked to a number of mathematicians and physicists in that book. One of them being Albert Einstein. So here is a quote from Albert Einstein writing to Jacques Hadamard about the nature of his thinking. It's written this rather formal style. But you see we'll elaborate on what he's getting at here. "The psychical entities which seek to serve as elements in thought are certain signs and more or less clear images which can be voluntarily reproduced and combined. The above mentioned elements, are in my case, of visual and some of muscular type." That's an interesting addition. "Conventional words or other signs, have to be sought for laboriously only in a secondary stage, when the mentioned associate of play is sufficiently established and can be reproduced at will." Paraphrasing of this is consistent with what quite a number of mathematicians and physical scientists report is that their initial ideas are emerged from a play of images in their mind, and the symbolic or linguistic or algebraic representation of those ideas comes as a second step. As Einstein says, often as they come laboriously. It's effortful to translate these intuitive mental images of some mathematical idea or some physical phenomenon. It's difficult to translate those intuitions into symbols or algebraic equations and that takes place in a second step. At least this is the case as reported by many scientists, and if that's true, it's extremely interesting. It suggests that if we want to understand how scientific thinking proceeds, how creative science and mathematics is done, then one of the things we should understand is the nature of mental imagery. So naturally, as someone interested in education in those fields, I'm also interested in how people make use of mental imagery. In a book by Eugene Ferguson called Engineering and the Mind's Eye, he makes a case for actually teaching and sharpening students mental imagery if they're interested, in this case, in engineering. But I think he would agree that the same kind of idea applies to other natural sciences as well. Despite the low academic status of visual thought, It is an intrinsic and inseparable part of Engineering. So what have we got so far? We've got this phenomenon that is a little bit puzzling. That is to say, we are able to form pictures or images in our heads and answer questions about them, and throughout the centuries, people have made a variety of explanations of this. Often very loose or intuitive, and there are other people like Einstein who say, "well, this is important. This is important than my thinking." So we're not talking about some phenomenon that just maybe is some interesting little trick of the mind that maybe only of interest to scholars. This seems to be a very fundamental issue. One of the recurring questions about mental imagery is, what exactly is its relation to normal vision? The kind of visual tests that we've been talking about in the past few discussions. So again, we've talked about things like binocular vision for depth perception, object recognition, Edge recognition, and there are many, many others. Recognizing motion or color or texture, all kinds of interesting tasks and vision. What did those tests and vision tell us or fail to tell us about the activity of mental imagery? How are they similar, if at all, and how are they different? To begin with, let's look at some immediate questions. Just things that maybe could pique your interest about mental imagery, to see you start thinking about this question of, first of all, let's take Aristotle's idea, that the mental image is a copy of the things seen. We could begin by taking a pot shot at that statement by saying, "Well, lots of times we form mental images of things that are not copies, that we've never seen, like unicorns. But even if we take Aristotle's definition at his word, let's try something. Imagine, form a mental image in your mind right now of a tiger. Try to form a good strong mental image of a tiger. Now, how many stripes on the tiger? If you had a photograph in front of you of a tiger, you could answer that question by simply counting the stripes. You could count them. Give back an answer. With mental imagery, most people report that they can't do anything like that. They can say, "Well, I know there are stripes, and maybe more than three or four, but I can't hold the image strongly enough to sit there in front of it, place it in my mind vividly enough so that I can count the stripes and give you a distinct answer to that question. That already, by the way, resonates with the quote from Hobbes, who said that the image is there but a little more indistinct than it was when we actually see the object. So the tiger image is there and we might even report that it's pretty vivid, but it's not like looking at a photograph. A particularly interesting experiment about mental imagery was done, I think in the 1980s, and it involved forming images of ambiguous figures. The one that is most discussed is the duck-rabbit ambiguous figure. So I'm showing that here. This just a teeny little sketch. Actually, the original duck-rabbit ambiguous figure is much more detailed. But this sketch is for our purposes. I think it's a better example. So you look at that sketch and you can see, well, depending on how you interpret it, you could view this as a very crude sketch of a duck with the things on the left there being the beak or as a rabbit with things on the left there being the ears. Now, here's the experiment. People were shown this sketch very relatively briefly. Briefly enough so that they could just get a quick impression of it, but enough to form a mental image. So they were shown this particular picture briefly and then they were asked to form a mental image of it. I'm going to go back to the previous slide. Then, they were asked what was in the image that they had in their minds. People would either answer duck or rabbit. So let's say you would ask someone, what's your image of? I just showed you a picture. What do you think that's an image of? They would say duck. Then you could ask them, stare at the image in your mind. Can you also see a rabbit? Can you re-interpret your image to see a rabbit? In general, and I'm speaking in general here because people have done variations of the experiment and there have been some arguments about various forms of the experiment, but I think it's very safe to say the consensus is, this is the way the phenomenon works. So when people said that they saw a duck in their image, they were asked if they could re-interpret the image in their heads, and in general, they would say no. Then [inaudible] you would say, can you also see a rabbit? People would say, "No. I can't see a rabbit. I see a duck." Interestingly, people were then asked to draw out on a piece of paper, the image that they had in their mind. Once they drew that image out on a piece of paper and then looked at it visually, then they could say, "Yeah. I see how that could be a rabbit." In other words, they could re-interpret the image once it was drawn out, but not when it was just in their heads. Now, this too is counter-evidence to the idea that mental imagery is just like staring at a photograph. If it were like staring at a photograph, then it should be very easy to say. Oh yeah. I thought this was an image of a duck but now that I've looked at the photograph like image in my head, I can see that it could also be a rabbit. That doesn't happen. So something different is going on with mental images than with just plain vision. A similar experiment has been done with ambiguous images, figures. So you can show people something like the Star of David and ask them to form a mental image of it. So you don't keep the image in front. You show them that, again, much like the duck-rabbit example. You show them this fairly, briefly ask them to form a mental image of it. Now, once they form the mental image of it, you can then ask them, is this element part of your image, a portion of your image? Typically, people will report, yes, that's a part of my image. You could then also ask them, is this a part of your image? People will have a much more difficult time saying whether that's a portion of that image. They they may well say no. That's not a portion of the image. Again, when the full picture is placed in front of them, they can see that the parallelogram is indeed part of the Star of David image. But that only becomes apparent when they're looking at the picture right there in front of them. So this is another case in which having an image doesn't seem to be quite the same thing as looking at an actual perceived object. There's something different about them. Finally, there's a philosophical objection that people would often make. In this case, to the classic theory about mental images. Something a little bit like not only Aristotle, but also Hobbes and Locke, which is that if you think of this scenario as having a picture in your head that you're looking at, then your model is like a little person inside your head looking at a movie screen or something like that. Then the question comes up. Well, what's that little person looking with, their own eyes? If they're looking with their own eyes, does that little person also have mental images? Does that person have a little person inside them? So forth. You get into these questions of infinite regress. It's sometimes called the homunculus objection which is to say that sitting in our brains, or sitting in our minds is a little guy, a homunculus, who's looking at pictures. But then what's inside the mind of that little guy, yet another little guy? So there's something very philosophically unsatisfying about the the Aristotelian portrait of imagery. Before getting to some of the debates around the nature of mental imagery, what I'd like to do though is give you some of the landmark experiments, which are part of the shared culture around the understanding of mental imagery and which informed and sometimes inflamed, those debates. So this is perhaps the most famous mental imagery. It's one of the most famous cognitive psychology experiments. It was done by Shepard and Metzler in the 1970s and it was a mental rotation experiment. Here's the idea. It's worth taking a little bit to understand this. What you're looking at here are three pairs of drawn 3D objects. In other words, in each of these little frames, there are two sketches of three-dimensional objects. In two of those cases, you could view those two objects as being the same object but rotated so that they look different. For example, in that case on the upper left, you can see that if you take the object on the left and rotate it clockwise, more or less, you get the the object on the right in that first panel. In other words, you could view those those two sketches as pictures of the very same object, but with a rotation difference between them. The pair of objects at the bottom is similarly the same object, but you would have to take the object on the left and rotate it. In this case, you're rotating it around an axis. The axis is in the paper but you're rotating it around that axis so that it goes out of the plane of the screen. But then you would find that those two objects are the same. The two objects in the middle frame there, the one toward the right, they cannot be rotated into each other. So they are distinct objects. Now, here's what was going on. In the experiment, subjects were told to look at these pairs of objects and press a button that indicated whether one object could be rotated to coincide with the other object. So they were given pairs of objects like this and they were told to look at them and as quickly as possible, press a button to say whether one object could be rotated into the other so that they would be the same thing or if not, to press a button to say that they're not the same object. Now, there are a number of interesting things here, but here's what happened. I'm going to show you one results graph from this experiment. When two objects were the same object, we're going to forget about cases where the two paired objects are distinct. Let's just focus on those cases where the two paired objects happen to be the same object but with a rotation difference between them. When the two paired objects were the same, but they were distinct by a rotation, and in this case, it's a rotation within the plane of the screen. The time taken to answer whether the two objects were the same, was linearly related to the angular distance between the two objects. In other words, let's go back to the previous slide. Look at those two objects in the upper left. The time taken to answer that those two objects are the same was a linear function of the angular discrepancy between those two objects. If you had to rotate the left object 90 degrees to get it to coincide with the right object, people took not quite twice as long because there's a y-intercept on this graph, but they took longer than they did if you had to rotate the left object 45 degrees. That in turn, took longer than if you had to rotate the object 30 degrees. That in turn, took longer than if you had to rotate the object 15 degrees. In fact, the times taken to answer that these two are the same object falls on a straight line. I'm not a psychologist, a cognitive psychologists by trade, but I had very memorable conversation with a colleague in the psychology department, who told me that when a graph comes out that linear, that close to linear in a psychology experiment, it's a shocker. I mean, people don't expect graphs that come out that perfect. That's not to say the experiment has been replicated, and there are many variations of this experiment. These are the results that you get from the experiment. But the linearity of this graph is really striking. What would be the colloquial interpretation of this? It seems, and we'll have to unpack this a little bit, but it seems as though when people see two objects like those at the upper left here, they're taking one object and rotating it at a certain standard pace at a certain angular velocity, they're rotating it from the left view to the right view. Naturally, if you're doing that, it takes longer to rotate the object by 90 degrees than it would to rotate the object by 45 degrees. Just as it would in the physical world. Consider this though: for those computer programmers who may be watching this at the moment, there are ways of rotating three-dimensional objects in computational representations. In computer graphics, you can take a polyhedron and rotate it by applying a matrix to it. If it's a three-dimensional object, you apply a three by three matrix to the object. Now, applying a three by three matrix to the object, takes every bit as long whether you're applying a 90 degree rotation, or 45 degree rotation, or 30 degree rotation. Applying a matrix multiplication for computer graphics folks, takes the same amount of time regardless of how much you're rotating the object. So if you were as a computer scientist to say, look at those top two objects there at the left, I see that I could take the left object and rotate it into the right object by applying say a 90 degree rotation matrix. Then you should not get a graph like this. Because applying a 90 degree rotation matrix should take exactly the same amount of time as applying a 45 degree rotation matrix. That is to say, you wouldn't get a phenomenon where it appears that there's a smooth rotation from one form to another that takes longer if there's a greater discrepancy. It seems again to be colloquial about this, it seems that it's almost like there's a 3D representation of the object in our head, and we're taking it and rotating it with a certain consistent velocity. Once we've rotated it to coincide with the other object, we stop and say yes. That's a little odd. It's certainly is not the easiest thing to reconcile with a computer scientists standard ideas about how to rotate one object into another. Here's a similar mental imagery experiment that was done. The idea here, this is a famous experiment by Kosslyn, Ball, and Reiser, the idea here is that people were shown this sketch of an island, the sketch that you see at the top there. On the island, there are a number of objects. There's a tree, and the hut, and a well, and what looks like a patch of grass, or something like that, and a little pond, so people were asked to form and to look at this picture long enough, so that they could form a good strong mental image of it. Then the original picture was taken away. Then what people were asked to do with their image. Remember, we were saying that one of the things about images is you can ask questions about them. So people have the mental image of this island in their minds, and they were asked to do something like focus your attention on the tree on the island. Now, shift your focus to the hut and signal when your focus has been shifted from the tree to the hut. They would do that. Then they were asked, focus your attention on the tree, and shift your attention to I guess that grassy patch if that's what it is toward the upper part of the island. Then people would again, focus their visual, their image attention on the tree, and then in their image, shift their focus to the grassy patch. They would report the amount of time, they would report how long it took them to shift focus. The results as seen at the bottom here, indicated that the amount of time it took people to shift their focus from object A to object B was again, linearly related not quite as perfect as the Shepard-Metzler graph, but still pretty good. The amount of time needed to shift focus from one object to the other was linearly related to the distance, the Euclidean distance on the island map itself between the two objects. So if the distance between the tree and the grassy patch is, whatever, five centimeters, and the distance between the tree and the hut is two centimeters, then it would take longer to shift from the tree to the grassy patch than it would from the three to the hut. Again, intuitively, it seems like your explanation of this would be something like, well, it's almost like there's a little dot of attention that's sitting there on the island in our minds, and when we move that dot of attention from one place to another, it moves at a certain rate, and it takes longer to shift that dot of attention over a long distance than it does over a short distance. There's some things about this that should be a little troubling, and we could get back to them, when I first read about these experiments, I totally believed the results. I mean, that's not that the question. But I was thinking, from the computational standpoint, there's some hidden questions here. But let's mention one of them before we go on. One of them has to do with, let's look at this example again. You're told in your mental image to focus on the tree and then shift your focus to the grassy patch. In our colloquial interpretation of the experiment that had little dot of attention starts at the tree and then it travels upward to the grassy patch. You were then told to refocus your attention on the tree and shift your attention to the hut, and then your little dot and attention move to the hut. As we said, it takes longer to go the longer distance than it does to go to the shorter distance. All well and good. From the computational standpoint, how do you know when you're shifting the attention, that focus of attention? How do you know which direction to move? A naive computer program might start doing a search of the island. In other words, you've got your computational attention on the tree and you're told to now find the grassy patch. A naive computer program might start scanning the island back and forth looking for the grassy patch and then finally when it gets up to this part of the island saying, okay, I found it. That's not the same as just moving straight like an arrow from the tree to the grassy patch, it's more like doing a search or maybe you might spiral outward from the tree in the ever-increasing circles until you find the grassy patch. In other words, if you are writing a computer program to answer these questions, you might again answer it along very different lines and get very different results than the experimenters here did. Why is it that people whatever is in this image is enough to tell them which direction to go in scanning from one object to another? It's enough to tell them which direction to go but they still have to take more time in going from the tree to an object at a fair distance at this way than from the tree to an object at a modest distance this way. That's a little puzzling. The body of experiments that started coming out about mental imagery along these lines prompted tremendous debate in the cognitive science community. One camp within the cognitive science community which as a shorthand I'll call the symbol camp said that, and I'm making a strong version of their argument. It might be an extreme version for many of the people who would have called themselves in the symbol camp. It's the most extreme version but it's not a totally implausible caricature of this view. What they would say is, there is no visual element of mental imagery. It is not like looking at a picture and it's not like turning an object in our heads. What we have are symbolic representations of things like the 3D objects and the island here. When you are told to do tasks like move your attention from one object to another, you do that by a manipulation of symbols, not by scanning anything like a pictorial image in the mind. So that was one view and there were a number of arguments made for that view one being that we already know that we do have symbols in the mind, at least as far as or at least that symbols are a reasonable representation for many kinds of language or puzzled type situations. We can represent search in many cases reliably, we can represent it symbolically, we can represent many linguistic tasks symbolically. So one of the arguments for the symbol camp was why postulate of another pictorial representation if we don't need one? If we don't need a pictorial representation and we can show how we can answer questions like this through a purely symbolic representation, why not just stick with that? It's a economy, in some ways it's a, I wouldn't push this too far but it's a little bit like an Occam's razor type of argument. We're saying, we have one representation that gives us a simple portrait of the basis of the mind. If that's all we need, why postulate more complication than we need? Another group which I could call the visual list argued that though in fact experiments like these show that there is a strong visual component of mental imagery and that in fact the tasks involved in vision and mental imagery strongly interact. This debate raged for a number of years and there were a number of relevant experiments that pushed the debate one way or another. I won't quite get to what is acknowledged to be more or less the resolution of this debate just now, but I will in the next discussion. But here's an interesting experiment, the top one here I'm showing. It was done by an experimenter, a psychologist named Ronald Fink. His experiment was the following. He showed people a variety of grids. So you have a coarse grained grid. Look at the picture on the left of this trio at the top graph. Look at those three little circles in the bottom there and imagine superposing those three little circles at the center of that target just above them. So in other words, you could be looking at that target and at the center of that target is one of those three grid pictures that you're seeing below. The grid picture at left is coarse-grained, the one in the center is medium grained, and the one at the right is fine grained. Now, let's just stick with this question visually for a moment. So no imagery involved. If you place one of these grids at the center of the target and ask people to look at that picture and then shift their gaze along one of the radii of the target. So let's imagine that you take that coarse-grained grid, you place it at the center of the target. Again, this is all vision you're actually staring at this thing. Now, you're asked to shift your attention along one of the spokes of this wheel long, one of the radii of this target, so you move out towards the edge. Then you're asked to report at what point along that radius does the central image blur or you no longer able to really tell the vertical from the horizontal stripes. At what point does the central image become a blurred to you? So you mark the distance along each radius where the central image becomes a blur. As you might imagine, those distances are longer for the coarse-grained grid than they are for the fine-grained grid. That is, if you put the fine-grained grid at the center of that wheel and then move your attention along one of the radii, if you put the fine-grained grid there, it doesn't take very long before the image starts to blur. Now, instead of doing this with the grid there, what we will now do is, have somebody look at the grid and form a mental image of the grid at the center of the target. So in the second condition, instead of actually looking at the grid at the center of the target, they have a mental image of one of those grids at the center of the target. Again, they are asked to move along radius and report in their image when the line start to blur. Couple of very interesting things that are represented in the center and right graph in the upper pictures here. First, what you're looking at in the second picture is that as the spatial frequency as the frequency increases, in other words, that the grid goes from coarse to medium to fine, that's the x-axis. Then the radius at which the distance that you have to go to blur an image along some particular radius goes down just as we said. So on the x-axis there is the think of it as coarse, medium, fine, the spatial frequency of the grading. On the y-axis is how far you have to go along a particular radius for the image to blur. Naturally, the image blurs at a higher radius for the coarser grain. There are two graphs here, black and red. I'd have to look carefully at the caption to this but as I recall the red graph is the imagery graph and the black graph is the vision graph. The reason I'm not sure it doesn't even matter really for the purposes of this discussion. The point is those two graphs look pretty similar. The red and black line look pretty similar. So people respond to this purely psycho-physical question of how an image blurs. They respond to that question pretty much identically whether they're dealing with an actual perceived picture or a mental image. That's interesting, that the graph at the right is even more interesting. It's showing how the blur distance changes all around the target. In other words, if you look at it because its so squashed and it's longer on the left and right than it is the top and bottom. One of the things you can say about this is that the blurring distance as we move horizontally tends to be greater than the blurring distance as we move vertically. This is for one particular choice of grading frequency. So in other words, you take a particular choice of grading frequency, and then see along each radius how far you have to go to blur the frequency at the center. One thing you find is that it's simply true that you have to go further along the left and right distances to blur the image we seem to have sharper vision that way than if you go up and down. Again, there's a red and a black graph here and the red line is the visual imagery condition and the black line is the true picture condition. Again, they look virtually the same. Now, one of the questions that was asked about this is, if the symbol is all right. If it's really true that you don't need to postulate a visual component of mental imagery. If the symbol is all right, how did they explain this experiment? It's a very tricky experiment to explain for the symbolist. The question would be, why would such a purely visual phenomenon be replicated with a representation that was purely symbolic? People, the subjects going into the experiment have no idea that they should take longer to blur the grading if they move left and right than if they move up and down. The subjects going into the experiment had no idea about the nature of the sharpness of vision that way. It's not as though they're expecting to give some kind of answer from common sense knowledge. This is not part of our common sense knowledge. So the fact that people would give very similar answers when they were forming a mental image as when they were looking at the picture itself, strongly suggests the debates raged on so wasn't pure proof, but it strongly suggests at least to a reader like me that yeah there's a visual component of mental imagery and it can't be reduced entirely the symbols. A simpler experiment involves something called the Ponzo illusion. The Ponzo illusion, so you're seeing that toward the bottom here. So it's one of the common optical illusions that you'll see along with the what's called the Muller-Lyer Illusion. Which is the illusion where you have two lines which are of the same length, one has wise at the end of it and one has arrow bars and people naturally see the arrow bar blindness being shorter than the Y line even though they're the same length. In the Ponzo illusion you take two lines of equal length. These blue lines are of equal length and you have two diagonal lines surrounding them. People reliably see may not be the strongest illusion but they generally see the upper line as being longer than the lower line, even though in fact they are the same length. The explanation of this is usually that the diagonal lines conjure up some of the response that you would have from looking at a road in front of you or a railroad track if you want to think of it that way. If you were thinking of those two lines as parallel lines of a railroad track going off into the distance, then the line toward the back corresponds to a larger object because it's more distant than the line toward the front. They'd look they have the same length on your retina, but you interpret the upper line as being more distant and hence longer. Naturally though this is still an illusion, as written on the paper these are just two diagonal black lines and two horizontal blue lines. When you look at them you see the blue lines as being of equal length, excuse me, even though the blue lines are of equal length you see them as being of unequal length. Now here's the experiment that was done. You could induce, you could show people these two blue lines purely without anything else. Then ask them to form a mental image of the black line, the black diagonal lines around them. Once people formed the mental image of the black lines around them, they could induce the Ponzo illusion, that is to say they would then say "Oh the upper blue line looks longer to me than the bottom blue line." Note, in this case they are not actually looking at the full Ponzo Illusion. There's just looking at two blue lines of equal length and they're imagining the surrounding railroad tracks. But the mental image was enough for them to get the optical illusion. Again, as with the think experiment up here at the top, the question arises, how would you explain that purely symbolically? The Ponzo illusion has a visual explanation. If it has an explanation in terms of vision and it's replicated by mental imagery, it seems. It seems suggestive, I'm tempted to say unavoidable, but it seems suggestive that there is a visual component to mental imagery. It's replicating a visual optical illusion. The people, again, coming into the experiment, they don't know about the Ponzo Illusion. So it's not like they're expecting the lines to change length. As with most optical illusions, despite their better efforts, they see the upper line as longer when they form the mental image of the two black diagonal lines. Maybe the most striking experiment along these lines had to do with something called the McCollough effect. Now, I should warn you if you're looking at this slide, please don't stare too long at the picture on the right of the screen here. I actually mean this. The McCollough effect is a color-related afterimage effect. Visual afterimage effect that is orientation-specific. Here's how it works. Again, don't, unless you're really into this kind of thing, I would suggest not doing this. But if you stare for a long time at the green patch on the left and the red patch on the right, and the green patch on the left and the red patch on the right, and you did this for a while, then when you are shown plain grids, that is plain striped patterns of black stripes, then you get a color afterimage where the vertical stripes are associated with because the vertical stripes were associated with green here, when you look at the vertical stripes afterward, plain vertical stripes, you see pink background behind them, and with the horizontal stripes, you see green background behind them. This is not as some color images, color illusions are. This is not a retinal effect. It seems to have to do with the primary visual cortex. I am told, I've never really gone wholeheartedly into becoming a subject of the McCollough effect, but I'm told that this aftereffect, this color aftereffect can last days, maybe even longer before it finally wears off. So I don't really want to get myself into this, where every time I'm looking at a series of black vertical stripes, I see pink behind them. It takes a long time to go away. So it's not a matter of the fatigue of the cones in the retina. It seems to have to do with primary visual cortex. So this is, again, a visual effect that you can replicate if you stare at these two patches long enough. Now, when people are shown, I'm going to risk showing you the previous thing one more time, when people are shown just a block of green without stripes and a block of red without stripes, and are asked to form mental images of black vertical stripes on the green patch and black horizontal stripes on the red patch, when they're shown these two color patches but without the black bars, and asked to form mental images of the black bars, they still get the McCollough effect. I'm told weaker than in the original case, but they still report getting the McCollough effect after doing that. Now think about that. Nobody who is a subject comes into this experiment knowing about the McCollough effect. In fact, if I came into this experiment, I wouldn't even want this effect to happen to me. So if I could I'd resist it. The idea is you're shown these two plain color patches and then asked to form images of black stripes in different orientations on the two of them, and then later, you report seeing these color afterimages. No one would have, no subject would have expected that to happen. It again, it seems they purely visual effect but it's one that's induced by mental imagery. So all of these experiments that we've been talking about lend themselves to some really interesting questions about the relationship between vision and imagery. I will sum up for now and talk about this some more in the next discussion. Certain types of operations may be performed on mental images. It seems that we can rotate mental images as in the Shepard/Metzler experiment with 3D blocks, or we can scan mental images as in the Kosslyn, Ball, and Reiser experiment with the island. There are some visual effects that mental imagery does seem to recreate. An optical illusion like the Ponzo Illusion, the McCollough effect, this blurring of grids as in the experiment by Finke. But mental imagery is not like vision in other respects. We find it difficult to reinterpret ambiguous figures like the duck-rabbit experiment. So to sum up for now, it seems that we've got a bit of a puzzle. We have this phenomenon of mental imagery that seems to overlap in many important respects with vision, but it isn't quite the same as vision. It also seems to be, to go back to our earlier discussion, it also seems to be really important to human thinking. All that said then, we've got something very interesting for the person interested in both machines and minds. How does the mind work that it produces a phenomenon like this, and what can we say about this phenomenon computationally that would help us understand it better? That's where we're headed in the next discussion.
