How does the eye and the computer see and how does the computer learn to recognize what it sees?

An interview with Prof. Yair Weiss on the occasion of his winning the Bruno Prize

The eye is one of the great mysteries of nature, and is the most important interface of animals, including humans, with their environment. However, in fact, explains Prof. Yair Weiss from the School of Computer Science and Engineering, the Hebrew University of Jerusalem, when we look and see, for example, a field of flowers in front of us, only some of the operations are done with the eye and most of them are done with the brain.

The brain interprets the signals received in the retina of the eye and processes them using an algorithm that to this day computers are not able to imitate effectively. So much so, that even the brain of a three-year-old girl is equipped with the ability to analyze images better than the most powerful computers available today. Prof. Weiss's research question is what is the algorithm and what are the principles that guide these calculations. Understanding the algorithm used by the brain is important and interesting. Improving the vision of computers will be one of the byproducts of this understanding.

Prof. Weiss emphasizes that understanding the algorithm used by the brain can be done without understanding exactly which specific neuronal cell performs the algorithm, just as one does not need to understand how the circuits in a computer work, when using the WORD program.

So what does it mean to see? Prof. Weiss explains that to see is actually to perform an act of calculation. "When a person looks at a red page, for example, the perception that the page is red is a very complex operation because the signal that enters the retina depends on both the properties of the page and the color of the light. The purple room lighting will cause a different signal than the signal that will arrive in white light. We as humans have no problem looking at a fruit and saying what color it is, and it doesn't matter if we are looking at it at sunset, in the shade, under a bright sun or a fluorescent light. On the other hand, for computers or robots trying to decide what color things are in the world, this is a difficult problem.

Humans perform a calculation operation that relies on the image we see and our experience. We look at more details in the scene such as the boundaries of light and shadow, and if the fruit is in light or shadow. We examine the image received on the retina against insights about the behavior of light and connect the signals we know from our experience and decide, for example, whether a check is worthy of velvet or not. Computers can't do that.

"The approach I take is an approach of learning. There is prior knowledge that we assume about the world in addition to the signals that reach the retina and one way is to ask how we learned this prior knowledge. When a baby grows up, he is interested in light and shadow, and when he is old enough to be able to do this, he turns the light on, turns the light off, looks at things from all kinds of angles.

In my research I ask how the laws by which the visual world is governed can be learned from many experiences. Some also call it computer learning or computational learning. And another question, after we learned how the world is supposed to behave and now I saw a new check, how do I connect these signals to know what the color of the check is.

Who is the person who influenced you the most?
Maybe not the man but a book that really influenced me: VISION by a researcher at MIT named David Marr, who died before the book was published. Although I don't deal specifically with the problems he addressed, he encouraged me to venture into the field.

According to him, the academic community should not be closed and should make science accessible to the general public, in order to attract the curious to a scientific career. He personally does this as part of the Magid Institute of the Hebrew University in a lecture on optical illusions.

About
Prof. Yair Weiss is an associate professor at the School of Computer Science and Engineering, The Hebrew University of Jerusalem. He received his PhD in Neural Computation from the Massachusetts Institute of Technology in 1998, after researching the field of visual illusions. and then spent three years as a postdoctoral fellow at the University of California, Berkeley. In 2001 he joined the Hebrew University.