Concern: Face recognition in the crowd will harm privacy

Facial recognition technology could be used to severely limit individual freedoms. What will prevent the police from using the photo database from the identity cards and driver's licenses? More

Israel Benjamin, "Galileo" magazine

In March 2007, the National Institute of Standards and Technology (NIST - National Institute of Standards) of the United States published the results of the 2006 Face Recognition Vendor Test (FVRT - Face Recognition Vendor Test).
The test included, among other things, a comparison of the recognition ability of some algorithms (methods that can be implemented as computer programs) to human ability. The results: three algorithms were able to recognize a face better than the humans who participated in the experiment, in all the conditions included in the experiment for the lighting levels and for the level of separation (resolution) of the face images.
This is probably the first time that a comparative test was performed between humans and computers in this field. However, the best results achieved by computers in previous years are known, so it is also possible to conclude that 2006 was the year when computers really managed to catch up and surpass human performance.
To make this clear, it is necessary to explain the method by which recognition achievements are tested: NIST and other bodies participating in the development and testing of facial recognition software have collected large databases of images, so that for each person in the database there are several images in the database. In one repository, for example, there are 108,000 photos of 36,000 people.
To test the recognition ability, the computer (or the person participating in the experiment) is shown two images selected from the database. The computer must report how much it "believes" that the images are photographs of that person.
This experiment is similar to the actual use when comparing an image of an unknown person to a database of images of people whose identity is known, hoping to find a match of the first image to one of the known identities.
In such tests it is customary to measure the probability of two types of error: the first type is "false accept", meaning a situation in which the computer will make a wrong identification, and report that two photos are of the same person even though this is not the case.
The other type is "false reject", meaning a situation in which the computer fails to detect that the two images shown to it are of the same person. In many reports, the computer programs are required to reach a false acceptance level of about 0.001: only in a thousand of the cases where the computer reports an identification will it eventually turn out that it was a false identification.
It is easy to control the parameters of the software so that the computer reaches such a low level of errors, but this control has a price: when you lower the chance of "wrong acceptance" errors, the chance of "wrong rejection" errors increases. If the computer were human, we would interpret this as if the computer becomes cautious and hesitant, and therefore fears a mistake to such an extent that it avoids in many cases reporting a possible identification.
With the help of these concepts it is possible to compare the progress of the software in the last two decades, when the comparison is made in any case for a false acceptance level of about 0.001: in 1993 the best algorithms reached a disappointing result of 79% false rejection: the computer did not recognize four out of five "suspects" .

An ambitious goalThese algorithms had another limitation: they required human help. For each picture, a person was needed to indicate within the picture the locations of the two eyes. In 1997, fully automatic algorithms already appeared, reaching a false rejection rate of 54%. In 2002, another improvement was achieved, up to 20%. In May 2004, the United States government announced the "face recognition grand challenge" (FRGC - face Recognition Grand Challenge).
This program, which ended in March 2006, was launched to advance facial recognition technologies, and set an ambitious goal: improving performance by one order of magnitude. This goal has been achieved. In the 2006 test, a result of 1% was achieved:
99% of the required identifications were indeed carried out. Although this result was achieved only under the best conditions, of high separation (resolution) and controlled shooting conditions, it is still an impressive progress.
In comparison, for uncontrolled shooting conditions (such as those obtained from security cameras), the best reported achievements were 11% false rejection for the highest resolution (six million pixels in the photo, which, since it was taken in uncontrolled conditions, the face occupied only a small part of it, and the distance The average between the centers of the two eyes was about 190 pixels) and 13% for high resolution (four million pixels in the image, 110 pixels between the eyes).
It is assumed that even for these conditions the results will improve in the near future. It is worth noting that what is called in the experiment "highest resolution" is already now the accepted resolution in many digital cameras.

Who needs facial recognition?In the same test report, the results of two other identification technologies are also presented: identification according to photographs of the iris (the colored part of the eye, which surrounds the pupil), and according to a XNUMXD facial scan (currently there are commercial devices that allow such a scan). These technologies have reached similar achievements to those of facial recognition based on XNUMXD photographs.
Why does the United States government invest so much effort and resources in identification technologies? A clue to the answer is found on the first page of the test report, in the list of funders, which includes the FBI, the DNI (Director of National Intelligence - the United States government's supervisor of the intelligence community) and the Homeland Security department. It's easy to imagine the possibilities. Today, cameras are installed covering many parts of the world's public spaces: government and police security cameras, cameras to prevent theft and cameras operated by tourist organizations and private individuals.
If the American intelligence community stores photos of suspected terrorists, it will be able, with the help of this technology, to scan photos from these many sources, and receive reliable alerts when these suspects appear anywhere in the world. How many attacks can be prevented this way?

How many victims can be saved?It is easy to think of other uses: we have already seen how anxious the population was when a dangerous prisoner escaped, because of the fear that he would commit further acts of violence. When every place the prisoner goes through will be covered with cameras, and the pictures taken there will be checked, his way back to the prison will be faster. Parents whose child has gone missing can also use the tireless ability of computers to scan image after image, camera after camera, until the lost child is found.
You can check not only photos taken at this moment, but also photos taken at some point in the past: more and more cameras are connected to the Internet, so the owners of the cameras can decide whether to give access to anyone who wants it or only to those authorized to do so.
Once access is achieved (legally or otherwise), any image received from all accessible video cameras can be recorded. Even if the missing child is not currently visible on any camera, you can look for him in photos from the past, and thus try to decipher what happened to him and where he disappeared to.
Another example: if the police are trying to link a suspect to a crime committed in a certain location, they can scan cameras in that area near the time the crime was committed, hoping to find the suspect. She can also try to match the recordings to the criminal photo database she has.
Some readers may feel a chill at this point: the same technology could be used to severely limit individual freedoms. What will prevent the police from using the photo database from the identity cards and driver's licenses?
Can an employer look for the face of his employees among the participants in a demonstration who does not agree with its goals? Can he check where an employee who reported that he is sick really is? Education Minister Yuli Tamir says that when she was in high school, she avoided school to participate in the march, but unfortunately the school principal recognized her in a photo published in a newspaper next to an article about that march.
Will the principals of tomorrow's schools receive at their desks every morning a report on the recreation sites where their students were observed? Should parents be interested in monitoring the actions of their teenage children? (It must be remembered that it is no less likely that those boys and girls, who are the first to adopt technological innovations, will find the way to follow their friends, parents and teachers).

Face recognition will help the forgetful to remember namesIf this person's picture is not in my database, the computer will turn to the Internet, through a wireless connection, and check if the mysterious face appears somewhere next to the person's name. That didn't work either? I am forced to ask who that old acquaintance is, then says: "Very nice, Aaron"

Little brother is watching you
It's not just governments that are interested in facial recognition. One of the companies that achieved high results in the FVRT test is called Neven Vision. In August 2006, an internet company known to all of us - Google - acquired Nevan.
Google's explanation for this purchase: Google plans to equip Picasa, its personal photo album management software, with the ability to classify photos by subject (people or landscape, for example), and in the future also identify the people and places that appear in them. Who wouldn't want to search their archives for all the photos taken in a certain place, or all the photos in which the cute niece appears?
The acquisition of Google is just one example: other companies that have achieved great things are Samsung and Toshiba, who also certainly have ideas on how to improve (?) our lives with the help of facial recognition. One option, which has already been presented in several laboratories around the world: to avoid the embarrassing situation where I don't remember who the person who just called my name is, I will wear a miniature camera on my glasses. The camera will be connected to a computer into which I have entered photos and names, and a small earphone hidden in the stem of the glasses will whisper to me the name I have forgotten.
If this person's picture is not in my database, the computer will turn to the Internet, through a wireless connection, and check if the mysterious face appears somewhere next to the person's name. That didn't work either? I am forced to ask who that old acquaintance is, then says: "Very nice, Aaron."
The computer will recognize this statement as a code that requires it to save Aharon's picture next to the name "Aharon", and the next time Aharon will be sure that I remember him (the computer will also be able to remind me where I was when I spoke with Aharon the last time, and play part of our conversation for me on that occasion, but This is a topic for another article).
Here, too, a slippery slope stretches before us. More and more people are uploading their photo archives to the Internet. If somewhere there is my picture next to my name, that is enough to search for my face in any other picture that appears on any site on the Internet. It will also be possible to search for me in the recordings of the broadcasts of any web cam (Web Cam) whose images are accessible to the general public.
Today, only celebrities are recognized wherever they go. Will each of us be an object of search in the future? When one of us prepares for an important, business or social meeting, will he be able to find on the Internet not only where the name of the person he met appeared, but also where he was in the last few days (or even five years ago), where he usually hangs out, which stores he goes to, etc. '?
What will criminals do with the help of such technology? They will be able to know when you are far away and therefore your house can be broken into, they will be able to harass and intimidate, they will look for opportunities to blackmail, they will certainly think of more creative ideas.
These alarming scenarios are not possible yet, but at the current rate of progress, there is no reason why we won't see them come to fruition even before the end of the decade.

Technology gives, technology takes
Ralph Gross, of Carnegie Mellon University's Robotics Institute, knows both sides of the issue well. Among other things, he helped licensing agencies in the United States look for people whose driver's license was revoked in one state, then applied for a new license in another state, or under a different name in that state. It turns out that by comparing the photos of different people, several such criminals were apparently caught, and it is possible that several fatal accidents were avoided.
On the other hand, Gross is working with Carnegie Mellon's Data Privacy Lab to find ways to stop the slide. The approach they propose is based on an interesting fact: although both humans and computers reach a high capacity in facial recognition, they perform the same task in completely different ways.

The mystery of the human method
In fact, we cannot say much about the human method or the computerized method: regarding humans, despite many findings about the places in the brain where recognition takes place, and about the causes of pathological conditions in which the ability to recognize faces is lost (Prosopagnosia - from Greek; prosophon means "face" and agnosia means "Ignorance"), we still do not know how to describe the principles of the method in which the mind works (and see: Yifat Levi and Raphael Malach - "From small to large in the human brain", "Galileo" 49).
On the technological side, although some effective facial recognition algorithms have been published in the past, the companies that achieved the impressive advances reported in FRVT do not reveal the methods by which they reached these improvements.
Even if we cannot examine the different methods of the person and the computer, we can see that there are differences in the test result: people are better at recognizing faces they know well (meaning we have already seen them in different shapes, in different lighting, etc.), and at recognizing faces viewed from unconventional angles (from the side or from below, for example). Computers are better at recognizing faces seen only once and when both eyes are clearly visible.
The methods that make it difficult for humans to recognize faces do not necessarily make it difficult for computers: Gross and his teammates found that the squares that hide the faces of characters on television and in the press still often allow the software to discover the identity of the interviewee - even at extremely high blur levels (square size). Sometimes these squares even improve the recognition ability of the computer, because they remove details that tend to "confuse" the software! (At the end of the article there is a link to an article about this, and the method presented below.)
If so, is it possible to find a reverse process - a process that would prevent a computer from recognizing the face, but still not harm the usability of the image for humans? It seems that such a method exists.
Researchers at the Information Privacy Lab, led by Letania Sweeney, have developed a process in which the facial details of several different people are combined to create a new image, and this image is superimposed on top of the original photograph or video (link at the end of the article).
Their purpose is to deal with situations where the police, for example, think that certain surveillance films may contain information related to the investigation of a crime, but do not have sufficiently convincing arguments for a judge to approve access to those films. The arguments can only be presented after the police actually watch the films - a "catch 22" situation.
Sweeney's team suggests a method of masking identity: the police could watch a version of the film in which all the faces have been replaced by "composite faces", so that each of them is made up in a different way of several different characters. In this film you can even see facial expressions and understand the events, but the participants can only be identified after receiving a judge's order.
During the development, it became clear that a similar method can also be used to create images that people recognize better than the computer (although the recognition will still not be certain) - try the demo (link at the end of the article) to see how many times your guess is more successful than the competing software in you.
This study is part of a broad and important field of research that examines two sides of the same question: on the one hand, the study shows the ability of software to break through the veil of anonymity and privacy - the same veil that we fool ourselves if we believe that it exists and will continue to exist in the future. On the other hand, he develops tools to disguise identity while doing as little harm as possible to the usability of that technology that helps and protects us in our wanderings on the net, in medical processes, or even when we walk down the street. Artificial intelligence plays an important part in both of these aspects.

Israel Binyamini works at ClickSoftware developing advanced optimization methods.
From the August issue of "Galileo" magazine