See the night in a new light

Researchers at the Technion and the University of Toronto have developed technology to generate knowledge about the environment - from the level of the office to the entire city - based on the flickering of light bulbs

Artificial lighting fills our lives at home, in the office, on the road and in other places. This lighting is produced by a variety of bulbs found in street lights, offices, spotlights, car headlights, billboards and computer monitors. All the lamps connected to the mains flicker continuously, but due to the high flickering speed we do not feel these flickering.

It is now becoming clear that the flickering of artificial lighting may provide extensive useful information. In a study that will be presented this week at the conference IEEE CVPR To be held in Hawaii, researchers from the Technion and the University of Toronto will present a new way to extract information about the environment from the jitter patterns of artificially lit arenas. The approach combines various research fields including optics, computer vision, image processing and electrical network engineering.

The co-researchers in the study are PhD student Mark Sheinin and Prof. Yoav Shechner from the Viterbi Faculty of Electrical Engineering at the Technion and their colleague Prof. Kyros Kotoulakos from the University of Toronto. The three have developed a system that produces the aforementioned information from passive video recording, meaning without additional lighting, of the desired scene: an office, a corridor, a street and even an entire city. From the analysis of the information obtained from the photograph, the system concludes, for example, how the scene would have looked if we turned off some of the bulbs, increased them or replaced some of them with bulbs of a different type. This system also makes it possible to cancel, in digital processing of the photo, reflections returned from windows when we look through them in or out; eliminate the shadow created by artificial lighting on our face in a selfie; and analyze the urban lighting network and the types of bulbs used in it. All this, as mentioned, through passive video recording only.

Prof. Shrinivas Narasimhan from the School of Computer Science at Carnegie Mellon University, who was not involved in the research, said that the paper demonstrates "a very innovative study that extracts significant information from an optical phenomenon that we do not normally notice. This development has significant potential applications, including monitoring light pollution, monitoring air pollution at night, imaging areas that are not in the viewer's field of vision, and monitoring power consumption and fluctuations in the electric current."

The reason for the flickering of artificial light is that electrical networks operate on alternating current (AC), where the flow of electrons constantly reverses its direction in a cyclical manner. In Israel, for example, the frequency in the electricity network is 50 Hz, and this means that the electron current changes its direction 100 times per second and this is the rate at which the light bulb blinks. The flickering pattern depends on the type of bulb - fluorescent, mercury, halogen, LED, etc. - since each bulb converts electrical energy into light in a different process. In other words, Each bulb type has a unique time signature. It is the time signature that allows the researchers, using the system and a dedicated algorithm they developed for this purpose, to identify the type of bulb.

For photographers, photographing the flicker is a challenge that has a contradiction: on the one hand, to detect the dynamics of the flicker, a very short exposure is required. On the other hand, shooting at night requires a long exposure to collect enough light to create a picture. To resolve the contradiction, the researchers developed a unique electro-optical camera called ACam that is able to "sense" the flickering of the alternating current. The camera, which is itself connected to the mains, takes advantage of the jitter cycles to capture fast signals from the arena. The electronic shutter of the camera is open the entire time of the photograph, but the sensor is optically exposed only for the desired time segment in each flashing cycle.

The researchers note that the technology they developed paves the way for further research regarding a wide variety of tasks, including controlled illumination of objects, measuring XNUMXD objects, identifying their surface texture based on the shadow they cast, and analyzing the characteristics of the power grid remotely in an optical way.

The research grew out of Prof. Shechner's interest in a completely different field. Being an amateur astronomer, well aware of the limitations that urban light pollution places on night observations, he hypothesized that if we connect the fractions of a second where the light is minimal during the jitter, this will neutralize the light pollution and allow a clean observation of the sky. The researchers soon discovered that the various light bulbs throughout the city were not flickering in sync; As soon as the light in one of them drops to the minimum point, another bulb actually brightens. "Therefore," he says, "the astronomical application was put in a drawer for now, but this is how we discovered a land unknown to computer vision: the electrical grid."

the article Computational Imaging on the Electric Grid Will be presented on July 22 at the IEEE CVPR conference held by the Society of Electrical and Electronics Engineers (IEEE). The research was supported by the Taub Foundation, the National Science Foundation (ISF) and the German Minerva Foundation.