New system turns camera into an optical directional microphone

[10:09 Sat,25.June 2022 by Thomas Richter]

Researchers at Carnegie Mellon University have developed a camera system that can detect sound vibrations with such precision and detail that it is able to reconstruct the sound of a single instrument in the midst of an orchestra using only video, thus functioning as a quasi-optical directional microphone that can pick up a specific sound source even from some distance.

Even the most powerful and well-directed microphones cannot eliminate background noise, ambient noise, and the effects of acoustics such as reverberation when recording audio. The novel system, on the other hand, uses two cameras capable of capturing video at 60 and 130 fps, respectively, and a low-power laser (with the light intensity of a standard laser point) to capture high-velocity, low-amplitude surface vibrations that are invisible to the human eye. These vibrations can be used to reconstruct sound, capturing isolated audio signals without inference or a microphone.

The team conducted several successful demos to test the system&s function in capturing vibrations and the quality of sound reconstruction. A primary instrument can be used as a sound source as well as an object that vibrates only indirectly. Thus, isolated audio data were successfully recorded from individual guitars playing simultaneously as well as from individual loudspeakers playing different music simultaneously. But the sound of a speaker could also be analyzed indirectly by means of vibrations from a tuning fork or a chip bag located near the speaker.

The new system is a significant improvement over previous attempts to capture sound using computer vision-so it now uses two ordinary cameras (one with a rolling shutter, one with a global shutter) that cost only a fraction of the high-speed versions used in previous research, but now deliver higher recording quality at the same time.

How does the optical microphone work?

The system works by analyzing the differences in the speckle patterns of images captured with a rolling shutter and a global shutter. An algorithm calculates the differences in these patterns of the two video streams and converts these differences into oscillations to reconstruct the sound.

A speckle pattern refers to the way coherent light behaves in space after reflecting off a rough surface. The team creates the speckle pattern by pointing a laser at the surface of the object producing the vibrations, such as a guitar body. This speckle pattern changes as the surface vibrates. A rolling shutter captures an image by scanning it quickly, usually from top to bottom, creating the image by stacking a row of pixels. A global shutter captures an image in a single instant.

What it can do

.
In the future, the team&s new optical vibration detection system could allow sound engineers, for example, to monitor the music of individual instruments without disturbing the rest of the ensemble in order to fine-tune the overall mix. Or manufacturers could use the system to monitor the vibrations of individual machines on a factory floor to detect early signs that maintenance is needed. But of course, the new system could also be used to record conversations undetected from a distance, for example, as is already possible with special laser-based systems.

The research paper titled "Dual-Shutter Optical Vibration Sensing" can be downloaded

here.

more infos at bei imaging.cs.cmu.edu

deutsche Version dieser Seite: Forschungsprojekt macht Kamera zu optischem Richtmikrofon