Passive Ultra-Wideband Single-Photon Imaging

Abstract

We consider the problem of imaging a dynamic scene over an extreme range of timescales simultaneously—seconds to picoseconds—and doing so passively, without much light, and without any timing signals from the light source(s) emitting it.

Because existing flux estimation techniques for single-photon cameras break down in this regime, we develop a flux probing theory that draws insights from stochastic calculus to enable reconstruction of a pixel’s time-varying flux from a stream of monotonically-increasing photon detection timestamps. We use this theory to (1) show that passive free-running SPAD cameras have an attainable frequency bandwidth that spans the entire DC-to-31 GHz range in low-flux conditions, (2) derive a novel Fourier-domain flux reconstruction algorithm that scans this range for frequencies with statistically-significant support in the timestamp data, and (3) ensure the algorithm’s noise model remains valid even for very low photon counts or non-negligible dead times.

We show the potential of this asynchronous imaging regime by experimentally demonstrating several never-seen-before abilities: (1) imaging a scene illuminated simultaneously by sources operating at vastly different speeds without synchronization (bulbs, projectors, multiple pulsed lasers), (2) passive non-line-of-sight video acquisition, and (3) recording ultra-wideband video, which can be played back later at 30 Hz to show everyday motions—but can also be played a billion times slower to show the propagation of light itself.

Flux Probing

Photon arrivals

Our theory is based on the observation that photon arrivals (white dots in illustration) are rich with information about their underlying flux function (cyan plot). While we cannot measure the flux directly, the photon arrivals can be used to indirectly observe the flux.

Flux reconstruction

We show a simulated stream of photon arrivals with a flux function (top plot) containing 3 frequencies. Using our flux reconstruction algorithm, we can scan the frequencies of the flux function and estimate its Fourier coefficients (bottom left plot). Using a statistical threshold (red curve), we can remove noisy frequencies and reconstruct the flux function (bottom right plot).

Results

Passive ultra-wideband 1D imaging

We demonstrate recovering signals with frequencies spanning roughly 9 orders of magnitude from DC to 10 GHz. We place a single-pixel SPAD in the scene to capture flux variations from (1) pulse-width modulation of a light bulb (900 Hz), (2) backscattered light from a raster-scanning laser projector (60 Hz–10 MHz), and (3) two unsynchronized picosecond lasers (40 MHz–10 GHz). The 1D plot directly visualizes the flux function across multiple timescales

Passive non-line-of-sight video

We demonstrate passive NLOS video reconstruction using light measured indirectly from a raster-scanning laser projector. The SPAD observes a single point on a diffuse box during the projector beam’s raster scan, collecting light that bounced twice (i.e., diffuser→diffuse box→SPAD). This configuration is analogous to dual photography. By reconstructing the 1D flux function over a one-second span, we recover the video being played. We show 1-second reconstructions for non-overlapping exposure times.

Passive ultra-wideband video

We also demonstrate passive ultra-wideband video by raster scanning a scene in which a pulsed laser, with 20 MHz repetition and 80 ps FWHM, is diffused to illuminate a fan spinning at 54 Hz. Our method can render the flux at whatever timescale, essentially freezing time at all timescales. This allows us to show both the fan blades rotating and the propagation of the laser pulse. In contrast, conventional approaches reconstruct the scene at only one of the aforementioned frame rates, temporally blurring either slow or fast events.

Flux probing with 2D SPAD arrays

Our method can be applied off the shelf to data from 2D SPAD sensors. To demonstrate this, we apply flux probing to the data from Seets et al to recover high-speed video with a 32×32 SPAD array.

@inproceedings{wei2023ultrawideband, author = {Wei, Mian and Nousias, Sotiris and Gulve, Rahul and Lindell, David B and Kutulakos, Kiriakos N}, title = {Passive Ultra-Wideband Single-Photon Imaging}, booktitle = {Proc. ICCV}, year = {2023}, }

Passive Ultra-Wideband Single-Photon Imaging