In this paper we consider the problem of reconstructing the 3D position and surface normal of points on an unknown, arbitrarily-shaped refractive surface. We show that two viewpoints are sufficient to solve this problem in the general case, even if the refractive index is unknown. The key requirements are (1) knowledge of a function that maps each point on the two image planes to a known 3D point that refracts to it, and (2) light is refracted only once. We apply this result to the problem of reconstructing the time-varying surface of a liquid from patterns placed below it. To do this, we introduce a novel ``stereo matching'' criterion called refractive disparity, appropriate for refractive scenes, and develop an optimization-based algorithm for individually reconstructing the position and normal of each point projecting to a pixel in the input views. Results on reconstructing a variety of complex, deforming liquid surfaces suggest that our technique can yield detailed reconstructions that capture the dynamic behavior of free-flowing liquids.

Kiriakos N. Kutulakos (University of Toronto)
Nigel Morris (University of Toronto)

Nigel J. W. Morris and Kiriakos N. Kutulakos, Dynamic Refraction Stereo, In Trans. on Pattern Analysis and Machine Intelligence (TPAMI) , Vol. 33, Issue 8, pages 1518--1531, Aug. 2011.[Pdf].

Nigel J. W. Morris and Kiriakos N. Kutulakos, Dynamic Refraction Stereo, In Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV) , pages 1573--1580, 2005.[Pdf].[BibTeX].

The ICCV 2005 poster is here
The ICCV 2005 video is here
Nigel's related thesis is available in pdf here.
Powerpoint presentation here.

Experimental setup

The Figure shows our experimental setup with dual SONY DXC-9000 3CCD cameras mounted above the water tank. The pattern is visible at the bottom of the tank and was lit from below. We captured data at 60 frames per second at a pixel resolution of 640x480.

 

Results

Pouring Water Data set

This sequence shows a reconstruction of water slowly being poured into our tank. The water is at most several millimeters deep as it advances across the reconstructed area. We show the stereo inputs as well as two rendered reconstructions below, the first is of a solid mesh, and the second attempts to more closely match the appearance of the water with a transparent, reflective material:

Ripple Data set

This set shows a reconstruction of a 2cm deep water surface when a drop of water has been dropped into it. Our reconstruction is replayed at 30 fps to allow better viewing. We began reconstruction a few frames after the initial drop. We show the stereo inputs as well as the rendered reconstructed mesh blow:

Waves Data set

This sequence shows waves propagating from the left to the right on a water surface at a level of ~24mm.