text: Non-line-of-sight (NLOS) imaging of objects not visible to either the camera or illumination source is a challenging task with vital applications including surveillance and robotics. Recent NLOS reconstruction advances have been achieved using time-resolved measurements which requires expensive and specialized detectors and laser sources. In contrast, we propose a data-driven approach for NLOS 3D localization requiring only a conventional camera and projector. We achieve an average identification of 79 object identification for three classes of objects, and localization of the NLOS object's centroid for a mean-squared error (MSE) of 2.89cm in the occluded region for real data taken from a hardware prototype. To generalize to line-of-sight (LOS) scenes with non-planar surfaces, we introduce an adaptive lighting algorithm. This algorithm, based on radiosity, identifies and illuminates scene patches in the LOS which most contribute to the NLOS light paths, and can factor in system power constraints. We further improve our average NLOS object identification to 87.8 accuracy and localization to 1.94cm MSE on a complex LOS scene using adaptive lighting for real data, demonstrating the advantage of combining the physics of light transport with active illumination for data-driven NLOS imaging.
  @cite_30: Imaging objects obscured by occluders is a significant challenge for many applications. A camera that could “see around corners” could help improve navigation and mapping capabilities of autonomous vehicles or make search and rescue missions more effective. Time-resolved single-photon imaging systems have recently been demonstrated to record optical information of a scene that can lead to an estimation of the shape and reflectance of objects hidden from the line of sight of a camera. However, existing non-line-of-sight (NLOS) reconstruction algorithms have been constrained in the types of light transport effects they model for the hidden scene parts. We introduce a factored NLOS light transport representation that accounts for partial occlusions and surface normals. Based on this model, we develop a factorization approach for inverse time-resolved light transport and demonstrate high-fidelity NLOS reconstructions for challenging scenes both in simulation and with an experimental NLOS imaging system.
  @cite_37: By using time-of-flight information encoded in multiply scattered light, it is possible to reconstruct images of objects hidden from the camera’s direct line of sight. Here, we present a non-line-of-sight imaging system that uses a single-pixel, single-photon avalanche diode (SPAD) to collect time-of-flight information. Compared to earlier systems, this modification provides significant improvements in terms of power requirements, form factor, cost, and reconstruction time, while maintaining a comparable time resolution. The potential for further size and cost reduction of this technology make this system a good base for developing a practical system that can be used in real world applications.
  @cite_14: The functional difference between a diffuse wall and a mirror is well understood: one scatters back into all directions, and the other one preserves the directionality of reflected light. The temporal structure of the light, however, is left intact by both: assuming simple surface reflection, photons that arrive first are reflected first. In this paper, we exploit this insight to recover objects outside the line of sight from second-order diffuse reflections, effectively turning walls into mirrors. We formulate the reconstruction task as a linear inverse problem on the transient response of a scene, which we acquire using an affordable setup consisting of a modulated light source and a time-of-flight image sensor. By exploiting sparsity in the reconstruction domain, we achieve resolutions in the order of a few centimeters for object shape (depth and laterally) and albedo. Our method is robust to ambient light and works for large room-sized scenes. It is drastically faster and less expensive than previous approaches using femtosecond lasers and streak cameras, and does not require any moving parts.
  @cite_18: Recent works have demonstrated non-line of sight (NLOS) reconstruction by using the time-resolved signal from multiply scattered light. These works combine ultrafast imaging systems with computation, which back-projects the recorded space-time signal to build a probabilistic map of the hidden geometry. Unfortunately, this computation is slow, becoming a bottleneck as the imaging technology improves. In this work, we propose a new back-projection technique for NLOS reconstruction, which is up to a thousand times faster than previous work, with almost no quality loss. We base on the observation that the hidden geometry probability map can be built as the intersection of the three-bounce space-time manifolds defined by the light illuminating the hidden geometry and the visible point receiving the scattered light from such hidden geometry. This allows us to pose the reconstruction of the hidden geometry as the voxelization of these space-time manifolds, which has lower theoretic complexity and is easily implementable in the GPU. We demonstrate the efficiency and quality of our technique compared against previous methods in both captured and synthetic data.
  @cite_36: Avalanche photodiodes, which operate above the breakdown voltage in Geiger mode connected with avalanche-quenching circuits, can be used to detect single photons and are therefore called single-photon avalanche diodes SPAD’s. Circuit configurations suitable for this operation mode are critically analyzed and their relative merits in photon counting and timing applications are assessed. Simple passive-quenching circuits (PQC’s), which are useful for SPAD device testing and selection, have fairly limited application. Suitably designed active-quenching circuits (AQC’s) make it possible to exploit the best performance of SPAD’s. Thick silicon SPAD’s that operate at high voltages (250–450 V) have photon detection efficiency higher than 50 from 540- to 850-nm wavelength and still 3 at 1064 nm. Thin silicon SPAD’s that operate at low voltages (10–50 V) have 45 efficiency at 500 nm, declining to 10 at 830 nm and to as little as 0.1 at 1064 nm. The time resolution achieved in photon timing is 20 ps FWHM with thin SPAD’s; it ranges from 350 to 150 ps FWHM with thick SPAD’s. The achieved minimum counting dead time and maximum counting rate are 40 ns and 10 Mcps with thick silicon SPAD’s, 10 ns and 40 Mcps with thin SPAD’s. Germanium and III–V compound semiconductor SPAD’s extend the range of photon-counting techniques in the near-infrared region to at least 1600-nm wavelength.
  @cite_46: Can we reconstruct the entire internal shape of a room if all we can directly observe is a small portion of one internal wall, presumably through a window in the room? While conventional wisdom may indicate that this is not possible, motivated by recent work on ‘looking around corners’, we show that one can exploit light echoes to reconstruct the internal shape of hidden rooms. Existing techniques for looking around the corner using transient images model the hidden volume using voxels and try to explain the captured transient response as the sum of the transient responses obtained from individual voxels. Such a technique inherently suffers from challenges with regards to low signal to background ratios (SBR) and has difficulty scaling to larger volumes. In contrast, in this paper, we argue for using a plane-based model for the hidden surfaces. We demonstrate that such a plane-based model results in much higher SBR while simultaneously being amenable to larger spatial scales. We build an experimental prototype composed of a pulsed laser source and a single-photon avalanche detector (SPAD) that can achieve a time resolution of about 30ps and demonstrate high-fidelity reconstructions both of individual planes in a hidden volume and for reconstructing entire polygonal rooms composed of multiple planar walls.
  @cite_29: A confocal scanning technique solves the reconstruction problem of non-line-of-sight imaging to give fast and high-quality reconstructions of hidden objects.
  @cite_9: 
  @cite_1: We explore the question of whether phase-based time-of-flight (TOF) range cameras can be used for looking around corners and through scattering diffusers. By connecting TOF measurements with theory from array signal processing, we conclude that performance depends on two primary factors: camera modulation frequency and the width of the specular lobe (“shininess”) of the wall. For purely Lambertian walls, commodity TOF sensors achieve resolution on the order of meters between targets. For seemingly diffuse walls, such as posterboard, the resolution is drastically reduced, to the order of 10cm. In particular, we find that the relationship between reflectance and resolution is nonlinear—a slight amount of shininess can lead to a dramatic improvement in resolution. Since many realistic scenes exhibit a slight amount of shininess, we believe that off-the-shelf TOF cameras can look around corners.
  @cite_32: We demonstrate an imaging technique that allows identification and classification of objects hidden behind scattering media and is invariant to changes in calibration parameters within a training range. Traditional techniques to image through scattering solve an inverse problem and are limited by the need to tune a forward model with multiple calibration parameters (like camera field of view, illumination position etc.). Instead of tuning a forward model and directly inverting the optical scattering, we use a data driven approach and leverage convolutional neural networks (CNN) to learn a model that is invariant to calibration parameters variations within the training range and nearly invariant beyond that. This effectively allows robust imaging through scattering conditions that is not sensitive to calibration. The CNN is trained with a large synthetic dataset generated with a Monte Carlo (MC) model that contains random realizations of major calibration parameters. The method is evaluated with a time-resolved camera and multiple experimental results are provided including pose estimation of a mannequin hidden behind a paper sheet with 23 correct classifications out of 30 tests in three poses (76.6 accuracy on real-world measurements). This approach paves the way towards real-time practical non line of sight (NLOS) imaging applications.
  @cite_40: An important goal in optics is to image objects hidden by turbid media, although line-of-sight techniques fail when the obscuring medium becomes opaque. use ultrafast imaging techniques to recover three-dimensional shapes of non-line-of-sight objects after reflection from diffuse surfaces.
  @cite_27: A remote-sensing system that can determine the position of hidden objects has applications in many critical real-life scenarios, such as search and rescue missions and safe autonomous driving. Previous work has shown the ability to range and image objects hidden from the direct line of sight, employing advanced optical imaging technologies aimed at small objects at short range. In this work we demonstrate a long-range tracking system based on single laser illumination and single-pixel single-photon detection. This enables us to track one or more people hidden from view at a stand-off distance of over 50 m. These results pave the way towards next generation LiDAR systems that will reconstruct not only the direct-view scene but also the main elements hidden behind walls or corners.
  @cite_5: 
  @cite_15: We show that multi-path analysis using images from a timeof-flight (ToF) camera provides a tantalizing opportunity to infer about 3D geometry of not only visible but hidden parts of a scene. We provide a novel framework for reconstructing scene geometry from a single viewpoint using a camera that captures a 3D time-image I(x, y, t) for each pixel. We propose a framework that uses the time-image and transient reasoning to expose scene properties that may be beyond the reach of traditional computer vision. We corroborate our theory with free space hardware experiments using a femtosecond laser and an ultrafast photo detector array. The ability to compute the geometry of hidden elements, unobservable by both the camera and illumination source, will create a range of new computer vision opportunities.
  @cite_10: Non-line-of-sight (NLOS) imaging utilizes the full 5D light transient measurements to reconstruct scenes beyond the cameras field of view. Mathematically, this requires solving an elliptical tomography problem that unmixes the shape and albedo from spatially-multiplexed measurements of the NLOS scene. In this paper, we propose a new approach for NLOS imaging by studying the properties of first-returning photons from three-bounce light paths. We show that the times of flight of first-returning photons are dependent only on the geometry of the NLOS scene and each observation is almost always generated from a single NLOS scene point. Exploiting these properties, we derive a space carving algorithm for NLOS scenes. In addition, by assuming local planarity, we derive an algorithm to localize NLOS scene points in 3D and estimate their surface normals. Our methods do not require either the full transient measurements or solving the hard elliptical tomography problem. We demonstrate the effectiveness of our methods through simulations as well as real data captured from a SPAD sensor.
  @cite_25: We use a femtosecond laser and a single-photon avalanche diode (SPAD) camera to develop a system that is able to locate objects hidden from view and track their motion.
  @cite_11: Recent advances in computer vision and inverse light transport theory have resulted in several non-line-of-sight imaging techniques. These techniques use photon time-of-flight information encoded in light after multiple, diffuse reflections to reconstruct a three-dimensional scene. In this paper, we propose and describe two iterative backprojection algorithms, the additive error backprojection (AEB) and multiplicative error backprojection (MEB), whose goal is to improve the reconstruction of the scene under investigation over non-iterative backprojection algorithms. We evaluate the proposed algorithms’ performance applied to simulated and real data (gathered from an experimental setup where the system needs to reconstruct an unknown scene). Results show that the proposed iterative algorithms are able to provide better reconstruction than the unfiltered, non-iterative backprojection algorithm for both simulated and physical scenes, but are more sensitive to errors in the light transport model.
summary: introduced looking around corners" using 5D time-resolved light transport @cite_15 . This was later realized using streak cameras and a femto-second laser to perform elliptic backpropagation @cite_40 @cite_5 . Researchers demonstrated NLOS reconstruction using continuous-wave time-of-flight sensors via optimization @cite_14 and NLOS tracking using array signal processing @cite_1 . Single-photon avalanche diodes (SPADS) @cite_36 have been used for NLOS reconstruction @cite_37 @cite_29 . Advances in reconstruction algorithms for SPADs include accounting for partial occluders and surface normals @cite_30 , confocal NLOS reconstruction @cite_29 , geometric modeling @cite_46 , backpropagation @cite_18 @cite_11 and space-carving @cite_10 . Higher level applications such as detection and tracking have been shown @cite_25 @cite_27 . Recently, neural networks have been applied to time-resolved measurements for NLOS imaging @cite_9 @cite_32 .