  We propose a novel method that renders point clouds as if they are surfaces.
The proposed method is differentiable and requires no scene-specific
optimization. This unique capability enables, out-of-the-box, surface normal
estimation, rendering room-scale point clouds, inverse rendering, and ray
tracing with global illumination. Unlike existing work that focuses on
converting point clouds to other representations--e.g., surfaces or implicit
functions--our key idea is to directly infer the intersection of a light ray
with the underlying surface represented by the given point cloud. Specifically,
we train a set transformer that, given a small number of local neighbor points
along a light ray, provides the intersection point, the surface normal, and the
material blending weights, which are used to render the outcome of this light
ray. Localizing the problem into small neighborhoods enables us to train a
model with only 48 meshes and apply it to unseen point clouds. Our model
achieves higher estimation accuracy than state-of-the-art surface
reconstruction and point-cloud rendering methods on three test sets. When
applied to room-scale point clouds, without any scene-specific optimization,
the model achieves competitive quality with the state-of-the-art novel-view
rendering methods. Moreover, we demonstrate ability to render and manipulate
Lidar-scanned point clouds such as lighting control and object insertion.
