  Layer segmentation is important to quantitative analysis of retinal optical
coherence tomography (OCT). Recently, deep learning based methods have been
developed to automate this task and yield remarkable performance. However, due
to the large spatial gap and potential mismatch between the B-scans of an OCT
volume, all of them were based on 2D segmentation of individual B-scans, which
may lose the continuity and diagnostic information of the retinal layers in 3D
space. Besides, most of these methods required dense annotation of the OCT
volumes, which is labor-intensive and expertise-demanding. This work presents a
novel framework based on hybrid 2D-3D convolutional neural networks (CNNs) to
obtain continuous 3D retinal layer surfaces from OCT volumes, which works well
with both full and sparse annotations. The 2D features of individual B-scans
are extracted by an encoder consisting of 2D convolutions. These 2D features
are then used to produce the alignment displacement vectors and layer
segmentation by two 3D decoders coupled via a spatial transformer module. Two
losses are proposed to utilize the retinal layers' natural property of being
smooth for B-scan alignment and layer segmentation, respectively, and are the
key to the semi-supervised learning with sparse annotation. The entire
framework is trained end-to-end. To the best of our knowledge, this is the
first work that attempts 3D retinal layer segmentation in volumetric OCT images
based on CNNs. Experiments on a synthetic dataset and three public clinical
datasets show that our framework can effectively align the B-scans for
potential motion correction, and achieves superior performance to
state-of-the-art 2D deep learning methods in terms of both layer segmentation
accuracy and cross-B-scan 3D continuity in both fully and semi-supervised
settings, thus offering more clinical values than previous works.
