  Transformers have achieved great success in pluralistic image inpainting
recently. However, we find existing transformer based solutions regard each
pixel as a token, thus suffer from information loss issue from two aspects: 1)
They downsample the input image into much lower resolutions for efficiency
consideration, incurring information loss and extra misalignment for the
boundaries of masked regions. 2) They quantize $256^3$ RGB pixels to a small
number (such as 512) of quantized pixels. The indices of quantized pixels are
used as tokens for the inputs and prediction targets of transformer. Although
an extra CNN network is used to upsample and refine the low-resolution results,
it is difficult to retrieve the lost information back.To keep input information
as much as possible, we propose a new transformer based framework "PUT".
Specifically, to avoid input downsampling while maintaining the computation
efficiency, we design a patch-based auto-encoder P-VQVAE, where the encoder
converts the masked image into non-overlapped patch tokens and the decoder
recovers the masked regions from inpainted tokens while keeping the unmasked
regions unchanged. To eliminate the information loss caused by quantization, an
Un-Quantized Transformer (UQ-Transformer) is applied, which directly takes the
features from P-VQVAE encoder as input without quantization and regards the
quantized tokens only as prediction targets. Extensive experiments show that
PUT greatly outperforms state-of-the-art methods on image fidelity, especially
for large masked regions and complex large-scale datasets. Code is available at
https://github.com/liuqk3/PUT
