  Text-to-image synthesis has achieved high-quality results with recent
advances in diffusion models. However, text input alone has high spatial
ambiguity and limited user controllability. Most existing methods allow spatial
control through additional visual guidance (e.g., sketches and semantic masks)
but require additional training with annotated images. In this paper, we
propose a method for spatially controlling text-to-image generation without
further training of diffusion models. Our method is based on the insight that
the cross-attention maps reflect the positional relationship between words and
pixels. Our aim is to control the attention maps according to given semantic
masks and text prompts. To this end, we first explore a simple approach of
directly swapping the cross-attention maps with constant maps computed from the
semantic regions. Some prior works also allow training-free spatial control of
text-to-image diffusion models by directly manipulating cross-attention maps.
However, these approaches still suffer from misalignment to given masks because
manipulated attention maps are far from actual ones learned by diffusion
models. To address this issue, we propose masked-attention guidance, which can
generate images more faithful to semantic masks via indirect control of
attention to each word and pixel by manipulating noise images fed to diffusion
models. Masked-attention guidance can be easily integrated into pre-trained
off-the-shelf diffusion models (e.g., Stable Diffusion) and applied to the
tasks of text-guided image editing. Experiments show that our method enables
more accurate spatial control than baselines qualitatively and quantitatively.
