Inoue lived in Ichikawa, Chiba in the 1970s, and moved to Kamakura, Kanagawa from 1989, where he lived until his death. He had three daughters by his first wife, Yoshiko Nishidate, who was a stage actress and political activist. His second wife, Lily, was the sister of essayist and translator Mari Yonehara, and the daughter of Arika Yonehara, a senior member of the Japan Communist Party. Together they had a son.
