Yoshiyuki was born in Okayama, the oldest child of author Yoshiyuki Eisuke, but his family moved to Tokyo when he was 3. He attended Shizuoka High School, where he grew interested in Thomas Mann's stories, and in 1945 entered the University of Tokyo. He left the university without a degree and began working full-time as an editor at a weekly scandal magazine, while spending much of his leisure time gambling, drinking, and frequenting prostitutes. Sexuality and prostitution would form a consistent theme in his writing.
