Nakae was the eldest son of a farmer in Ōmi Province. When he was nine years old, he was adopted by his grandfather, Yoshinaga Tokuzaemon, who was a samurai with a stipend of 150 koku serving Yonago Domain in Hoki Province. In 1617, the daimyō of Yonago, Kato Sadayasu was transferred to Ōzu Domain in Iyo Province and Nakae relocated to Shikoku with his grandparents. In 1622, his grandfather died and Nakae inherited a position with a stipend of 100 koku. However, in 1634, at the age of 27, he left his position without officially resigning due to filial piety for his mother and health reasons. After hiding for a time in Kyoto, he returned to his home village of Ogawa in Ōmi (currently part of Takashima, Shiga, where he opened a private academy for Confucian studies. This was the . The Tōju Shoin takes its name from a giant wisteria which grew behind Nakae's house. His students nicknamed him Mr Wisteria Tree, or "Tōju" in the kanbun pronunciation of its kanji.
