After Tairō Ii Naosuke was assassinated in 1860, Iwakura supported the Kobugattai Movement, an alliance of the Court and the Shogunate. The central policy of this alliance was the marriage of the Shōgun Tokugawa Iemochi and Princess Kazu-no-Miya Chikako, the younger sister of the Emperor Kōmei. Andō and Kuze became rōjū and began to have power, and they suggested the marriage for the Kōbu gattai, the settlement of the imperial court and the shogunate. Although the shogunate ordered marriage of Kazunomiya and Iemochi, the emperor Kōmei opposed to this request because Kazunomiya and Arisugawanomiya had been already engaged. Also, he could not understand about Kōbu gattai. At that time, the emperor Kōmei demanded an opinion of Iwakura, because he knew that Iwakura was a talented person. Iwakura suggested that the emperor should let the shogunate promise the executing of the treaty withdrawal and allow the marriage of Kazunomiya and Iemochi. The emperor accepted Iwakura's opinion, and the marriage of Kazunomiya and Iemochi was officially decided and the shogunate declared to execute jōi. Samurai and nobles who supported the more radical Sonnō jōi policy saw Iwakura as a supporter of the Shogunate, and put pressure on the Court to expel him. As a result, Iwakura left the Court in 1862 and moved to Iwakura, north of Kyoto. Emperor Kōmei claimed that Iwakura was innocent, but could not stop people who supported radical Sonnō jōi. He decided to hide because he received a threatening message from Takechi Hanpeita from Tosa clans that if Iwakura did not leave the urban area of Kyoto, he would be exposed to Shijōkawara, which meant his death. He started to live in the Reigenji-temple in Nishigamo, where a grave of Iwakura ancestor was. On that day, he wrote about his despair in his diary. After that he found that the Reigenji-temple was not suitable for a long-term stay, so he moved to the Saihōji-temple. Moreover, the imperial court issued the order of expulsion from the city, so he had to move to Iwakura village. Sanshirō, a foster parent of Iwakura's third son, suggested this place.
