By this time, Ranajang Pande, the youngest son of Damodar Pande, was stationed as a captain in the army in Kathmandu. He was aware of the disunity between Samrajya Laxmi and Bhimsen, and thus he had secretly expressed his loyalty to Samrajya Laxmi and had vowed to help her in bringing Bhimsen down for all the wrongs he had committed against his family. Beforehand, Karbir Pande and his three brothers including Rana Jang, were already called back from Terai region and were given the governmental posts by Bhimsen himself based on the directives issued on traditionally dated 1872 Bikram Samvat, Poush Sudi 12 Roj 6 (1815-16 CE). Historian Chittaranjan Nepali contends that this excessive decency and benevolence towards his arch rivals Pandes is one of the major reason for his downfall. In November 1834, Ranajang requested to the King for the restoration of his family properties and honors, to which the King of Nepal favorably received the request. Ranajang also tried to communicate with the Chinese Amban in Lhasa to restore his family connections with Tibet and falsely tagged Bhimsen being inclined to the enemy of China i.e. the British. Factions in the Nepalese court had also started to develop around the rivalry between the two queens, with the Senior Queen supporting the Pandes, while the Junior Queen supporting the Thapas. About a month after Mathabar's return to Kathmandu, a child was born out of a sexual relationship between him and his widowed sister-in-law. It was a customary practice for a widowed woman to marry and have a sexual relationship with her husband's brother among all 4 Varna and 36 castes of Nepal including the high caste Upadhyaya Brahmans and Chhetris, which was safeguarded by then prevalent Sanatana practices. However, this news was spread all over the country by the Pande faction to defame Mathabar Singh, and the resulting public disgrace forced Mathabar Singh to leave Kathmandu and reside in his ancestral home in Pipal Thok, Borlang, Gorkha. To save face, Bhimsen gave Mathabar the governorship of Gorkha. Thus, British Historian Perceval Landon considered the charge to be "a mere prejudice" against Mathabar and a part of "unceasing intrigues" of the Pande faction for the "exhibition of their hostility".
