Paula Watson in The History of Parliament suggests that Hardy's dismissal came about because of a suspicion that he was a Jacobite; after the Hanoverian succession much of the Tory party was believed to be supportive of the deposed House of Stuart, and the new administration removed many from positions of power. Hardy was also not returned for his seat in the 1715 election. He continued in naval circles after his dismissal, serving two terms as Master of Trinity House between 1729 and 1730. The naval historian John Knox Laughton suggests that Hardy was reinstated in the navy and promoted to vice-admiral of the red, but there is no official report of such a promotion occurring. Hardy died on 16 August 1732, age 67, and was buried with his late wife in Westminster Abbey. His monument, designed in 1737 by Sir Henry Cheere, shows him dressed in Roman armour lying on top of a sarcophagus and is a pair to the monument to John Conduitt opposite his.
