Broughton fell into decay in the 19th century, but was eventually rescued by The Ven. Frederick Fiennes, 16th Baron Saye and Sele, who served as Archdeacon of Hereford, and who brought in the prominent Victorian architect Sir George Gilbert Scott. At the turn of the century, the house was rented by the Edwardian socialite Lady Algernon Gordon-Lennox, who transformed the gardens and hosted Edward VII at the castle. It was the childhood home of Ivy Cavendish-Bentinck, Duchess of Portland.
