Speaker Charles Abbot wrote in his diary in 1803 that the "rebuilding and altering the Speaker's House, which Mr. Wyatt had promised to complete before winter, proceeded very slowly", but he had still been able to host parliamentarians for dinners. At a dinner at the home of Lord Camden in 1808, The 1st Duke of Cumberland and Teviotdale told Abbot that the Whig MP George Tierney would be complaining in the Commons about the expenditure of £70,000 on the Speaker's House (). Abbot told Cumberland that Tierney should take the issue up with Wyatt. John Britton, writing in his 1815 book Beauties of England and Wales, describes the Speaker's House as having been greatly altered, enlarged and beautified under the direction of Wyatt and that it was "most exquistly and tastefully ornimated" under Speaker Abbot. Following the fire in 1834, the Speaker's residence was partly located at the nearby Jewel Tower. The speaker also lived in a house in Eaton Square in Belgravia while the Palace of Westminster was being rebuilt.
