In 1848 Governor Charles Fitzgerald expressed an interest in a summer residence on Rottnest. He resided temporarily in the First Superintendent's House. His successor, Sir Arthur Kennedy authorised funding for the construction of an official residence and works commenced in 1858. Architect Richard Roach Jewell as Superintendent of Public Works and Towns, designed the two-storey building, which was completed and occupied in 1864.
