The castle was designed and built after Vanbrugh had been the architect of the baroque houses at Castle Howard and Blenheim Palace, and shortly after Vanbrugh succeeded his architectural mentor Christopher Wren as Surveyor to the Royal Naval Hospital in 1716. Vanbrugh took a lease of a 12-acre triangular site of the Westcombe estate from Sir Michael Biddulph, 2nd Baronet in 1718, now known as Vanbrugh Fields.
