The Ormond manor house seems to have been at the southern end of the village. It was still standing when Fritwell was assessed for the hearth tax in 1655 but seems to have been demolished by 1677, when a map of the village was made that showed no trace of it. Dovehouse Farm seems to have been built on its site and incorporating fragments of the old house. A large dovecote was built for it in 1702 and was still standing in 1897. By 1955 the dovecote had gone and the farm had been renamed Lodge Farm.
