The Domesday Book of 1086 records the village as Celgrave. Later spellings include Chealgraue in 1170 and Chalcgrava in 1236. It is derived from the Old English cealc-græf or cealc-grafu, meaning "chalk or limestone pit". The manor house is early 15th-century, making it the second-oldest building in Chalgrove. It stands partly on the site of an earlier building, originally the de Plessis manor and manorial court-house.
