After Robert Baker's death in 1623 and the death of his eldest son Samuel shortly afterward, his widow and her father purchased the wardship of their surviving children; the death of the next eldest son, Robert, in 1630, allowed them to effectively control the estate. Their only daughter died, and her widower Sir Henry Oxenden retained an interest in the land. Several relatives claimed it, but after Mary Baker's death in about 1665, the estate reverted to the Crown. A great-nephew, John Baker, obtained possession of part of it, but squabbled over the lands with his cousin, James Baker; trying to play one another off, they paid or granted rights to Oxenden and a speculator, Colonel Thomas Panton, eventually losing out to them. By the 1670s, Panton was developing the lands; despite the claims of some distantly-related Bakers, he steadily built them up.
