Rayleigh Castle gradually fell into disuse, and in 1394 King Richard II gave permission for the townspeople of Rayleigh to use the foundations as a source of stone. By this time, the site was in royal ownership and used for pasture:
