Stafford was the third son of Humphrey Stafford ("With the Silver Hand"), of Hooke, Dorset and of Southwick, by his wife (and step-sister) Elizabeth Maltravers (died 1420), the heiress of Hooke. He married Catherine Chidiock (died 10 April 1479), daughter of Sir John Chidiock. They had an only son, Humphrey Stafford, 1st Earl of Devon (c.1439–1469). Catherine survived her husband and re-married firstly to Sir John Arundell (died 1473) of Lanherne, Cornwall, and secondly to Sir Roger Lewkenor (died 1478).
