Yonge was a direct descendant of James Yonge (1646–1721). He was the fourth son of Duke Yonge, vicar of Otterton, Devonshire, and his wife Catherina, daughter of Thomas Crawley-Boevey of Flaxley Abbey, Gloucestershire. He was born in Devonshire in 1794, and educated at Eton and Exeter College, Oxford, where he graduated B.A. on 13 May 1815, M.A. on 22 October 1817, M.B. on 8 June 1819, and M.D. on 20 June 1821.
