He married Mary Frances Stewart Mackenzie (1819-1913), daughter of James Alexander Stewart-Mackenzie, the Governor of Ceylon (1837-1941), on 27 July 1838 in Scotland. They had eleven children: Mary Helen (b. 1839); Robert Durham (b. 1841); Annie Nairn (b. 1843); Jemima Stewart (b. 1845); Philip Morrison (b. 1846); Stewart (b. 1848); John Newdegate (b. 1850); Caroline Petre (b. 1928); Keith Francis George (b. 1854); Henry Lewis (b. 1856); and Charles William (b. 1858).
