By his first wife, Margaret, daughter of Alexander Irvine of Drum, Aberdeenshire, Ogilvy had a daughter Helen, married to James Ogilvy, 2nd Earl of Airlie; and by his second wife, Janet, daughter of William Sutherland, 9th of Duffus, Elgin, he had a son George, the second Lord Banff, and two daughters.
