They had seven children, including Henry, 2nd Viscount, Arthur, 3rd Viscount and John, 4th Viscount Harberton. His youngest daughter, Mary, married Sir John Craven Carden, 1st Baronet of Templemore, County Tipperary.The gentleman's magazine, and historical chronicle, Volume 79, Part 1, pg 130. Another daughter, Henrietta, married James Hewitt, 2nd Viscount Lifford, but died after only two years of marriage. Two other children, George and Elizabeth, died unmarried. He was an ancestor of General George Colley (1835–81).
