Clarges was the son of Thomas Clarges of Aston, Hertfordshire and his wife Anne Shute of John Shute, 1st Viscount Barrington and was born on 4 October 1751. He succeeded his grandfather Sir Thomas Clarges, 2nd Baronet in the baronetcy on 19 February 1759. He was educated Eton College in 1765 and matriculated at Christ Church, Oxford in 1770. He married Louisa Skrine, daughter of William Skrine on 20 October 1777.
