Wharton was born in Covesville, Albemarle County, Virginia; studied law at Dickinson College, was admitted to the Virginia bar, and practiced in Albemarle County. He married Mary "Polly" Philips (6 Sep 1786 - 11 Apr 1813), the daughter of Joseph Philips Jr. and Milberry Horn, on April 20, 1804, in Davidson County, Tennessee. They had five children, John Overton, Joseph Philips, Rhoda Ann, Sarah Angelina, and Mary Philips. Mary died at the age of 26. He also had five children with his second wife, his cousin, Elizabeth Auston Rice, of Virginia.
