Challoner's parentage is obscure; however, it is known that he came from the numerous and prosperous Bristol family of the same name. He may have been a son or grandson of Robert Challoner, a previous Sheriff of Bristol, and treasurer of the Society of Merchant Venturers in 1646. The Challoners were well connected in the Bristol shipping and slaving industry, with previous members connected in marriage to the Colston family, and the Knight and Aldworth families, who both owned a sugar refinery in Bristol.
