Driver married Hazel Kelly on 8 August 1935. They had two children. During Driver's term as Administrator of the Northern Territory, they also adopted a Savunese boy named Bas Wie, who had arrived in the country as a stowaway in the undercarriage of an aeroplane. Bas Wie lived with the Drivers at Government House, until Arthur left Darwin to take up his new post in Italy.
