Williams was the son of Virginia state legislator and slave owner Saml. C. Williams. He spent his early life in the Shenandoah Valley in Virginia, later moving to Dubuque, Iowa after graduating from the University of Virginia in 1857. Later that year, he became a part of the law firm that was owned by his brother in law John T. Lovell.
