Waller was born November 21, 1845, in Maysville, Kentucky to parents James B. Waller and Lucy Alexander. Around 1860, the family moved to the western suburbs of Chicago, on a family estate now known as Buena Park. At age 24, Waller became close friends with future White City architect Daniel Burnham as they traveled to Nevada in 1869 to prospect for gold. While they did not get rich, the close friendship continued upon their return to Chicago.
