Brimhall was the son of Sylvanus Brimhall and his wife the former Lydia Ann Guiteau. He was born along Canada Creek in the state of New York. In 1827, the family moved to Olean Point, New York and a short time later to Melville, Cattaraugus County, New York. After cutting and milling lumber at this location, the Brimhalls loaded it into a boat, went down the Allegany River to Pittsburgh and sold it, and then went down the Ohio to Dearborn County, Indiana. In 1837, Brimhall moved with his family to McHenry County, Illinois.
