Ezra spent his early years growing up at the Roswell Reed estate on the banks of the Hudson in Coxsackie built by his grandfather around 1850. The estate included a large Gothic Revival mansion, numerous dependent structures and a dock on the river from which the family conducted a profitable freighting business. R. R. Fitch lived in Brooklyn for a time, working as an agent for an asphalt company.
