Brown died in December 1865 as a result of a severe fall while walking on a canal boat. A legend exists that Jim had hidden some of his wealth somewhere in the Cuyahoga Valley, and he died before being able to retrieve it.
