Smith was born August 20, 1825, in Milton, Vermont. When he was 16, he left home and found work as a boatman on a canal boat on Lake Champlain; in his spare time, he went hunting and fishing in the Adirondacks, which at that time was largely wilderness with some Native American settlements. In time, Smith became known as a prominent hunting and fishing guide in the Loon Lake region.
