Contrary to Brown's wish, none of his family would remain long at the North Elba farm. In 1860 his three oldest sons, John Jr, Jason, and Owen, were all living in Ohio. Salmon, who later remarked that the Brown family was "despised bitterly" and “our family was long buffeted from pillar to post,” also departed, in his case for California. He was accompanied by his wife, children, mother Mary Brown, and sisters Sarah and Ellen, Mary seeking "a chance to start over in a 'new country". In 1863 Mary leased the farm to Alexis Hinckley, brother of Salmon's wife. In 1865 he purchased it from her, the grave site being exempted, for $800. It was with the proviso, added to the deed, that any interested party should be allowed to cross the property to access her husband's grave. Registers were kept so that visitors could write their name and any comments; Joshua Young left remarks in 1866.
