During the American Revolutionary War, Fisher and his brothers were exiled in Winchester, Virginia, in 1777, after refusing to swear an oath of allegiance and deliver their mercantile firm's business records to the authorities. They were under house arrest for one year and two of their company, Thomas Gilpin and John Hunt, died in exile. They were eventually pardoned by George Washington and allowed to return to Philadelphia after the British evacuated.
