Smith was brother of Jeremiah Smith and uncle of Robert Smith. He married Sally Garfield on November 10, 1793 and they had twelve children: Jeremiah, Frederick A., Maria, Samuel Garfield, Albert, William Sidney, Alexander Hamilton, Elizabeth Morison, Sarah Jane, Maria, Mary Soley, and Ellen.
