Crosby was born on November 3, 1817. He was one of twelve children born to William Bedlow Crosby (1786–1865) and Harriet Ashton (née Clarkson) Crosby (1786–1859). His parents built one of Manhattan's largest houses, modeled after a Regency house in London, that occupied two blocks on Monroe Street (now known as Rutgers Place).
