Singer was born on May 3, 1924, in Detroit, Michigan, to Polish Jewish immigrants. His father Simon was employed as a printer and only spoke Yiddish, and his mother, Freda (Rosemaity), worked as a seamstress. Singer learned English swiftly and subsequently taught it to the rest of his family. Isadore was born with a prominent hemangioma birthmark under his right eye.
