Levine was born in a slum in Boston, Massachusetts, on September 9, 1905. The youngest of six children of a Russian-Jewish immigrant tailor, Joe did whatever work he could to help support his mother, a widow who had remarried only to have her second husband abandon her. This led Joe (in his later years) to tell an interviewer that he had known (in his words) "not one happy day" growing up. At 14 years of age he was hired for full-time work in a dress factory and left school, never to re-enroll.
