Robert Alan Rafsky was born July 22, 1945, to civil servant William L. Rafsky of Łódź, Poland and Selma Rafsky née Chafets in Philadelphia. His family was politically active. Lawrence C. Rafsky was his brother. He enrolled at Harvard College in the fall of 1963. Rafsky lived in Wigglesworth and volunteered at the Loeb Drama Center. He was expelled for academic reasons but was later readmitted in 1964 and later became the managing editor of The Harvard Crimson. Rafsky graduated Harvard in 1968.
