Charlotte Leof was born in Philadelphia, Pennsylvania, on July 26, 1911, the youngest of three children of Morris V. Leof, a physician, and his wife Jennie Chopin. She had an older brother, Milton, and an older sister, Madeline. Her father was a member of the Socialist Party of America and was involved in left-wing activities. Visitors to their home included playwright Clifford Odets, journalist I. F. Stone, and physicist Wolfgang Pauli. She entered the University of Pennsylvania in 1929 and graduated in 1933. She married physicist Robert Serber soon after. Her father was the uncle of his stepmother, and they had known each other for many years.
