Pattabhiraman's father, Sir C. P. Ramaswami Iyer was one of India's front ranking statesman and at the time of his birth, the lawyer with the highest income in India. On 26 January 1925, Pattabhiraman married Saraswathi, the daughter of Captain Dr. P. Krishnaswami, Principal of Stanley College, Madras and niece of Pennathur Subramania Iyer, founder of P. S. High School. The couple had four daughters - Shakunthala (1927-2000), Seetha (1933-1958), Lakshmi and Padmini.
