Her mother was Mary Elizabeth Whittle, an English woman from the Isle of Wight, and her older brother, Travis James Johnson, was the first African-American graduate of Columbia University's College of Physicians and Surgeons in 1908. He was born in Chichester, England, in 1883,Obituary of Travis James Johnson. Crisis Magazine, July 12, 1917. and the family moved to New York in 1884. Elise spent her early days growing up in Manhattan, but also spent summers in New Jersey, as her father's family owned a truck farm there. She would later inherit and manage the farm.
