Frankel was born in London on 31 January 1906, the son of Polish Jewish parents. He began to learn the violin at an early age, showing remarkable talent; at age 14, his piano-playing gifts attracted the attention of the American pianist and teacher Victor Benham (1867-1936) who persuaded his parents to let him study music full-time.
