Lytle grew up in Springfield, Ohio in a family of musicians, the son of a trumpeter father and an organist mother. He began playing the drums and piano at an early age. Before studying music in earnest, he was a boxer, and was a successful Golden Gloves champion. During the late '50s, Lytle continued to box, but landed jobs as a drummer for Ray Charles, Jimmy Witherspoon and Gene Ammons. Then he switched from drums to vibraphone and toured with organist Hiram "Boots" Johnson from 1955 to 1957. He formed his first group in 1957 with saxophonist Boots Johnson, organist Milton Harris and drummer William "Peppy" Hinnant. He impressed the producer Orrin Keepnews who signed him to his Jazzland label in 1960.
