Born in Pikesville, Maryland, in his teens he began working at the Baltimore racetrack, as did his younger brother, Eddie. His learning of the business from the bottom up was interrupted by service with the United States military during World War II in which he received two Purple Hearts. Following his discharge, Kelly returned to Thoroughbred racing and obtained his trainer's license in 1945. From then until his retirement from training fifty-four years later in 1998, Kelly won numerous important races and conditioned sixty-five stakes race winners. He trained several very successful horses for owner John M. Schiff including Plugged Nickle, the 1980 American Champion Sprint Horse, and Droll Role, a top runner on both dirt and grass and a winner of the 1972 Canadian International Stakes at Woodbine Racetrack in Canada and the Washington, D.C. International Stakes at the Laurel Park Racecourse in his native Maryland.
