The son of the Anglo-Irish cricketer and British Army officer Lionel Milman and his wife, Marjorie Aletta Clark-Kennedy, he was born at Eltham in October 1912. He was educated at Bilton Grange and Uppingham School, before going up to Corpus Christi College, Cambridge. While at Cambridge, he made two appearances in first-class cricket for Cambridge University. The first came in 1932 against Sussex, while the second came against Northamptonshire in 1933. Playing as a slow left-arm orthodox bowler, he took 4 wickets in his two matches, with best figures of 3 for 55. In addition to playing first-class cricket, Milman also played minor counties cricket for Bedfordshire from 1931 to 1936, making 36 appearances in the Minor Counties Championship. He made four rugby union Test appearances for England, debuting against Wales at Twickenham in the 1937 Home Nations Championship. The following year he gained three further Test caps in the 1938 Home Nations Championship.
