Miller received his secondary education at the academically selective Melbourne High School, where his mathematics teacher was the incumbent Australian cricket captain Bill Woodfull. He was an academically poor student who neglected his studies but excelled at sports. He made the school's first-choice cricket team at the age of 14 and gained attention for his technical and defensive ability, which was compared to that of Woodfull. Miller was rejected by the St Kilda Cricket Club so he joined South Melbourne in district competition, making his first-grade debut at the start of the 1935–36 season, still only tall. In one match, he scored 61 runs to orchestrate a lower-order fightback in a low-scoring match against Woodfull's Carlton team, prompting his teacher to award him a silver eggcup during maths class. It remained one of his most treasured possessions.
