Groves was born in Bow, London, but as a boy he played for Cornard Dynamos in the village of Great Cornard on the Suffolk - Essex border. Groves then had a trial with Wolves as a schoolboy before signing as an apprentice for Colchester United in 1981. He turned professional a year later and over the next four seasons he played 142 league games for the U's, scoring 26 goals.
