Ferguson was born in Belfast to a Roman Catholic family."History", The Irish Rovers Official Website. Gives his age in 1963 as 23. He emigrated to Canada as a young man, and met George Millar in 1963 at an Irish function in Toronto. They decided to form a folk duo and soon after were joined by Millar's cousin, Joe Millar plus two Torontonians, Vic Marcus and Doug Henderson while adopting the name The Irish Rovers. In 1979, Jimmy told Canadian Music Magazine, "We actually formed the group in Toronto. I sang, George played guitar and at that time Joe was playing a little button key accordion. We were ambitious in those days, we'd played the clubs to death in Toronto and we didn't know what to do next; what direction to take. So we (George, Joe and Jim) decided to visit Will in Calgary, ended up staying there, and the Irish Rovers became four."". Joe moved his family out from Toronto, and the band continued in Calgary.
