Born in Wester-Common, near Glasgow, Milne was educated in Glasgow and emigrated to South Australia, arriving in the Palmyra at Port Adelaide on 29 October 1839, where he married Eliza Disher on 4 March 1842. He formed a partnership with brother-in-law James Disher as wine merchants and as "Disher & Milne", in July 1846, purchased the Hindley Street wine and spirit business of Patrick Auld, who was returning to England. The business was highly successful, eventually becoming Milne and Company.
