Brown was born in London in 1854. For his parents, Jessie Gilmour and John Brown, it was their third boy and last child. Both parents had Scottish ancestry. His father worked for a bank, and was later a commercial traveller. The family briefly lived in France before emigrating to Victoria, Australia. John Vigor Brown, his brothers and their mother arrived in Melbourne on 22 January 1862 on the Water Nymph. It is assumed that his father was already there. They made their home in South Yarra. He was educated at Melbourne Church of England Grammar School. James Drysdale Brown was an elder brother.
