In his retirement, Oliver lived in Cowandilla. He died on 15 November 1958, aged 67. The only club representatives at his funeral were Bob McLean and Charles Darwent, leading to the former to say in his book 100 Years with the Magpies that "There is a very true saying, when referring to the champions, that they are peacock's feathers today and feather dusters tomorrow ... How the mighty fall!".
