His son William was created the 1st Milner Baronet, of Nun Appleton Hall in the County of York, in 1717 and was later Member of Parliament for York. The estate then descended in the Milner family until 1875, when the estate's owner, Sir William Mordaunt Milner, 6th Baronet, was more interested in gambling than looking after it. By 1877 it had been leased to William Beckett-Denison, a wealthy Leeds banker. After the death of Sir William Milner in Cairo in 1881, his brother Frederick inherited the estate and in 1882 married Adeline, eldest daughter of William Beckett-Denison. After William Beckett's gruesome death in 1890, the Hall and estate were sold to Angus Holden, a sometime M.P. (later created Baron Holden), a woollen manufacturer from Bradford, whose ownership was somewhat brief as he died in 1912.
