Olivia's final years were difficult. Initially there were three executors of her husband's will, Olivia, her son Anson and son-in-law William Earl Dodge. Anson and Dodge withdrew because of conflicts of interest in respect to their business connections to Phelps Dodge & Co. The will was contested on many points by the beneficiaries, including their children and grandchildren, and Olivia sought clarification via the courts. The complexity of the arguments resulted in up to 12 lawyers in court at any one time. In 1858, Olivia's only son, Anson Jr., died of smallpox and within a year, Olivia herself died in New York on April 24, 1859. It was 1861 before the final verdict on Anson Greene Phelps's will was given by the Court of Appeal.
