When Hart became governor in 1872, he needed someone he could trust to fill his position on the bench. He appointed Fraser to the Court January 16, 1873. After Hart died in March 1874, Fraser resigned from the court that May, expressing dissatisfaction with his salary and with Southern society. He had served 16 months on the court, having written opinions in seven minor cases. He returned to his law practice in Pennsylvania. He died of pneumonia on November 10, 1879.
