Anthony was the son of Eliza (Stout) and John Anthony. He studied drawing and engraving under the best teachers in New York, and was one of the original members of the American Water Color Society. He passed part of his professional life in New York and California, but settled in Boston in 1878.
