Charles was the youngest son of William Armytage-Moore J.P. (1806–1883) who managed the Annesley Estates at Castlewellan in Northern Ireland. He was brother of Priscilla Cecilia, the Countess of Annesley, (wife of the 3rd Earl). William Armytage-Moore (sometimes written 'Armitage-') married Mary Elizabeth Lockwood, née Metcalfe (1845–1932), in the British Embassy Paris on 8 June 1869.The Leeds Mercury, Saturday 12 June 1869. It was a second marriage for both, both having been widowed. Their home was at Arnmore, just outside Cavan.
