Herbert was born at 49 Belgrave Square, London, the second son of Sidney Herbert, 1st Baron Herbert of Lea (who was the son of George Augustus Herbert, 11th Earl of Pembroke, by his second wife Catherine Woronzow) and Mary Elizabeth, daughter of Lieutenant-General Charles Ashe à Court. George Herbert, 13th Earl of Pembroke, was his elder brother, and Sir Michael Henry Herbert his younger brother. Catherine Woronzow was the daughter of a prominent aristocratic Russian family, the Woronzows.Woronzow, HumphrysFamilyTree, accessed 4 April 2012. Catherine's father, Count Semyon Vorontsov, the Russian ambassador to Britain, brought the family to London in 1785. He was educated at Eton and Christ Church, Oxford.
