James Gell was born on 13 January 1823, the son of John Gell, of Kenna, in the parish of German, Isle of Man and Margaret (née McHutchin) of Peel, Isle of Man. He was the second of three children, his brothers being John Gell (Jnr) and Captain Phillip Gell (1827-1898).
