Born in Lincoln on 21 June 1787, he was the eldest son of William Willson (1745-1827) of Lincoln by his wife Clarissa, daughter of William Tenney. Robert William Willson was his younger brother. He was brought up a Roman Catholic, and, after education at the grammar school in the Greyfriars, Lincoln, he joined his father's building company. His father was a cabinet maker and joiner, but also described himself as a "master-builder". He was working as a woodcarver in Lincoln Cathedral around 1805 when a chance meeting with John Britton led to his friendship with the latter and developed his interest in architectural writing. He is said to have received some architectural training from a "local architect". This is likely to have been William Lumby, a Lincoln architect, who was also the Cathedral surveyor.E. J. Willson Lincolnshire Collection in the Society of Antiquaries
