Swift was born in Amenia, New York, the son of Job Swift and Mary Ann (Sedgwick) Swift. In 1786, at the age of five, he moved with his father to Bennington in the Vermont Republic. He attended the common schools in Bennington before attending Litchfield Law School in 1801. He studied law and was admitted to the bar in 1806. He began the practice of law in Bennigton before moving to Manchester to practice law. In 1809 he moved to St. Albans to practice law. He also engaged in banking and farming in the area.
