York was born in rural Tennessee, in what is now the community of Pall Mall in Fentress County. His parents farmed, and his father also worked as a blacksmith. The eleven York children had minimal schooling because they helped provide for the family, including hunting, fishing, and working as laborers. After the death of his father, York assisted in caring for his younger siblings and found work as a blacksmith. Despite being a regular churchgoer, York also drank heavily and was prone to fistfights. After a 1914 conversion experience, he vowed to improve and became even more devoted to the Church of Christ in Christian Union. York was drafted during World War I; he initially claimed conscientious objector status on the grounds that his religious denomination forbade violence. Persuaded that his religion was not incompatible with military service, York joined the 82nd Division as an infantry private and went to France in 1918.
