Carlton is located in southeastern Madison County at (34.042142, -83.034991). The city lies along former Georgia State Route 72, east of Comer and southwest of Elberton. The South Fork of the Broad River flows just south of Carlton, reaching the Broad River east of town. Watson Mill Bridge State Park, on the South Fork, is southwest of Carlton.
