Caryville was a large and prosperous mill village that at one point in the 1800s petitioned to be set off as a separate town. Into the early 20th century, the village had an active baseball rivalry with the Dean College baseball team in the center of Franklin.
