Rogers was born in Bridgehampton, New York on August 1, 1788. He was the eldest son of a farmer, John T. Rogers, and Sarah Brown, the daughter of a Presbyterian minister. Around the age of sixteen, he began to apprentice with a shipbuilder.
