Broderick was born in 1820 in Washington, D.C., on East Capitol Street just west of 3rd Street. He was the son of an Irish stonecutter and his wife. His father had come to the United States in order to work on the United States Capitol. In 1823 Broderick moved with his parents to New York City. There he attended public schools and was apprenticed to a stonecutter.
