Edward's elder brother, Isaac A. Coles, served as private secretary to both Thomas Jefferson and James Madison during their administrations. Thomas Jefferson's Monticello plantation was nearby in Albemarle County. Furthermore, the wife of James Madison, Dolley Payne Todd Madison, was Coles's first cousin, and Coles became a frequent guest at their Montpelier plantation also nearby. James Monroe owned Ash Lawn-Highland plantation on the other side of Green Mountain for 24 years (until forced to sell it in 1825 due to financial problems), and offered young Coles use of his library, although the relationship with this family was more distant since Monroe split his time at Oak Hill plantation in Loudoun County.
